AI Toolkit for VScode - Install Locally - Run LLMs Locally in Windows

---

---

AI Nuggets

Detailed Instructions from YouTube Video Transcript

Prerequisites

• Make sure you have Visual Studio Code (VS Code) installed.
  • If you haven’t used VS Code, download it from code.visualstudio.com.

Installing AI Toolkit Extension in VS Code

1. Open VS Code.
2. Click on the Extensions icon (second last menu item on the left).
3. In the search box, type AI toolkit.
4. Look for “AI toolkit” with the publisher “Microsoft”.
5. Click on “Install”. (You may install the pre-release version as the release version might not be available yet.)

Using AI Toolkit Extension in VS Code

1. After installation, click on the AI toolkit icon on the left.
2. Sign in using GitHub when prompted.
3. Select Microsoft Edge when asked to choose a browser.
4. Log in with your GitHub username and password for authentication.
5. In the terminal, an agent will start, which is used for the model playground.

Exploring Model Playground and Model Catalog

1. Click on “Model Playground” in the AI toolkit sidebar.
2. To download models, click on “Model Catalog” at the top left.
3. Choose a model (e.g., Microsoft’s 53 mini 4K CPU) and click “Download”.
4. Once downloaded, click “Load in Playground” to load the model.

Requirements for Running Models

• A GPU is recommended (e.g., 20 GB of VRAM).
• At least 32 GB of memory RAM is suggested.

Interacting with the Model

1. Provide context and set parameters like temperature, top P, penalties, and response length.
2. Enter your query in the text box (e.g., “What is happiness?”).
3. The model will generate a response based on your system’s resources.
4. You can monitor the response statistics in the terminal window.

Additional Information

• The models use ONNX (Open Neural Network Exchange) runtime for interoperability.
• You can embed the model in your code or ask it to write code for you.

Setting Up Linux on Windows with WSL (Windows Subsystem for Linux)

1. Open PowerShell as an administrator.
2. Type the command WSL --install to install WSL.

Additional Tips

• Some models can run on CPU, and you can try those if you don’t have a GPU.
• You can run models remotely using cloud resources.
• Model fine-tuning will be covered in a subsequent video.

Call to Action

• Subscribe to the channel if you like the content.
• Share the video with your network.

Video Description

• The video description may contain additional information or links related to the content discussed in the video. However, without access to the actual video description, I cannot provide those details here.