Using Llama 3.2 Locally

Learn how to download and use Llama 3.2 models locally using Msty. Also, learn how to access the Llama 3.2 vision models at the speed of light using the Groq API.

By Abid Ali Awan, KDnuggets Assistant Editor on October 1, 2024 in Artificial Intelligence

Image generated with ChatGPT

The new Llama 3.2 models have arrived with lightweight and vision variants. The vision models are good at image reasoning. They take images and prompts to generate a response, while the lightweight models are good at multilingual text generation and tool calling for edge cases.

In this tutorial, we will learn how to access both Lightweight and Vision Llama 3.2 models using the Msty application. Msty is a free chatbot desktop application with a ton of features. You can download and use the open-source model, or you can connect to an online model using the API key.

Downloading Msty and Llama 3.2 Model

To use the Llama 3.2 models locally on your laptop, we first have to download and install Msty. Then, we will download the Llama 3.2 model.

Go to the msty.app website and download the Msty application.
Install it with the default options.

Screenshot from Msty

Click on the Settings button in the bottom left, select the "Local AI" tab, click on the "Manage Local AI Models" button, and then click on "Browse & Download the Online Models."

Change the model provider tab to "Hugging Face" and type the following model repository link: "bartowski/Llama-3.2-3B-Instruct-GGUF".
Select the "Q4_k_M.gguf" version of the model file and click on the download button. It will take a few minutes to download the full model.

Using Llama 3.2 3B Instruct Locally

After completing the download, we will navigate to the chat menu and choose the “Llama-3.2-3B-Instrcut” model to begin using it.

Write the sample prompt in the chatbox and press enter.

Prompt: "What is the fastest and most popular sorting algorithm? Please provide the code example."

The results are quite accurate, with detailed explanations. I am impressed. I wasn't expecting this from smaller AI models.

The response was super fast, with 99.16 tokens per second. I am impressed by the speed.

Using Llama 3.2 Vision Model with Groq API

We will now access the Llama 3.2 vision model using the Groq API. Currently, there is no GGUF file available for the vision model, so we have to access the vision model using the remote AI model provider.

Create the account on GroqCloud and generate the API key.
Go to the Msty settings and click on the “Remote Model Providers” tab, click on the “+ Add New Provider” button
Select the model provider as "Groq AI", then paste the API key and click on the "+ Add Custom Model" button. Next, type the model name as "llama-3.2-11b-vision-preview".

Go to the chat menu and select the Groq AI Llama 3.2 vision model.
Add the image of your choice and then type the prompt in the chat box.

Prompt: "Explain the image in detail."

The Llama 3.2 vision model is quite accurate in describing the image.

You can try out all the Groq AI models by adding them individually using the “+ Add Custom Model” button. Details on the models are available on GroqCloud.

Conclusion

Using open-source and closed-source chatbot desktop applications with large language models locally has become quite easy and feasible. Even without an internet connection, you can still use these models to generate code, debug code, or solve any kind of issue.

In this short tutorial, we have learned about the Msty desktop application and how to use it to access local and remote Llama 3.2 models. Please let me know in the comments if you are facing any issues running these models.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.