Step-by-step guide to using Groq Llama 3 70B locally

Translator| Bugatti

Reviser| Chonglou

This article describes how to use the Groq LPU inference engine to generate ultra-fast responses in Jan AI and VSCode.

Everyone is working on building better large language models (LLM), such as Groq focusing on the infrastructure side of AI. Rapid response from these large models is key to ensuring that these large models respond more quickly.

This tutorial will introduce the Groq LPU parsing engine and how to access it locally on your laptop using the API and Jan AI. This article will also integrate it into VSCode to help us generate code, refactor code, enter documentation and generate test units. This article will create our own artificial intelligence programming assistant for free.

Introduction to the Groq LPU Inference Engine

The Groq LPU (Language Processing Unit) inference engine is designed to generate fast responses for computationally intensive applications (such as LLMs) that have sequential components.

Compared with CPU and GPU, LPU has more powerful computing power, which reduces the time required to predict words and greatly speeds up the generation of text sequences. Furthermore, LPU can also handle memory bottlenecks compared to GPU, thus providing better performance on LLM.

In short, Groq LPU technology makes your LLM super fast for real-time AI applications. You may wish to read the Groq ISCA 2022 paper (https://wow.groq.com/isca-2022-paper/) to learn more about the LPU architecture.

Install Jan AI

Jan AI is a desktop application that runs open source and proprietary large language models natively. It is available for download in Linux, macOS and Windows versions. We will download Jan AI and install it in Windows. To do this, go to https://github.com/janhq/jan/releases and click on the file with the ".exe" extension.

If you want to use LLM locally for enhanced privacy, please read the "5 Ways to Use LLM on Your Laptop" blog post (https://www.kdnuggets. com/5-ways-to-use-llms-on-your-laptop) to start using the most advanced open source language models.

Create Groq Cloud API

To use Grog Llama 3 with Jan AI, we need an API. To do this, we will go to https://console.groq.com/ and create a Groq Cloud account.

If you want to test the various models provided by Groq, you can do this without any setup, just go to the Playground tab, select the model, and add user input.

In this example, it is very fast, generating 310 tokens per second, which is the fastest I have seen so far. Not even Azure AI or OpenAI can achieve this type of results.

To generate an API key, click the "API Key" button on the left panel, then click the "Create API Key" button to create and copy the API key .

Using Groq with Jan AI

In the next step, we will paste the Groq Cloud API key into the Jan AI application.

Launch the Jan AI application, go to settings, select the "Groq Inference Engine" option in the extension section, and add the API key.

Then, return to the thread window. In the Model section, select Groq Llama 370B in the "Remote" section to start typing the prompt.

The responses are generated so fast that I can’t keep up.

Note: The free version of this API has some limitations. Please visit https://console.groq.com/settings/limits to learn more about them.

Using Groq in VSCode

Next, we will try to paste the same API key into the CodeGPT VSCode extension and build our own free AI programming assistant.

Search for the CodeGPT extension in the extensions tab and install it.

#The CodeGPT tab will appear allowing you to select a model provider.

When you select Groq as the model provider, it will ask you for your API key. Just paste the same API key and we are good to go. You can even generate another API key for CodeGPT.

Now we will ask it to code the snake game. It only took 10 seconds to generate and run the code.

Below is a demonstration of our snake game.

You might as well learn about the top five AI programming assistants (https://www.kdnuggets.com/top-5-ai-coding-assistants-you-must-try) and become AI-driven developers and data scientists. Remember, AI is meant to help us, not replace us, so be open to it and use it to improve your coding.

Conclusion

In this tutorial we learned about the Groq inference engine and how to access it locally using the Jan AI Windows application. Finally, we integrated it into our workflow by using the CodeGPT VSCode extension, which is awesome. It generates responses in real-time for a better development experience.

Original title: Using Groq Llama 3 70B Locally: Step by Step Guide, author: Abid Ali Awan

Link: https://www.kdnuggets.com/using-groq-llama- 3-70b-locally-step-by-step-guide.

To learn more about AIGC, please visit:

51CTO AI.x Community

https://www.51cto.com/ aigc/