70B model generates 1,000 tokens in seconds, code rewriting surpasses GPT-4o, from the Cursor team, a code artifact invested by OpenAI

70B model, 1000 tokens can be generated in seconds, which translates into nearly 4000 characters!

The researchers fine-tuned Llama3 and introduced an acceleration algorithm. Compared with the native version, the speed is 13 times faster!

Not only is it fast, its performance on code rewriting tasks even surpasses GPT-4o.

This achievement comes from anysphere, the team behind the popular AI programming artifact Cursor, and OpenAI also participated in the investment.

You must know that on Groq, the famously fast inference acceleration framework, the inference speed of 70B Llama3 is only more than 300 tokens per second.

Cursor’s speed can be said to achieve near-instant complete code file editing.

Someone said, "Hey guys, if you put Llama3 after Cursor's magic modification on Groq, will you be able to generate tens of thousands of tokens per second?"

Some people are even more excited to say that in the field of large models, we are eliminating the concept of "delay".

Introducing a new inference acceleration algorithm

The acceleration method designed by the author this time is mainly used to solve a task called "Fast Apply". That is, quickly modify and apply the code content.

First of all, it needs to be explained that although the final effect of the task is a local modification of the code, during the actual operation, the output is not only the changed content, buta direct global reset. Write.

The reason for this is the choice made by the team after pre-testing - they found that, except for Claude-3-Opus, most models did not perform satisfactorily on the true partial modification task.

There are three main reasons why this is the case:

• Firstly, more tokens will be output when rewritten directly, allowing more forward passes to determine the correctness. s solution.
• Secondly, most of the model’s training data are complete codes, and are relatively unfamiliar with local modifications.
• Also, the poor math of large models does not guarantee that line numbers will be handled correctly when outputting differences.

(However, the author believes that this is still a potential future research direction.)

Determined to adopt the global After rewriting the plan, the Cursor team used task-related data to fine-tune Llama3.

The data used comes from two sources: real edited data and synthetic data, which are mixed at a ratio of 1:4.

Synthetic data refers to using GPT-4 to generate code editing suggestions, and then using other models to "apply" these suggestions to the original code.

In order to improve the quality of the data set, the author also downsampled small files, duplicate files, and unchanged samples.

To evaluate the performance of these models, the authors ran them through 450 code editing tasks (each of no more than 400 lines) and evaluated the output with Claude3-Opus. Score.

In the end, the performance of the 70B Llama3 model fine-tuned by the author almost matched that of Claude3-Opus-diff, and was better than GPT-4-Turbo and GPT-4o.

The fine-tuning so far has solved the performance problem, but it is not difficult to see that Llama3 is still very slow at this time, and can only output less than 300 characters per second( Note that it is a character, not a word or a token).

And what makes the rewriting work so fast is another secret weapon.

For the code rewriting task, the Cursor team specially introduced an algorithm calledPredictive editing(speculative edits).

This method uses an a priori algorithm to predict multiple subsequent tokens, and then uses the ontology large model for verification, which reduces the number of calls to the large model, thus reducing the amount of calculations.

This a priori algorithm comes from a feature of the coding task-compared to other texts, its vocabulary is smaller, and its grammatical structure, indentation rules, etc. have higher certainty. Using a priori knowledge can make it more precise. Accurately predict future tokens.

This approach also has something in common with GPT-4 and Meta-

The reason why traditional language model reasoning is slow is mainly because the process of predicting the next token is usually Autoregressive, that is, when the model generates each token, it must consider all previously generated tokens.

In order to reduce the amount of calculations, large models represented by GPT-4 use an acceleration algorithm calledPredictive decoding(speculative decoding), through small The approximate model makes predictions in advance, and then the ontology large model verifies the prediction results.

The difference between Cursor and GPT-4 is that the former’s small “model” is a more deterministic algorithm, while the latter only reduces the size of the model and is still essentially a probabilistic prediction.

Meta has introduced an algorithm for predicting multiple subsequent tokens at once, using n independent output heads to predict n future tokens in parallel, and found thatperforms well on programming tasks Especially excellent, because the logical structure of the programming language is more rigorous and the internal connection of knowledge is closer.

Of course, Cursor makes full use of this feature. Instead of using attention heads, it directly uses a more certain algorithm to make multi-token predictions.

The final result is that the prediction algorithm brings a nearly 13-fold speed increase to the 70B Llama3 without any loss in evaluation performance.

In addition, the author also cooperated with the enterprise AI model infrastructure platform fireworks.ai, using its optimized inference engine and customized hardware environment to further improve the operation of the model. efficiency.

In the future, the team also plans to conduct knowledge distillation and migrate the predictive editing algorithm to the smaller 8B Llama3, and expand it to more programming languages and tasks.

At the same time, the author also plans to improve the true partial modification (Diff) algorithm that the Cursor team has studied but has not adopted.

One More Thing

In the experiment, the author not only accelerated Llama3 using the prediction algorithm, but also accelerated GPT4-Turbo.

However, the author did not introduce how to implement it in GPT. Instead, he left some thinking questions and even held a "prize-winning guessing game".

Those who can answer correctly will get a 1-month Cursor membership; if they can achieve prediction acceleration in vllm and TensorRT-LLM, they will get a half-year and one-year membership respectively.

If you feel you have an idea, you might as well try the challenge (manual dog head).