Based on LLaMA but changing the tensor name, Kai-Fu Lee's large model caused controversy, and the official response is here

Some time ago, the field of open source large models ushered in a new model - the context window size exceeded 200k and can handle "Yi" of 400,000 Chinese characters at a time.

Kai-fu Lee, Chairman and CEO of Innovation Works, founded the large model company "Zero One Thousand Things" and built this large model, including Yi-6B and Yi-34B. Version

According to Hugging Face English open source community platform and C-Eval Chinese evaluation list, Yi-34B achieved a number of SOTA international best performance indicator recognitions when it was launched, becoming a global open source giant. The model is a "double champion", defeating open source competing products such as LLaMA2 and Falcon.

Yi-34B also became the only domestic model to successfully top the Hugging Face global open source model rankings at that time. , called "the world's strongest open source model."

After its release, this model attracted the attention of many domestic and foreign researchers and developers

But recently, some researchers discovered that, The Yi-34B model basically adopts the LLaMA architecture, except that two tensors are renamed.

Please click this link to view the original post: https://news.ycombinator.com/item?id=38258015

The post also mentioned:

The code of Yi-34B is actually a reconstruction of the LLaMA code, but it does not seem to have made any substantial changes. This model is obviously an edit based on the original Apache version 2.0 LLaMA file, but makes no mention of LLaMA:

Yi vs LLaMA Code comparison. Code link: https://www.diffchecker.com/bJTqkvmQ/

In addition, these code changes are not submitted to the transformers project through Pull Request. , but instead attach it as external code, which may pose security risks or be unsupported by the framework. The HuggingFace leaderboard won't even benchmark this model with a context window up to 200K because it doesn't have a custom code strategy.

They claim this is a 32K model, but it is configured as a 4K model, there is no RoPE scaling configuration, and there is no explanation of how to scale (note: zero one thing before means that the model itself is on the 4K sequence for training, but can scale to 32K during inference phase). Currently, there is zero information about its fine-tuning data. They also did not provide instructions for reproducing their benchmarks, including the suspiciously high MMLU scores.

Anyone who has worked in the field of artificial intelligence for a while will not turn a blind eye to this. Is this false advertising? License violation? Was it actually cheating on the benchmark? Who cares? We could change a paper, or in this case, take all the venture capital money. At least Yi is above the standard because it is a basic model and its performance is really good

A few days ago, in the Huggingface community, a developer also pointed out:

According to our understanding, except for renaming two tensors, Yi completely adopts the LLaMA architecture. (input_layernorm, post_attention_layernorm)

#In the discussion, some netizens said: If they want to use Meta LLaMA’s architecture, code base and other related resources exactly, Must abide by the license agreement stipulated by LLaMA

In order to comply with LLaMA's open source license, a developer decided to change his name back and republish it On huggingface

01-ai/Yi-34B, the tensors have been renamed to match the standard LLaMA model code. Related links: https://huggingface.co/chargoddard/Yi-34B-LLaMA

By reading this content, we can infer that the news that Jia Yangqing left Alibaba and started a business was mentioned in his circle of friends a few days ago

Regarding this matter, the Heart of the Machine also sought verification from Zero One and All Things. Lingyiwu responded:

GPT is a mature architecture recognized in the industry, and LLaMA made a summary on GPT. The structural design of the large R&D model of Zero One Thousand Things is based on the mature structure of GPT, drawing on top-level public results in the industry. At the same time, the Zero One Thousand Things team has done a lot of work on the understanding of the model and training. This is the first time we have released excellent results. one of the foundations. At the same time, Zero One Thousand Things is also continuing to explore essential breakthroughs at the model structure level.

The model structure is only part of the model training. Yi's open source model focuses on other aspects, such as data engineering, training methods, baby sitting (training process monitoring) skills, hyperparameter settings, evaluation methods, depth of understanding of the nature of evaluation indicators, and depth of research on the principles of model generalization capabilities. , the industry's top AI Infra capabilities, etc., a lot of research and development and foundation work have been invested. These tasks often play a greater role and value than the basic structure. These are also the core technologies of Zero One Wagon in the large model pre-training stage. moat.

In the process of conducting a large number of training experiments, we renamed the code according to the needs of experimental execution. We attach great importance to the feedback from the open source community and have updated the code to better integrate into the Transformer ecosystem

We are very grateful for the feedback from the community. We have just started in the open source community and hope to work with everyone to create a community. Prosperity, Yi Kaiyuan will do its best to continue to make progress

The above is the detailed content of Based on LLaMA but changing the tensor name, Kai-Fu Lee's large model caused controversy, and the official response is here. For more information, please follow other related articles on the PHP Chinese website!