Big Data Digest Produced
"We are very short of GPUs"
In a recent interview, Sam Altman, the head of OpenAI, responded to the host's question about "the dissatisfaction is the API" reliability and speed.”
This interview comes from Raza Habib, CEO of the artificial intelligence startup Humanloop. He compiled the highlights of the interview on Twitter.
Twitter address:
https://twitter.com/dr_cintas/status/1664281914948337664
## In this interview, Altman also announced GPT plans for the past two years. For example, the 2023 plan is to reduce the cost of GPT-4 and improve response speed. Others include:
1. Longer The context window may support 1 million tokens;
2. Fine-tune the API to help developers develop better;
3. Support session state API, that is, API that supports session state.
The 2024 plan mentioned that GPT-4 should support multi-modality. The reason why it was postponed to 2024 is because there is too little GPU.
#In the interview, Altman also mentioned that they have been considering whether to open source GPT-3. There is no doubt that open source is very important. At the same time, he also expressed that current AI models are not that dangerous. Although it is very important to regulate future models, banning development is a very big misconception.
Raza Habib originally published more detailed information about this interview to Humanloop, but when Digest checked this morning, the page was already 404. According to the translation of @宝玉xp on domestic Weibo, he also mentioned the scaling rules for the future development of large models:
OpenAI’s internal data shows that the scaling rules for model performance continue to be effective, making the model Bigger will continue to yield performance. OpenAI has scaled its models millions of times, so it cannot continue to scale at a pace that will be unsustainable in the future. This doesn't mean that OpenAI won't continue to try to make models bigger, it just means that they will probably only double or triple every year instead of increasing by many orders of magnitude.
The fact that scaling continues to be effective has important implications for the timeline of AGI development. The scaling assumption is that we probably already have most of the parts needed to build an AGI, and most of the remaining work will be scaling existing methods to larger models and larger data sets. If the era of scaling is over, then we should probably expect AGI to be even further away. That the Law of Scaling continues to be in effect strongly hints at a shorter timeline.
Obviously, the scaling law is the fastest path to AGI.
What is the scaling rule?
Scaling Laws, the English name is Scaling Laws, is a description of the phenomenon, which generally refers to: the effect of the language model and the amount of parameters, amount of data, The amount of calculation basically follows a smooth power law.
In other words, as the number of parameters of the model (Parameters), the amount of data involved in training (Tokens), and the amount of calculation accumulated in the training process (FLOPS) increase exponentially, the performance of the model on the test set increases exponentially. Loss decreases linearly, which means the model performs better.
Legend: Empirical performance exhibits a power-law relationship with each individual factor when not constrained by the other two factors.
In 2022, DeepMind conducted further analysis in ScalingLaw. The research has verified through quantitative experiments that the size of the language model training data should be enlarged in proportion to the size of the model parameters. When the total amount of calculation remains unchanged, the effect of model training has an optimal balance point between the parameter size and the training data amount. The lowest point under the curve is a very good compromise between the parameter size and the training data amount. point.
OpeaAI’s Success and GPT-4
OpenAI began as a non-profit artificial intelligence research lab that acquired Sam Altman and Elon Musk in 201610 billion in funding.
In 2019 OpenAI transformed into a for-profit artificial intelligence research laboratory to absorb investors' funds.
When the laboratory is running out of funds to support its research, Microsoft announced that it will invest another US$1 billion in the laboratory.
Every version of the GPT series launched by OpenAI can cause carnival in the industry. At the Microsoft Build 2023 Developer Conference, OpenAI founder Andrej Karpthy gave a speech: State of GPT (the current status of GPT), saying They have been training large models as "human brains".
Andrej mentioned that the current LLM large language model can be compared to System 1 (fast system) of human thinking mode, which is compared to System 2 (slow system) that responds slowly but has longer-term reasoning. .
“System one is a fast automatic process that I think kind of corresponds to LLM, just sampling tokens.
System two is the slower, well-thought-out planning part of the brain.
The prompt project basically hopes to restore some of the capabilities of our brains to LLM."
Andrej Karpthy also mentioned that GPT-4 is an amazing artifact, and he is very Be grateful that it exists. It has a ton of knowledge in many areas, it can do math, code, and more, all at your fingertips.
CEO Altman said that in the early days, GPT-4 was very slow, had bugs, and did many things poorly. But so did the earliest computers, which still pointed the way to something that would become very important in our lives, even if it took decades to develop.
It seems that OpenAI is an organization that insists on dreams and wants to do things to the extreme.
As Zhou Ming, former vice president of Microsoft Research Asia and founder of Lanzhou Technology, mentioned in an interview:
OpenAI’s greatest achievement is to achieve perfection in all aspects and is a model of integrated innovation.
There are several types of people in the world. Some people just want to study underlying innovation. Some are applications based on underlying innovations, while general applications are to solve a single task. It can be rewritten as: Another approach is to achieve integrated innovation, concentrating all work, applications and algorithms on a large platform to create milestones. OpenAI happens to do a really good job of integrating innovation.
Reference:
https://mp.weixin.qq.com/s/p42pBVyjZws8XsstDoR2Jw https://mp.weixin.qq.com/s/zmEGzm1cdXupNoqZ65h7yg https://weibo.com/1727858283/4907695679472174?wm=3333_2001&from=10D5293010&sourcetype=weixin&s_trans=6289897940_490769 5679472174&s_channel=4 https://humanloop.com/blog/openai- plans?cnotallow=bd9e76a5f41a6d847de52fa275480e22
The above is the detailed content of Latest interview with OpenAI founder Sam Altman: GPT-3 or open source, scaling rules accelerate the construction of AGI. For more information, please follow other related articles on the PHP Chinese website!
