Human beings do not have enough high-quality corpus for AI to learn, and they will be exhausted in 2026. Netizens: A large-scale human text generation project has been launched!

AI’s appetite is too big, and human corpus data is no longer enough.

A new paper from the Epoch team shows that AI will use up all high-quality corpus in less than 5 years.

You must know that this is a predicted result taking into account the growth rate of human language data. In other words, the number of newly written papers and newly edited books by humans in recent years Even if all the codes are fed to AI, it will not be enough.

If this development continues, large language models that rely on high-quality data to improve their level will soon face a bottleneck.

Some netizens can’t sit still:

This is ridiculous. Humans can train themselves effectively without reading everything on the Internet.

We need better models, not more data.

Some netizens ridiculed that it is better to let the AI ​​eat its own vomit:

You can generate the AI ​​itself Text is fed to AI as low-quality data.

#Let’s take a look, how much data is left by humans?

How about "inventory" of text and image data?

The paper mainly predicts text and image data.

The first is text data.

The quality of data usually ranges from good to bad. The authors divided the available text data into low-quality and high-quality parts based on the data types used by existing large models and other data.

High-quality corpus refers to the training data sets used by large language models such as Pile, PaLM and MassiveText, including Wikipedia, news, code on GitHub, published books, etc.

Low-quality corpus comes from tweets on social media such as Reddit, as well as unofficial fan fiction (fanfic).

According to statistics, there are only about 4.6×10^12~1.7×10^13 words left in the high-quality language data stock, which is less than an order of magnitude larger than the current largest text data set.

Combined with the growth rate, the paper predicts that high-quality text data will be exhausted by AI between 2023 and 2027, and the estimated node is around 2026.

It seems a bit fast...

Of course, low-quality text data can be added to the rescue. According to statistics, there are currently 7×10^13~7×10^16 words left in the overall stock of text data, which is 1.5~4.5 orders of magnitude larger than the largest data set.

If the requirements for data quality are not high, then AI will use up all text data between 2030 and 2050.

Looking at the image data again, the paper here does not differentiate between image quality.

The largest image data set currently has 3×10^9 pictures.

According to statistics, the current total number of images is about 8.11×10^12~2.3×10^13, which is 3~4 orders of magnitude larger than the largest image data set.

The paper predicts that AI will run out of these images between 2030 and 2070.

Obviously, large language models face a more serious "lack of data" situation than image models.

So how is this conclusion drawn?

Calculate the average daily number of posts posted by netizens and get the result

The paper analyzes the text image data generation efficiency and the growth of the training data set from two perspectives.

It is worth noting that the statistics in the paper are not all labeled data. Considering that unsupervised learning is relatively popular, unlabeled data is also included.

Take text data as an example. Most of the data will be generated from social platforms, blogs and forums.

To estimate text data generation speed, there are three factors to consider, namely the total population, Internet penetration rate, and the average amount of data generated by Internet users.

For example, this is the estimated future population and Internet user growth trend based on historical population data and the number of Internet users:

combined with user-generated By averaging the amount of data, the rate at which data is generated can be calculated. (Due to complex geographical and time changes, the paper simplifies the calculation method of the average amount of data generated by users)

According to this method, the growth rate of language data is calculated to be around 7%. However, this growth rate will increase with Gradually decreases over time.

It is expected that by 2100, the growth rate of our language data will drop to 1%.

A similar method is used to analyze image data. The current growth rate is about 8%. However, by 2100, the growth rate of image data will also slow down to about 1%.

The paper believes that if the data growth rate does not increase significantly, or new data sources emerge, whether it is an image or a large text model trained with high-quality data, it may usher in a bottleneck period at a certain stage.

Some netizens ridiculed that something like a sci-fi storyline might happen in the future:

In order to train AI, humans have started large-scale text generation projects, and everyone is writing hard for AI.

He calls it an “education for AI”:

We send 140,000 to 2.6 million words to AI every year Amounts of text data, sounds cooler than using humans as batteries?

What do you think?

Paper address: https://arxiv.org/abs/2211.04325

Reference link: https://twitter.com/emollick/status/1605756428941246466

The above is the detailed content of Human beings do not have enough high-quality corpus for AI to learn, and they will be exhausted in 2026. Netizens: A large-scale human text generation project has been launched!. For more information, please follow other related articles on the PHP Chinese website!