When ChatGPT took part in China's college entrance examination and fed it the national A and B papers, it turned out to be seriously biased!

Xi Xiaoyao Technology Talk Original
Author | PythonChatGPT, as an intelligent human-machine conversation application, quickly became popular all over the world after its launch. In just one month, its number of users has exceeded 100 million. People also use ChatGPT to test many exam items, such as SAT, AP, GRE, etc. However, what would it be like if ChatGPT were allowed to participate in China’s college entrance examination? Will he be biased? Can ordinary people like us pass ChatGPT? Let’s take a look at the comments brought to you by students from Fudan University and East China Normal University.

Paper title:
Evaluating the Performance of Large Language Models on GAOKAO Benchmark
Paper link://m.sbmmt.com/link/de66dbc30377a1f7908db45298324aa3

How to use ChatGPT to answer college entrance examination questions?

This paper uses a zero-supervised prompt method to convert test questions into ChatGPT input, as shown in the figure below. Different inquiry methods are designed for different subjects and question types. For math questions, convert the formula into latex input.

College Entrance Examination Data Set

The test in this article uses the national A paper and the national B paper from 2010 to 2022, a total of 13 years. Each set of test papers contains 10 subjects Subjects include Chinese, Mathematics, English, Physics, Chemistry, Life, History, and Geography. Mathematics is divided into science mathematics and liberal arts mathematics.

The data set contains a total of 2811 test questions. The specific question types will not be discussed here. I believe readers are very familiar with the college entrance examination questions.

During the evaluation, high school teachers from Caoyang No. 2 Middle School in Shanghai were hired to mark the subjective questions.

Experiment and Analysis

The scores obtained by ChatGPT in the college entrance examination over the past years are shown in the figure below. Since each subject is normalized to 100 points when calculating scores, this score cannot be directly compared with your and my college entrance examination scores. But it can also be seen that this score is not ideal, and it is estimated that neither Fudan nor East China Normal University will be admitted. Why is this?

The above picture shows the performance of ChatGPT in various disciplines, as well as subjective and objective topics. Blue is objective questions and yellow is subjective questions. The analysis found that ChatGPT performed better on objective questions, especially English reading comprehension, multiple choice, and cloze, achieving 88.3%, 78.1%, and 73.8% accuracy respectively. But even for objective questions, the accuracy rate in science and mathematics is less than 40%. Mathematics is really difficult~

ChatGPT’s performance on subjective questions is poor. In the subjects of physics, chemistry, biology and mathematics, the performance of subjective questions is obviously not as good as objective questions. The scores on objective questions in science subjects are also poor. Maybe ChatGPT is biased towards liberal arts subjects? According to the reviewers' comments, the main shortcomings of ChatGPT are: 1. The complex equations in mathematical problems are difficult to solve correctly, and wrong formulas are used in the problem solving process. 2. Inadequate ability to understand and summarize when reading longer materials.

Summary

ChatGPT may not have used Chinese college entrance examination question data during the training process, so its performance is not affected by data leaks and has high credibility.

Observation results show that compared with foreign exams, ChatGPT’s performance on Chinese college entrance examination questions is slightly inferior. Therefore, domestic students do not need to worry too much about not being able to pass the ChatGPT exam for the time being. However, the long text summarization capabilities mentioned in the article have been significantly improved in GPT4-32K, and the domestic large model has also been further optimized on Chinese data. Therefore, we can expect more impressive performance on large-model college entrance examination questions in the future.

In addition, the idea of ​​using ChatGPT to solve college entrance examination questions may be able to answer the debate among netizens about which province’s test questions are more difficult?

The above is the detailed content of When ChatGPT took part in China's college entrance examination and fed it the national A and B papers, it turned out to be seriously biased!. For more information, please follow other related articles on the PHP Chinese website!