AI learns to hide its thinking and reason secretly! Solving complex tasks without relying on human experience is more black box

When AI does math problems, the real thinking is actually "mental arithmetic" secretly?

New research by a New York University team found that even if AI is not allowed to write steps and is replaced with meaningless "...", its performance on some complex tasks can be greatly improved!

First author Jacab Pfau said: As long as you spend computing power to generate additional tokens, you can bring advantages. It doesn’t matter what token you choose.

Picture

For example, let Llama 34M answer a simple question: How many of the first 6 digits of the natural constant e are greater than 5 ?

The AI's direct answer is equivalent to making trouble. It only counts the first 6 digits and actually counts 7.

Let AI write out the steps to verify each number, and you can get the correct answer.

Let AI hide the steps and replace them with a lot of "...", and you can still get the correct answer!

Picture

This paper sparked a lot of discussion as soon as it was released, and was evaluated as "the most metaphysical AI paper I have ever seen."

Picture

So, young people like to say more meaningless words such as "um...", "like...", is it okay? Strengthen reasoning skills?

Picture

From thinking "step by step" to thinking "little by little"

In fact, the New York University team The research starts from the Chain-of-Thought (CoT).

That’s the famous prompt “Let’s think step by step”.

Picture

In the past, it was found that using CoT inference can significantly improve the performance of large models on various benchmarks.

It’s unclear whether this performance improvement comes from imitating humans by breaking tasks into easier-to-solve steps, or whether it is a by-product of the extra calculations.

In order to verify this problem, the team designed two special tasks and corresponding synthetic data sets: 3SUM and 2SUM-Transform.

3SUM requires finding three numbers from a given set of number sequences so that the sum of the three numbers satisfies certain conditions, such as dividing by 10 with a remainder of 0.

Picture

The computational complexity of this task is O(n3), and the standard Transformer uses the input of the upper layer and the activation of the next layer Only secondary dependencies can occur between them.

That is to say, when n is large enough and the sequence is long enough, the 3SUM task exceeds the expression ability of Transformer.

In the training data set, "..." with the same length as the human reasoning steps is filled between the question and the answer. That is, the AI ​​has not seen how humans disassemble the problem during training.

Picture

In the experiment, the performance of Llama 34M that does not output the padding token "..." decreases as the sequence length increases, while the output When filling the token, 100% accuracy can be guaranteed until the length is 14.

Picture

2SUM-Transform only needs to determine whether the sum of two numbers meets the requirements, which is within the expressive capabilities of Transformer.

But at the end of the question, a step is added to "randomly replace each number of the input sequence" to prevent the model from directly calculating on the input token.

The results show that using padding tokens can increase the accuracy from 78.7% to 93.6%.

picture

In addition to the final accuracy, the author also studied the hidden layer representation of the filled token. Experiments show that by freezing the parameters of the previous layers and only fine-tuning the last Attention layer, the prediction accuracy increases as the number of available filling tokens increases.

This confirms that the hidden layer representation of the populated token does contain implicit computation related to downstream tasks.

Picture

Has AI learned to hide its thoughts?

Some netizens wonder, is this paper saying that the "thinking chain" method is actually fake? The prompt word project that I have been studying for so long has been in vain.

Picture

The team stated that theoretically the role of filling tokens is limited to the scope of TC0 complexity problems.

TC0 is a computational problem that can be solved by a fixed-depth circuit, in which each layer of the circuit can be processed in parallel and can be quickly solved by a few layers of logic gates (such as AND, OR and NOT gates) , which is also the upper limit of computational complexity that Transformer can handle in single forward propagation.

And a long enough thinking chain can extend the expression ability of Transformer beyond TC0.

And it is not easy for a large model to learn to use padding tokens, and specific intensive supervision needs to be provided to converge.

That said, existing large models are unlikely to benefit directly from the padding token method.

But this is not an inherent limitation of current architectures; if provided with sufficient demonstrations in the training data, they should be able to obtain similar benefits from padding symbols.

This research also raises a worrying issue: large models have the ability to perform secret calculations that cannot be monitored, posing new challenges to the explainability and controllability of AI.

In other words, AI can reason on its own in a form invisible to people without relying on human experience.

This is both exciting and terrifying.

Picture

Finally, some netizens jokingly suggested that Llama 3 first generate 1 quadrillion dots, so that the weight of AGI can be obtained (dog head) .

Picture

Paper://m.sbmmt.com/link/36157dc9be261fec78aeee1a94158c26

Reference Link:
[1]//m.sbmmt.com/link/e350113047e82ceecb455c33c21ef32a[2]//m.sbmmt.com/link/872de53a900f3250ae5649ea19e5c381