Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

For many years, we have been unable to understand how artificial intelligence makes decisions and produces output

Model developers can only decide on algorithms, data, and finally get the model Output results, and the middle part - how the model outputs results based on these algorithms and data, becomes an invisible "black box".

So there is a joke like "model training is like alchemy".

But now, the model black box is finally interpretable!

The research team from Anthropic extracted the interpretable features of the most basic unit neurons in the model’s neural network.

This will be a landmark step for mankind to uncover the black box of AI.

Anthropic expressed excitedly:

"If we can understand how the neural network in the model works, then we can diagnose the faults of the model. Patterns, design fixes, and safe adoption by businesses and society will become a within-reaching reality!"

at Anthropic In the latest research report "Towards Monosemanticity: Using Dictionary Learning to Decompose Language Models", researchers used dictionary learning methods to successfully decompose a layer containing 512 neurons into more than 4,000 interpretable features

Research report address: https://transformer-circuits.pub/2023/monosemantic-features/index.html

These features represent DNA sequences, legal language, HTTP requests, Hebrew text, and nutrition instructions, etc.

When we look at the activation of a single neuron in isolation, we It is impossible to see most of these model properties

Most neurons are "polysemantic", meaning that a single neuron is There is no consistent correspondence between network behaviors

For example, in a small language model, a single neuron is active in many unrelated contexts, including: academic citations, English Conversations, HTTP requests, and Korean text.

In the classic vision model, a single neuron responds to the face of a cat and the front of a car.

In different contexts, many studies have demonstrated that the activation of a neuron can have different meanings

One potential reason is that the polysemantic nature of neurons is due to the additive effect. This is a hypothetical phenomenon whereby neural networks represent independent features of data by assigning each feature its own linear combination of neurons, and the number of such features exceeds the number of neurons

If each feature is regarded as a vector on a neuron, then the feature set forms an overcomplete linear basis for the activation of network neurons.

In Anthropic’s previous Toy Models of Superposition paper, it was proved that sparsity can eliminate ambiguity in neural network training and help the model better understand features. relationship between them, thereby reducing the uncertainty of the source features of the activation vector and making the model’s predictions and decisions more reliable.

This concept is similar to the idea in compressed sensing, where the sparsity of the signal allows the complete signal to be restored from limited observations.

But among the three strategies proposed in Toy Models of Superposition:

# (1) Create a model without superposition, Perhaps activation sparsity can be encouraged;

(2) In models that exhibit superposition states, dictionary learning is used to find overcomplete features

(3) relies on a hybrid approach that combines the two.

What needs to be rewritten is: method (1) cannot solve the problem of ambiguity, while method (2) is prone to serious overfitting

So this time Anthropic researchers used a weak dictionary learning algorithm called a sparse autoencoder to generate learned features from the trained model that provide better performance than model neurons itself a more unitary unit of semantic analysis.

Specifically, the researchers adopted an MLP single-layer transformer with 512 neurons, and finally trained a sparse autoencoder on MLP activations from 8 billion data points. Decomposing MLP activations into relatively interpretable features, expansion factors range from 1× (512 features) to 256× (131,072 features).

To verify that the features found in this study are more interpretable than the model's neurons, we conducted a blind review and asked a human evaluator to evaluate their interpretability. Rating

As you can see, the feature (red) has a much higher score than the neuron (cyan).

It has been shown that the features discovered by the researchers are easier to understand relative to the neurons inside the model

Additionally, the researchers adopted an "automated interpretability" approach by using a large language model to generate a short description of a small model's features and having another model score that description based on its ability to predict feature activation. .

Likewise, features score higher than neurons, demonstrating a consistent interpretation of the activation of features and their downstream effects on model behavior.

Moreover, these extracted features also provide a targeted method to guide the model.

As shown in the figure below, artificially activating features can cause model behavior to change in predictable ways.

The following is a visualization of the extracted interpretability features:

Click on the feature list on the left to interactively explore the feature space in the neural network

Research Report Summary

This research report from Anthropic, Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, can be divided into four parts.

Problem setting, the researchers introduced the research motivation and elaborated on the trained transfomer and sparse autoencoder.

Detailed investigation of individual features proves that several features found in the study are functionally specific causal units.

Through global analysis, we conclude that the typical features are interpretable and they are able to explain important components of the MLP layer

Phenomenon analysis describes several properties of features, including feature segmentation, universality, and how they form systems similar to "finite state automata" to achieve complex behaviors.

The conclusions include the following 7:

Sparse autoencoder has the ability to extract relatively single semantic features

Sparse autoencoders are able to generate interpretable features that are actually invisible in the basis of neurons

3. Sparse Autoencoders Features can be used to intervene and guide the generation of transformers.

4. Sparse autoencoders can generate relatively general features.

As the size of the autoencoder increases, features tend to "split". After rewriting: As the size of the autoencoder increases, features show a trend of "splitting"

#6. Only 512 neurons can represent thousands of features

7. These features are connected together, similar to a "finite state automaton" system, to achieve complex behaviors, as shown in the figure below

Specific details can be found in the report.

Anthropic believes that to replicate the success of the small model in this research report to a larger model, the challenge we face in the future will no longer be a scientific problem, but an engineering problem

Achieving interpretability on large models requires more effort and resources in engineering to overcome the challenges posed by model complexity and size

Includes developing new tools, techniques and methods to cope with the challenges of model complexity and data scale; it also includes building scalable interpretive frameworks and tools to adapt to the needs of large-scale models.

This will become the latest trend in interpretive artificial intelligence and large-scale deep learning research

The above is the detailed content of Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier. For more information, please follow other related articles on the PHP Chinese website!