OpenAI is open source: Transformer automatic debug tool is online on GitHub

OpenAI, which has often been criticized recently for not being open source enough, suddenly opened it up.

Early this morning, OpenAI machine learning researcher Jan Leike announced that OpenAI has opened up its own internal tools that have been used to analyze the internal structure of Transformer.

GitHub link: https://github.com/openai/transformer-debugger

The The project has only been open for a few hours. Although it has not received much publicity, the number of stars has increased very quickly.

Transformer Debugger Introduction

Transformer Debugger (TDB) is developed by OpenAI’s alignment team (Superalignment) A tool developed to help users examine the specific behavior of small language models. It is understood that the tool combines the characteristics of automatic interpretability technology and sparse autoencoders.

Specifically, TDB enables rapid exploration before code needs to be written, and can intervene in the forward pass to help one see how it affects a specific behavior of the model. TDB can be used to answer questions such as "Why does the model output token A instead of token B in this prompt (prompt)?" or "Why does the attention head H pay attention to token T in this prompt?"

The approach involves identifying specific components that contribute to behavior (e.g. neurons, attention heads, autoencoder latent variables) and analyzing what causes the strongest activation of these components by presenting automatically generated explanations , and trace the connections between components to help people discover connections and assist in debugging Transformer.

OpenAI has released several videos outlining the capabilities of TDB and showing how to use it for the paper "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small 》Work in:

This time, OpenAI’s open source content includes:

• Neuron viewer: a React application, using for hosting the TDB as well as pages containing information about the individual model components (MLP neurons, attention heads, and autoencoder latents for both).
• Activation server: A backend server that performs inference on the topic model and provides data for TDB. It also reads and serves data from public Azure storage buckets.
• Models: A simple inference library for GPT-2 models and their autoencoders, with hooks to capture activations.
• Cleaned activation datasets: Examples of top activation datasets containing MLP neurons, attention heads, and autoencoder latents.

Installation Settings

Please follow the steps below to install the repository. Please note that before this you need python/pip and node/npm.

Although not necessary, OpenAI recommends using a virtual environment for operations:

# If you're already in a venv, deactivate it.deactivate# Create a new venv.python -m venv ~/.virtualenvs/transformer-debugger# Activate the new venv.source ~/.virtualenvs/transformer-debugger/bin/activate

After setting up the environment, please Follow these steps:

git clone git@github.com:openai/transformer-debugger.gitcd transformer-debugger# Install neuron_explainerpip install -e .# Set up the pre-commit hooks.pre-commit install# Install neuron_viewer.cd neuron_viewernpm installcd ..

To run the TDB application, you need to follow the instructions to set up the activation server backend and neuron viewer frontend.

To verify the changes you need to:

• Run pytest;
• Run mypy --cnotallow=mypy.ini;
• Run the activation server and Neuron Viewer and confirm that basic functions such as TDB and Neuron Viewer pages still work.

Jan Leike said that TDB is still a research tool in the early stages. OpenAI hopes to benefit more people through open source and looks forward to the community continuing to build on it. Improve.