How It Works
Neuronpedia is an open-source platform designed to facilitate mechanistic interpretability of neural networks, particularly focusing on sparse autoencoder (SAE) features and neurons. It provides a centralized hub for exploring, visualizing, and analyzing model internals through a combination of hosted feature dashboards, steering capabilities, and search functionalities across multiple models and SAEs. Users can interact with the platform via a web interface, API, or local development environments, making it accessible for both researchers and developers.
The core functionality of Neuronpedia revolves around its ability to host and query vast datasets of activations, explanations, and metadata. These datasets span over five terabytes and include probes, latent features, custom vectors, and concepts. The platform leverages PostgreSQL 16 with the pgvector extension to enable semantic search across explanations, allowing users to query by meaning rather than just keywords. This is particularly useful for identifying functionally important features within models, such as those identified through sparse dictionary learning techniques.
Neuronpedia’s architecture is divided into several key services, each serving a specific purpose:
- Webapp (Frontend + API): Handles user interactions, visualization, and API requests. It supports fast reloads during development and optimized builds for production. The webapp connects to a local or remote database to fetch and display data.
- Inference Server: Enables local model inference for tasks like steering, activation testing, and Jacobian lens (jlens) analysis. Users can configure this server to run specific models and source/SAE sets, with pre-loaded configurations available for models like GPT-2 Small, Gemma 2, and Llama 3.1. The server dynamically selects backend systems (e.g., vLLM for CUDA-supported architectures or eager PyTorch) and reuses weights from the Hugging Face cache.
- Database: Stores activations, explanations, and metadata. Users can import custom datasets or use Neuronpedia’s hosted data. The database is initialized via PostgreSQL and requires manual seeding with data through the admin panel.
Each service can be developed, tested, and deployed independently, with detailed documentation provided for setting up local environments. For example, users can configure a local database, import data, and run inference servers for specific models without relying on cloud-based resources.
Neuronpedia’s design emphasizes standardization and ease of use, aiming to lower the barrier for researchers to explore and interpret neural networks. By providing tools for steering model activations, searching latent spaces, and visualizing internal circuits, it enables deeper investigations into how models process information. The platform’s open-source nature, combined with its permissive Apache 2.0 license, encourages collaboration and adaptation by the broader AI community.
Use Cases
Neuronpedia is primarily used for mechanistic interpretability, a subfield of AI research focused on understanding the internal workings of neural networks. Its tools and datasets support a variety of applications, including the analysis of sparse autoencoders, circuit tracing, and activation steering. Below are some key use cases enabled by the platform:
Exploring Sparse Autoencoder (SAE) Features
Sparse autoencoders are a critical tool in mechanistic interpretability, as they decompose model activations into interpretable features. Neuronpedia hosts pre-trained SAEs for models like GPT-2 Small, Llama 3.1, and Gemma 2, allowing researchers to explore these features through interactive dashboards. Users can examine individual neurons or features, view their top activations, and test their influence on model outputs. For instance, a researcher might investigate how a specific SAE feature correlates with particular linguistic patterns or reasoning steps within a model.
The platform also supports custom SAEs, enabling users to upload and analyze their own models or feature dictionaries. The ability to export and share dashboards further facilitates collaboration, as users can create sharable links or embed dashboards in reports or presentations.
Circuit Tracing and Internal Reasoning
Circuit tracing is a technique used to map out the internal reasoning steps of a model, identifying how information flows through its layers. Neuronpedia provides tools to visualize these circuits, including attention patterns and weight-sparse transformers, which are essential for understanding complex decision-making processes. For example, researchers can trace how a model processes a prompt step-by-step, identifying which neurons or features contribute to specific outputs or behaviors.
The platform’s Circuit Tracer feature, inspired by work from Anthropic, allows users to input custom prompts and observe the model’s internal activations in real-time. By combining circuit tracing with other interpretability tools, such as Jacobian lens (jlens) or steering, researchers can gain a more comprehensive understanding of how models generate responses.
Steering Model Activations
Steering refers to the process of modifying a model’s behavior by directly manipulating its internal activations. Neuronpedia provides robust steering capabilities, allowing users to adjust latent features or custom vectors to achieve desired outputs. This is useful for testing hypotheses about model functionality or for developing interventions that alter model behavior in controlled ways.
For example, a researcher might steer a model’s activations to increase the likelihood of generating a specific type of response, such as a more detailed explanation or a particular tone. The platform supports customizable parameters like temperature, strength, and seed, enabling fine-grained control over steering experiments. Steering can also be applied to instruct or reasoning models, making it a versatile tool for both technical and applied research.
Searching and Benchmarking Features
Neuronpedia’s search functionality allows users to query over 50 million latents or vectors, either by semantic similarity to explanation text or by running custom text through a model to find matches. This is particularly powerful for identifying features associated with specific concepts, such as technical terms, cultural references, or domain-specific knowledge. The platform’s semantic search capabilities, powered by pgvector, enable users to find relevant features even when exact matches are unavailable.
API and Programmatic Access
Neuronpedia’s API is one of the first of its kind in the interpretability space, offering programmatic access to all core functionalities. Developers can integrate Neuronpedia’s tools into custom workflows, automate experiments, or build specialized applications. The API supports OpenAPI specifications, providing interactive documentation and client libraries for Python and TypeScript. This makes it easier for researchers to incorporate interpretability tools into their existing pipelines or to develop new tools that build on Neuronpedia’s infrastructure.
For example, a researcher might use the API to batch-process activations, generate explanations for large datasets, or deploy custom dashboards for specific use cases. The ability to embed dashboards via IFrame further extends the platform’s utility, allowing users to share interactive visualizations without requiring others to access the full Neuronpedia interface.
Key Features & Capabilities
Neuronpedia’s feature set is designed to address the needs of researchers and developers working on mechanistic interpretability. Below are some of its most notable capabilities:
Hosted Feature Dashboards
Neuronpedia provides interactive dashboards for exploring individual features, neurons, or concepts within models. Each dashboard includes detailed information such as top activations, top logits, activation density plots, and live inference testing. Users can navigate these dashboards to understand how specific features contribute to model outputs or behaviors.
Dashboards are uniquely linked and can be compiled into sharable lists, making it easy to organize and present findings. The platform also supports IFrame embedding, allowing users to integrate dashboards into external websites or documentation.
Steering and Activation Manipulation
Steering is a core capability of Neuronpedia, enabling users to modify model activations to achieve specific outcomes. The platform supports both latent-based steering (using SAE features) and vector-based steering (using custom embeddings). Users can adjust parameters like temperature, strength, and seed to fine-tune the steering effect, making it possible to test hypotheses or explore edge cases in model behavior.
The platform’s steering tools are compatible with a wide range of models, including instruct and reasoning models, and can be applied during both inference and fine-tuning phases.
Search Across Millions of Latents
Neuronpedia’s search functionality allows users to query over 50 million latents or vectors, either by semantic similarity or through custom inference. The platform uses pgvector to enable efficient semantic search, which is particularly useful for identifying features associated with abstract or context-dependent concepts. For example, a user might search for features related to ‘legal reasoning’ or ‘technical documentation’ and retrieve the most relevant matches based on their semantic content.
Search results can be filtered and refined using additional metadata, such as model architecture, layer, or source set. The ability to run custom text through a model to find matching latents further enhances the platform’s utility, as it allows users to explore how models encode specific ideas or patterns.
Autointerp and Automated Explanations
Autointerp is a feature of Neuronpedia that automates the generation of explanations for model activations. Using techniques like sparse dictionary learning, the platform can identify functionally important features and generate interpretable descriptions of their roles. This reduces the manual effort required to analyze large-scale models, making it easier for researchers to scale their interpretability work.
Autointerp explanations can be generated for both pre-trained SAEs and custom models, and they are integrated into Neuronpedia’s dashboards for easy exploration. Users can also export these explanations for further analysis or sharing. The platform’s ability to handle high-volume explanation generation makes it particularly valuable for large-scale interpretability projects.
OpenAPI and Programmatic Access
Neuronpedia’s API is built on OpenAPI specifications, providing comprehensive documentation and client libraries for Python and TypeScript. This makes it easier for developers to integrate the platform’s tools into custom workflows or to build specialized applications. The API supports all core functionalities, including steering, search, and dashboard generation, ensuring that users can automate their interpretability workflows.
For example, a researcher might use the API to batch-process activations for a large dataset, generate explanations for multiple models, or deploy custom dashboards for specific use cases. The ability to interact with the API programmatically also enables the development of new tools or extensions that build on Neuronpedia’s infrastructure.
Local Development and Customization
Neuronpedia is designed to be fully customizable and supports extensive local development. Users can set up their own instances of the platform, including the webapp, inference server, and database, to work with custom models or datasets. Detailed documentation is provided for each service, including steps for configuring a local database, running inference servers, and developing custom dashboards.
For example, a user might configure a local inference server to run a specific model and SAE set, then connect it to their own database to explore activations offline. This level of customization is particularly valuable for researchers working with proprietary or sensitive data, as it allows them to maintain full control over their workflows.
The platform also supports the generation of custom inference configurations, enabling users to load models not officially supported by TransformerLens or other interpretability frameworks. This flexibility ensures that Neuronpedia can accommodate a wide range of research needs, from standard models to specialized or fine-tuned architectures.