How ChainForge Works
ChainForge is a visual programming environment designed for prompt engineering and LLM evaluation, enabling users to test and compare prompts, models, and responses without extensive coding. It operates as a data flow system where users can define prompts, input variables, and evaluation metrics, then visualize results across permutations of prompts, models, and settings. The platform supports both browser-based and local installations, with the latter offering additional features such as API key management via environment variables and the ability to query locally hosted models through Ollama.
At its core, ChainForge uses a node-based interface built on ReactFlow and Flask. Users construct workflows by connecting nodes—such as prompt templates, model queries, and evaluators—to create evaluation pipelines. These pipelines can include RAG (Retrieval-Augmented Generation) components, allowing users to upload documents, chunk and retrieve information, and rerank results, either in the browser or locally with advanced RAG tools. The system also supports in-browser LLMs, eliminating the need for API keys for small models like Gemma 3 1B or Llama 3.2 1B.
ChainForge’s architecture emphasizes security and isolation. When running locally, the server restricts requests to localhost or explicitly allowed hosts, preventing unauthorized access. API keys are managed securely, either through environment variables or encrypted settings files, ensuring sensitive credentials are not exposed.
Use Cases
ChainForge is primarily used for prompt robustness testing, model comparison studies, and red-teaming exercises. Researchers and developers can systematically evaluate how variations in prompts, model parameters, or input data affect LLM outputs. For example, users can test a prompt template across multiple models—such as GPT-4, Claude, or Gemini—while adjusting settings like temperature or system messages to identify optimal configurations. This is particularly useful in scenarios requiring high reliability, such as automated reasoning, code generation, or compliance-sensitive applications.
Another key use case is ground truth evaluation, where ChainForge imports structured datasets (e.g., JSONL, CSV, or XLSX) to compare LLM responses against expected outputs. Users can define evaluation criteria using Python scripts or LLM-based scorers, then visualize results in tables, grids, or heatmaps. For instance, a user might evaluate an LLM’s ability to solve math problems by importing a dataset of questions and answers, then scoring responses against the ground truth. ChainForge’s combinatorial power allows users to generate all permutations of input variables, ensuring comprehensive testing.
With nodes for uploading documents, chunking, retrieval, and reranking, users can test how different retrieval strategies (e.g., BM25 vs. semantic search) impact response quality. Locally installed versions extend these capabilities with server-side embeddings, LanceDB vector stores, and advanced rerankers.
For collaborative work, ChainForge’s Share feature allows users to generate unique links to their experiments, enabling others to view or replicate workflows without local setup. Users are encouraged to export important workflows as .cforge or .cfzip bundles for long-term preservation.
Prompt and Model Comparison
ChainForge’s primary strength lies in its ability to compare prompts, models, and settings systematically. The platform then generates all permutations of these inputs, querying selected LLMs simultaneously. Results are displayed in interactive tables or visualizations, such as heatmaps, to highlight performance differences.
Evaluation Nodes
ChainForge provides multiple ways to evaluate LLM responses. Users can write custom Python or JavaScript evaluators to score responses based on specific criteria, such as response length, factual accuracy, or adherence to instructions. Alternatively, they can use LLM Scorers, which prompt a model (e.g., GPT-4) to evaluate responses against predefined criteria. For simpler checks, ChainForge offers boolean evaluators (e.g., contains, starts with) without requiring code. Multi-Eval nodes combine multiple evaluators, allowing users to assess responses across diverse metrics.
RAG Integration
ChainForge includes RAG-specific nodes for document upload, chunking, retrieval, and reranking. The browser-based version supports lightweight RAG workflows using small models for BM25 search and semantic retrieval, while local installations enable advanced features like LanceDB vector stores and TF-IDF chunking. Users can chain RAG nodes with prompt nodes to test how retrieval-augmented responses compare to vanilla prompts.
Chat and Prompt Chaining
ChainForge supports multi-turn conversations through Chat Turn nodes, which extend prompt chaining to include chat history. Users can continue conversations across multiple LLMs, switch models mid-conversation, or evaluate chat responses using the same evaluators as single-turn prompts. This is useful for testing dialogue systems, where context and follow-up questions are critical. Chat Turn nodes also support prompt templating, allowing users to define dynamic follow-up messages based on previous responses.
Supported Providers and Models
ChainForge integrates with a wide range of LLM providers, including OpenAI, Anthropic, Google Gemini, DeepSeek, HuggingFace, Together.ai, Ollama, and OpenRouter. OpenRouter is notable for supporting multiple providers (e.g., Claude, GPT, Gemini) with a single API key. The platform also supports in-browser LLMs, such as Gemma 3 1B and Llama 3.2 1B, which run without requiring API keys. Users can also route requests through OpenAI-compatible gateways, such as Tuning Engines, for governed inference.
Security and Local Customization
Security is a priority in ChainForge, particularly when running locally. The server enforces host restrictions, allowing requests only from localhost or explicitly configured hosts.
Example Experiments and Workflows
ChainForge includes 188 pre-built example flows, derived from benchmarks like OpenAI evaluations, to demonstrate common use cases. These range from simple prompt comparisons to complex RAG pipelines. For instance, one example evaluates math problem-solving across models by importing a dataset of questions and answers, then scoring responses against the ground truth. Another compares response lengths across models and prompt variations.
Performance and Scalability
The platform caches responses to avoid redundant API calls, and its asynchronous request handling ensures queries are sent within rate limits. The Response Inspector supports thousands of responses, with grid and table views that filter and sort results dynamically.