Kiln: A Workbench for AI Development, Evaluation, and Optimization
Kiln is a desktop application and Python library designed to streamline the entire AI development lifecycle. It integrates evaluation, optimization, prompt management, synthetic data generation, fine-tuning, and agentic workflows into a unified platform. Unlike traditional tools that force a choice between code-only frameworks or proprietary SaaS solutions, Kiln provides a local-first, open-source alternative that supports collaboration through Git-native dataset storage.
The tool is developed by Chesterfield Laboratories Inc. and operates under a custom or MIT license for its core components, ensuring flexibility for both individual and enterprise use. Kiln’s architecture allows teams to define a task once and apply it across multiple stages—such as evaluation, prompt optimization, fine-tuning, and RAG—while maintaining consistency and traceability. This approach minimizes redundancy and ensures that improvements compound across the development pipeline.
Kiln’s design prioritizes accessibility for non-technical stakeholders, such as product managers, subject-matter experts, and QA teams, who can contribute to evaluations and data generation without requiring coding expertise. Engineers, meanwhile, benefit from an MIT-licensed Python library that enables seamless deployment of AI tasks to production environments. The platform supports over 190 models across major providers, including OpenAI, Anthropic, Gemini, Bedrock, Ollama, and OpenRouter, ensuring compatibility with a wide range of AI infrastructure.
Since its launch in 2024, Kiln has evolved through iterative updates, introducing features like Evals V2, synthetic data batch generation, and support for advanced model providers such as Featherless AI. Kiln’s commitment to open-source principles and local-first operations ensures that users retain full control over their data and workflows.
By defining clear evaluation criteria and tracking performance across every dimension, Kiln helps teams avoid regressions and iterate with confidence. Its Auto-Optimize feature, for instance, automates the search for optimal prompt configurations, model selections, and tool integrations, reducing the manual effort required for fine-tuning and experimentation.
Kiln’s integration with Git further enhances collaboration by allowing teams to version-control datasets, tasks, and evaluation results. This approach eliminates the need for proprietary databases while providing the instant-syncing experience of SaaS platforms. Every API call or data modification is treated as a Git commit, ensuring transparency and reproducibility. The platform’s support for structured outputs, reasoning models, and multi-agent hierarchies (subagents) makes it particularly well-suited for complex AI systems that require modular and composable components.
For teams working with retrieval-augmented generation (RAG), Kiln offers drag-and-drop document ingestion, auto-generated evals, and seamless integration with vector stores like LanceDB, Pinecone, and Chroma. The RAG workflows are designed to be iterative, allowing users to refine chunking strategies, reranking algorithms, and semantic search parameters directly within the app.
Fine-tuning capabilities in Kiln are accessible to users without coding experience, thanks to a zero-code interface that supports over 60 models across providers like Fireworks, Together, and Vertex. This end-to-end workflow accelerates the development of custom AI products, from initial prototyping to production deployment.
Kiln’s Python library serves as the bridge between the desktop application and production environments. It allows developers to load Kiln projects, run tasks programmatically, and integrate datasets into existing workflows using familiar tools like Pandas or Polars. For teams using Kiln in research or experimental settings, the library’s compatibility with notebook environments and REST APIs provides additional flexibility.
The platform’s evaluation system, Evals V2, introduces specialized eval types such as code evals, tool-calling evals, regex evals, and turn-counting metrics. These evals are designed to be faster and more cost-effective than traditional LLM-as-judge approaches, with options to mix methods for hybrid evaluations. Kiln’s Eval Builder guides users through the creation of accurate evals, synthetic data, and judge prompts in minutes, democratizing access to rigorous evaluation practices.
Kiln’s AI Assistant acts as a conversational partner for data scientists and engineers, proposing improvements, optimizing prompts, and running experiments based on user feedback. The Assistant’s integration with the rest of the platform ensures that optimizations are aligned with predefined evals and performance metrics, maintaining consistency across the development loop.
One of Kiln’s distinguishing features is its local-first architecture, which eliminates dependency on cloud-based SaaS platforms. Users can operate entirely offline using Ollama or bring their own API keys for cloud-based models. Team synchronization is handled via Git, leveraging existing infrastructure to avoid vendor lock-in. Kiln’s fair-code licensing model ensures the platform remains free for individual users while sustaining long-term development through optional Pro features like the AI Assistant and Auto-Optimize.
Kiln’s adoption extends beyond individual developers to research institutions and enterprises. For example, the Vector Institute, founded by Geoffrey Hinton, has integrated Kiln into its AI research workflows, highlighting the platform’s utility in both academic and industrial settings. The tool’s ability to handle end-to-end AI projects—from evals and synthetic data generation to fine-tuning and deployment—makes it a versatile choice for teams seeking to build and optimize AI systems without sacrificing control or flexibility.
The platform’s documentation and community resources, including a Discord server, newsletter, and GitHub repository, provide ongoing support for users at all skill levels. Tutorials, video demos, and release notes offer step-by-step guidance for features like RAG configuration, agentic workflows, and model fine-tuning. Kiln’s commitment to transparency is reflected in its open-source contributions, with regular updates addressing compatibility, performance, and new model integrations.
How Kiln Works: A Unified AI Development Pipeline
A task encapsulates all the components required to define, execute, and evaluate an AI system, including prompts, input/output schemas, model configurations, and evaluation criteria. Tasks are stored as Git-friendly JSON files, allowing teams to collaborate seamlessly while maintaining version control. This approach ensures that every modification—whether a prompt adjustment, a model update, or a dataset revision—is tracked and reproducible.
The desktop application provides a user-friendly interface for non-technical team members to contribute to AI development. Product managers can rate model outputs, subject-matter experts can add training examples, and QA teams can flag regressions without writing code. Engineers, meanwhile, use the same tasks in production via the Python library, ensuring consistency between development and deployment. The app’s one-click installation and cross-platform support (Mac, Windows, Linux) lower the barrier to entry for teams of all sizes.
At the core of Kiln’s functionality is its evaluation framework, which enables users to define custom metrics and judge criteria for AI outputs. The Eval Builder automates the creation of eval datasets and judge prompts, reducing the time required to set up rigorous evaluations from hours to minutes. Evals V2 introduces advanced features like code-based judging, tool-calling validation, and regex pattern matching, providing granular control over evaluation logic.
Optimization in Kiln is driven by Auto-Optimize, an automated system that searches across hundreds of prompt variations, model selections, and configuration parameters to identify the best-performing setup for a given task. Unlike manual optimization, which relies on trial and error, Auto-Optimize uses predefined evals to guide its search, ensuring that improvements are measurable and aligned with business objectives.
Synthetic data generation in Kiln is designed to complement human-labeled datasets, filling gaps with high-quality, task-specific examples. The batch planning mode allows users to define edge cases and diversity requirements upfront, ensuring that generated data covers the full spectrum of potential inputs. Synthetic data can be used for fine-tuning, evaluation, or augmenting existing datasets, reducing the need for manual data collection. Kiln’s integration with fine-tuning workflows enables users to iterate between synthetic data generation and model training, refining outputs in a closed-loop process.
Fine-tuning in Kiln is accessible to users without coding experience, thanks to a zero-code interface that supports a wide range of models and providers. Users can select from over 60 models, including open-source and proprietary options, and deploy fine-tuned models serverlessly via platforms like Fireworks, Together, and Vertex. The platform’s synthetic data tools can generate datasets tailored to specific fine-tuning objectives, while its eval framework ensures that model performance is continuously monitored.
Kiln’s support for agents and multi-agent hierarchies enables the composition of complex AI systems. Subagents can be defined with focused context windows, allowing for modular and scalable architectures. The platform’s tool integration system, including support for MCP (Multi-Tool Calling Protocol), enables agents to interact with external systems and APIs seamlessly. Kiln’s drag-and-drop interface for RAG workflows simplifies the creation of retrieval-augmented systems, with auto-generated evals ensuring that document indexing, chunking, and reranking strategies are optimized for performance.
The Python library serves as the backbone of Kiln’s production deployment capabilities. It allows developers to load Kiln projects, run tasks programmatically, and integrate AI workflows into existing applications. The library’s adapter system abstracts interactions with AI providers, enabling users to switch between models or providers without modifying their code. Kiln projects are structured as directories of JSON files, making them easy to version-control and collaborate on. The library’s validation mechanisms ensure data integrity, while its compatibility with tools like Pandas and Polars facilitates integration with data science workflows.
Kiln’s Git-native collaboration model treats datasets and tasks as first-class Git objects. Every modification to a dataset or task is recorded as a commit, enabling teams to track changes, revert to previous versions, and resolve conflicts using standard Git tools. Kiln’s file-naming conventions use unique IDs to avoid conflicts, allowing multiple team members to work in parallel without interference.
The platform’s local-first architecture ensures that all processing occurs on the user’s machine, with optional cloud integrations for API-based models. Users can operate entirely offline using Ollama or bring their own API keys for cloud services, ensuring data privacy and minimizing latency. Team synchronization is handled via Git, leveraging existing infrastructure to avoid vendor lock-in. Kiln’s fair-code licensing model ensures that the platform remains free for individual users while sustaining development through optional Pro features.
Key Features and Capabilities of Kiln
Kiln’s evaluation system is one of its most powerful features, enabling users to define custom metrics and judge criteria for AI outputs. The Eval Builder automates the creation of eval datasets and judge prompts, reducing the time required to set up rigorous evaluations. Evals V2 introduces advanced features such as code-based judging, tool-calling validation, and regex pattern matching, providing granular control over evaluation logic.
Auto-Optimize is Kiln’s automated optimization engine, which searches across hundreds of prompt variations, model selections, and configuration parameters to identify the best-performing setup for a given task. Unlike manual optimization, Auto-Optimize uses predefined evals to guide its search, ensuring that improvements are measurable and aligned with business objectives. Auto-Optimize can also optimize RAG configurations, model parameters, and tool integrations, providing a comprehensive approach to AI system tuning.
Synthetic data generation in Kiln is designed to complement human-labeled datasets, filling gaps with high-quality, task-specific examples. The batch planning mode allows users to define edge cases and diversity requirements upfront, ensuring that generated data covers the full spectrum of potential inputs. Synthetic data can be used for fine-tuning, evaluation, or augmenting existing datasets, reducing the need for manual data collection. Kiln’s integration with fine-tuning workflows enables users to iterate between synthetic data generation and model training, refining outputs in a closed-loop process.
Fine-tuning capabilities in Kiln are accessible to users without coding experience, thanks to a zero-code interface that supports over 60 models across providers like Fireworks, Together, and Vertex. Users can select from a wide range of models, including open-source and proprietary options, and deploy fine-tuned models serverlessly. The platform’s synthetic data tools can generate datasets tailored to specific fine-tuning objectives, while its eval framework ensures that model performance is continuously monitored.
Kiln’s support for agents and multi-agent hierarchies enables the composition of complex AI systems. Subagents can be defined with focused context windows, allowing for modular and scalable architectures. The platform’s tool integration system, including support for MCP (Multi-Tool Calling Protocol), enables agents to interact with external systems and APIs seamlessly. Kiln’s drag-and-drop interface for RAG workflows simplifies the creation of retrieval-augmented systems, with auto-generated evals ensuring that document indexing, chunking, and reranking strategies are optimized for performance.
Users can drag and drop documents in various formats (PDF, images, video, audio) to create RAG systems, with auto-generated evals providing immediate feedback on performance. The platform supports semantic chunking, reranking, and semantic search, allowing users to refine their RAG configurations based on eval results. Supported vector stores include LanceDB, Pinecone, Chroma, and Qdrant, among others.
Kiln’s Python library provides the foundation for production deployment, allowing developers to load Kiln projects, run tasks programmatically, and integrate AI workflows into existing applications. The library’s adapter system abstracts interactions with AI providers, enabling users to switch between models or providers without modifying their code. Kiln projects are structured as directories of JSON files, making them easy to version-control and collaborate on. The library’s validation mechanisms ensure data integrity, while its compatibility with tools like Pandas and Polars facilitates integration with data science workflows.
Git-native collaboration is a cornerstone of Kiln’s design, treating datasets and tasks as first-class Git objects. Every modification to a dataset or task is recorded as a commit, enabling teams to track changes, revert to previous versions, and resolve conflicts using standard Git tools. Kiln’s file-naming conventions use unique IDs to avoid conflicts, allowing multiple team members to work in parallel without interference.
Kiln’s local-first architecture ensures that all processing occurs on the user’s machine, with optional cloud integrations for API-based models. Users can operate entirely offline using Ollama or bring their own API keys for cloud services, ensuring data privacy and minimizing latency. Team synchronization is handled via Git, leveraging existing infrastructure to avoid vendor lock-in. Kiln’s fair-code licensing model ensures that the platform remains free for individual users while sustaining development through optional Pro features, such as the AI Assistant and Auto-Optimize.
The AI Assistant in Kiln acts as a conversational partner for data scientists and engineers, proposing improvements, optimizing prompts, and running experiments based on user feedback. The Assistant’s integration with the rest of the platform ensures that optimizations are aligned with predefined evals and performance metrics, maintaining consistency across the development loop. It can also generate synthetic data, create evals, and run experiments through natural language interactions, making advanced AI workflows accessible to non-experts.
Kiln’s model compatibility extends to over 190 models across major providers, including OpenAI, Anthropic, Gemini, Bedrock, Ollama, OpenRouter, Fireworks, Groq, and custom OpenAI-compatible endpoints. The platform’s model library is regularly updated to include the latest releases, such as GPT-6 Astra, GLM 5.3, Claude Fable 5.1, and DeepSeek V4.1 Flash. Users can swap models with confidence, as Kiln provides tested configurations and performance benchmarks for each supported model.
Kiln’s fine-tuning capabilities are designed to be accessible and efficient, supporting zero-code workflows for users without programming experience. The platform integrates with providers like Fireworks, Together, and Vertex, enabling serverless deployment of fine-tuned models. Synthetic data generation tools can create datasets tailored to specific fine-tuning objectives, while the eval framework ensures continuous monitoring of model performance. This end-to-end workflow accelerates the development of custom AI models while maintaining transparency and control over the fine-tuning process.
The platform’s support for structured outputs, reasoning models, and multi-agent hierarchies makes it well-suited for complex AI systems. Subagents can be composed into hierarchies, each operating within its own context window, enabling modular and scalable architectures. Kiln’s tool integration system, including MCP support, allows agents to interact with external systems and APIs seamlessly. This flexibility is particularly valuable for teams building agentic systems that require dynamic interactions with external tools and services.
Tutorials, video demos, and release notes offer step-by-step guidance for features like RAG configuration, agentic workflows, and model fine-tuning. The platform’s Discord community and GitHub repository facilitate collaboration and knowledge sharing, while its newsletter keeps users informed about new features and updates. Kiln’s commitment to open-source principles ensures that the platform remains transparent and responsive to the evolving needs of AI practitioners.