Introduction
Dingo is a comprehensive quality evaluation tool for AI data, models, and applications developed by Migo Lab. It is designed to systematically assess and improve the quality of training data, fine-tuning datasets, and production AI systems. Targeted at machine learning practitioners, data engineers, and AI researchers, Dingo provides a structured approach to identifying and addressing issues in AI workflows.
The tool integrates rule-based and large language model (LLM)-based checks to evaluate training data, retrieval-augmented generation (RAG) contexts, and hallucination risks. It also includes a dashboard for visualizing evaluation results, making it easier to monitor and manage data quality at scale.
Dingo is available under the Apache 2.0 license and is implemented in Python. It supports both open-source and enterprise SaaS versions, with the latter offering additional features such as a web UI, access control, visual reports, and RESTful API integration.
The project was initiated in 2024 and has since evolved to include advanced features like multi-source data integration, hybrid evaluation strategies, and extensibility for custom rules and metrics.
How It Works
Dingo operates through a combination of rule-based and LLM-based evaluation mechanisms. Rule-based checks are fast and deterministic, applying predefined criteria to identify issues such as special characters, incomplete fields, or invalid formats. These checks are particularly useful for large-scale datasets where speed and consistency are critical.
LLM-based evaluations provide deeper insights by leveraging models like GPT-4o to assess text quality, factuality, and adherence to ethical guidelines.
The tool supports a hybrid approach, where rule-based checks are applied to the entire dataset for broad coverage, while LLM-based evaluations are applied to a sampled subset to balance cost and depth. This strategy ensures that both efficiency and accuracy are maintained in production environments.
The dashboard provides field-level insights, trend analysis, and export capabilities, facilitating collaboration and decision-making in enterprise settings.
Use Cases
Dingo is applicable across multiple stages of AI development, from pre-training data validation to production system monitoring.
- Training Data Evaluation: Assessing the quality of pre-training datasets to ensure completeness, effectiveness, and security. Metrics such as text quality, factuality, and similarity are applied to filter out low-quality or harmful data before training begins.
- Fine-Tuning Dataset Validation: Evaluating supervised fine-tuning (SFT) data for adherence to the Honest, Helpful, and Harmless (3H) principles.
- RAG System Assessment: Comprehensive evaluation of retrieval and generation quality in RAG systems using five academic-backed metrics: faithfulness, context precision, answer relevancy, hallucination detection, and factuality checks.
- This is critical for maintaining trust and reliability in production AI applications.
- This ensures that multimodal datasets are consistent and suitable for training or inference.
- Security and Privacy Compliance: Detecting personally identifiable information (PII) and toxic content using tools like the Perspective API. This helps organizations comply with data protection regulations and mitigate risks associated with sensitive information.
Practical examples demonstrate how Dingo can be integrated into workflows. For instance, evaluating LLM chat data involves defining evaluation arguments for an LLM model like GPT-4o and applying both rule-based and LLM-based checks to identify issues such as special characters or low-quality responses.
Users can specify evaluation rules for different fields, such as validating ISBNs or checking for abnormal characters, and save results for further analysis.
Key Features & Capabilities
Dingo’s design emphasizes production readiness, offering features that address scalability, customization, and integration challenges.
- Multi-Field Evaluation Pipeline: Dingo allows users to apply different quality checks to different fields within a dataset simultaneously. For example, one field might be validated for ISBN compliance while another is assessed for text quality. This approach streamlines evaluation processes for structured data without requiring separate scripts for each field.
- Stream Processing for Large Datasets: The tool leverages SQLAlchemy’s server-side cursors to process billions of rows without causing out-of-memory errors. Data is fetched and evaluated row-by-row, making it feasible to analyze production databases directly without exporting to intermediate files.
- Field Isolation in Memory: During RAG evaluations, Dingo isolates different field combinations in memory to prevent context bleeding. This ensures that metrics like faithfulness or answer relevancy are calculated accurately, even when evaluating multiple interdependent fields.
- Hybrid Rule-LLM Strategy: Dingo combines fast rule-based checks with sampled LLM evaluations to balance cost and coverage.
- Extensibility Through Registration: The tool’s plugin architecture allows users to register custom rules, prompts, and models without modifying the core codebase. This makes it adaptable to domain-specific requirements, such as adding a custom rule to detect empty content fields or extending existing metrics for specialized use cases.
- Rich Reporting and Visualization: Evaluation results are presented in detailed reports with graphical visualizations. The dashboard provides interactive charts, trend analysis, and field-level insights, enabling teams to track data quality over time and make informed decisions.
- Enterprise-Grade SaaS Version: The SaaS edition of Dingo offers additional features tailored for enterprise environments, including a web-based UI for non-technical users, access control via JWT and Google OAuth 2.0, and RESTful API integration for seamless system integration.
Recent updates to Dingo have expanded its capabilities, particularly in retrieval evaluation and hallucination detection.
This capability is increasingly relevant as synthetic data becomes more prevalent in AI workflows.
Dingo’s metric library comprises over 100 evaluations across categories like pre-training text quality, SFT data quality, RAG assessment, and security.