Data Contracts and Quality Validation with Soda Core
Soda Core is an open-source data quality and contract verification engine designed to validate datasets against predefined rules. It enables users to define data contracts in a human-readable YAML syntax, known as SodaCL, and automatically verify both schema and data integrity across diverse data stacks. The tool is part of the broader Soda ecosystem, with Soda Cloud serving as its commercial counterpart, offering additional features such as centralized management and anomaly detection.
The project was initiated in 2020 under the Apache 2.0 license, making it freely accessible for development and integration into open-source projects. Written in Python, Soda Core supports a wide range of data sources, including PostgreSQL, Snowflake, BigQuery, Databricks, DuckDB, and others. Its primary purpose is to maintain data and training integrity by enforcing data quality standards through automated checks.
Soda Core operates on the principle of declarative data contracts, where users define expectations for their datasets in YAML files. These contracts specify schema requirements, row counts, and column-level validations, such as missing values, invalid entries, or adherence to specific formats. The engine then executes these checks against the actual data, generating reports on compliance or deviations. This approach ensures that data pipelines adhere to predefined quality standards, reducing the risk of errors in downstream processes.
The tool provides a command-line interface (CLI) and a Python API, allowing users to generate, test, publish, and verify contracts both locally and remotely. Local execution is useful during development, while remote execution via Soda Cloud enables centralized monitoring and collaboration across teams. Soda Core’s flexibility extends to its integration with popular workflow orchestration tools like Airflow, Dagster, and Prefect, making it adaptable to various data engineering environments.
How It Works
Soda Core’s functionality is built around three core components: data source configuration, contract definition, and verification execution.
Once a data source is configured, users define data contracts in YAML files, specifying the checks to be applied. These checks can target entire datasets or individual columns, with options for schema validation, row count verification, and custom business logic. For example, a contract might enforce that a column named 'size' only accepts values from a predefined list ['S', 'M', 'L'], or that no more than 10% of rows in a 'name' column are missing. The YAML syntax is designed to be intuitive, reducing the learning curve for teams adopting the tool.
Verification is initiated via the CLI or Python API, where Soda Core scans the dataset against the defined contract. The tool supports over 50 built-in data quality checks, covering common validations such as uniqueness, completeness, and consistency. Results are generated in real-time, with detailed logs indicating which checks passed or failed.
Soda Core’s integration with Soda Cloud extends its capabilities by enabling remote execution and centralized monitoring. Users can publish contracts and verification results to Soda Cloud, where they are stored as the source of truth. This feature facilitates collaboration across distributed teams, as all stakeholders can access the same data quality metrics and historical trends. Additionally, Soda Cloud provides advanced anomaly detection, identifying patterns or outliers that might indicate deeper issues in the data pipeline.
The tool’s architecture supports both local and remote execution models. Remote execution, on the other hand, leverages Soda Cloud’s infrastructure to handle larger-scale validations and provide additional features such as scheduled scans and alerting.
Use Cases
By integrating checks into the data pipeline, teams can catch issues early, before they propagate to training stages.
This proactive approach minimizes downtime and ensures that only high-quality data moves through the system.
Another key use case is in data warehouse environments, where multiple teams contribute to a shared dataset. Soda Core helps maintain consistency by enforcing contracts across different schemas and tables. This ensures that downstream applications, such as business intelligence tools, receive data in the expected format.
Key Features and Capabilities
Soda Core’s feature set is designed to address common challenges in data quality management, including schema drift, missing values, and inconsistent data types. These checks include schema validation to ensure tables and columns match expected structures, row count verification to confirm data completeness, and custom validations for business-specific rules.
The tool’s YAML-based contract language is a key differentiator, offering a balance between readability and expressiveness. This declarative approach allows data teams to document their expectations clearly and enforce them automatically.
Soda Core’s integration with popular data sources is another strength, with native support for PostgreSQL, Snowflake, BigQuery, Databricks, and DuckDB. This compatibility ensures that users can apply data quality checks across heterogeneous environments without needing to rewrite their contracts.
Recent updates to Soda Core, such as version 4.25.0, have introduced improvements in performance and compatibility.
Soda Core’s CLI and Python API provide multiple ways to interact with the tool, catering to different user preferences. The CLI is ideal for quick validations or scripted workflows, while the Python API offers more granular control for integration into custom applications. Similarly, a DevOps engineer might use the CLI to automate data quality checks as part of a CI/CD pipeline, ensuring that only validated datasets proceed to production.
The tool’s support for Soda Cloud adds a layer of scalability and collaboration. By publishing contracts and results to Soda Cloud, teams can centralize their data quality efforts, making it easier to monitor trends, track issues, and share insights across departments.
Soda Core’s development environment is designed to be accessible, with clear documentation and support for modern Python practices. Users can install the tool via PyPI using either pip or the recommended UV package manager, with support for Python versions 3.9 through 3.12. For instance, users can clone the repository, set up a virtual environment, and run tests using pytest, ensuring that their local setup matches the project’s standards.
Integration and Extensibility
Its compatibility with orchestration platforms like Airflow, Dagster, and Prefect allows data teams to embed data quality checks directly into their pipelines. This tight integration ensures that data quality is treated as a first-class concern in the pipeline design.
Soda Core’s support for legacy versions, such as v3, ensures backward compatibility for users transitioning from older releases. While version 4 introduced changes to package naming conventions, the project maintains documentation and examples for migrating from v3 to v4. For instance, users of v3 can pin their package versions to 3.5.x while exploring the new features in v4, ensuring a smooth transition.