Introduction
Marquez is an open-source metadata service designed to collect, aggregate, and visualize lineage data within a data ecosystem. Developed as the reference implementation of the OpenLineage standard, Marquez ensures compatibility with the OpenLineage specification, enabling organizations to track how datasets are produced and consumed across their pipelines. Originally released by WeWork in 2018, Marquez is now a graduated project under the Linux Foundation AI & Data, maintaining active development and community contributions. The project operates under the Apache 2.0 license, ensuring permissive use and modification.
Marquez serves as a critical tool for maintaining data and training integrity by providing global visibility into job runtime, dataset access frequency, and dataset lifecycle management. Its architecture supports real-time metadata collection from jobs and applications, making it a versatile solution for data governance, quality monitoring, and performance analytics.
The project is structured as a multi-module system, with key components including the core API for metadata collection, a web-based UI for visualization, and client libraries for HTTP API integration. Marquez also includes a Helm chart for deployment and a GraphQL endpoint in beta for advanced querying capabilities.
Marquez maintains backward compatibility with OpenLineage specifications, ensuring that newer versions of the service remain interoperable with older versions of the standard. This approach minimizes disruption during upgrades and aligns with the OpenLineage community’s versioning practices. The project’s compatibility matrix categorizes releases into CURRENT, RECOMMENDED, and MAINTENANCE tiers, guiding users on which versions to adopt for stability and feature support.
The service is built using Java 17 and relies on PostgreSQL 14 for data storage, with configuration managed through a YAML file and environment variables. Marquez’s architecture is designed to handle large-scale metadata collection, making it suitable for enterprises with diverse data pipelines and orchestration platforms.
How It Works
Marquez operates by collecting OpenLineage events from data processing jobs and applications, storing them in a centralized metadata repository. These events include details about datasets, jobs, and runs, which are then aggregated to provide a comprehensive view of data lineage. The service supports real-time ingestion of metadata, allowing organizations to monitor their data ecosystems dynamically.
The core functionality of Marquez is divided into three primary components: the metadata service, the web UI, and the HTTP API. The metadata service collects and stores OpenLineage events, while the web UI provides a visual interface for exploring lineage graphs, job dependencies, and dataset relationships. The HTTP API offers programmatic access to metadata, enabling automation and integration with other tools.
To deploy Marquez, users can follow a Docker-based quickstart guide, which simplifies the setup process by automating the configuration of services. The quickstart script initializes PostgreSQL, starts the Marquez API, and launches the web UI, allowing users to begin exploring lineage data immediately. For advanced use cases, Marquez can be deployed from source, with configuration managed through a YAML file and environment variables.
The web UI is accessible at http://localhost:3000 and provides interactive visualizations of lineage graphs, job runs, and dataset metadata. Users can trace dependencies between jobs and datasets, view execution details, and analyze performance metrics. The UI also includes a Data Observability dashboard, which offers insights into OpenLineage event statistics over defined time periods, such as the past 24 hours or 7 days.
The HTTP API is exposed on port 5000 for standard requests and port 5001 for administrative tasks, including health checks and metrics collection. By default, the API does not require authentication, though users can configure security measures as needed. The API supports endpoints for querying jobs, datasets, and runs, with recent updates introducing features like job-to-job lineage tracking and pagination for dataset versions.
Marquez also provides a GraphQL endpoint at http://localhost:5000/graphql-playground, offering a flexible query interface for metadata exploration. This endpoint is currently in beta and supports advanced querying capabilities, though its primary use remains the HTTP API for most production environments. The GraphQL interface complements the HTTP API by providing a more granular and customizable way to retrieve metadata.
Use Cases
By centralizing metadata from across an organization, Marquez enables teams to enforce data policies, track compliance, and ensure data integrity. The lineage graphs and dependency visualizations help identify data sources, transformations, and consumers, simplifying governance and audit processes.
In data quality monitoring, Marquez provides visibility into dataset access patterns, job runtime frequencies, and execution details. This information allows organizations to detect anomalies, such as missing or corrupted datasets, and investigate root causes. The Lineage API further enhances this capability by enabling automated queries for backfills and impact analysis, reducing manual effort in troubleshooting.
Performance analytics is another key use case for Marquez, as it aggregates runtime metrics, job durations, and dataset versions. By tracking job dependencies and execution histories, Marquez supports capacity planning and bottleneck identification.
Marquez integrates seamlessly with popular data orchestration platforms, including Apache Airflow, Apache Spark, Apache Flink, dbt, and Dagster. The OpenLineage compatibility ensures that Marquez can ingest metadata from a wide range of tools, making it a versatile solution for heterogeneous data ecosystems.
This information can be used to enhance data discovery, improve documentation, and facilitate collaboration across teams. The ability to traverse dependency trees and establish dataset context across multiple pipelines makes Marquez particularly valuable for organizations with complex data architectures.
Key Features & Capabilities
One of its standout capabilities is the visualization of lineage graphs, which map out the relationships between jobs, datasets, and runs. This graphical representation allows users to trace data flows, identify dependencies, and understand the impact of changes across the ecosystem.
This feature supports data-driven decision-making by offering insights into dataset access patterns, job frequencies, and metadata trends. The dashboard also includes views for sources, datasets, and jobs, along with a job list displaying the latest runs and their durations.
Job-to-job lineage tracking is another significant feature, enabling users to visualize parent-child relationships between jobs.
This feature is critical for data governance and quality assurance, as it enables organizations to audit schema evolution and ensure compatibility across pipelines.
Security updates have been implemented in recent releases, including fixes for critical vulnerabilities identified through NPM’s audit command. These updates ensure that the web UI and API remain secure against potential threats.
The project maintains a modular architecture, with distinct modules for the API, web UI, clients, and Helm chart. The removal of the integrations module in version 0.21.0 encourages users to rely on OpenLineage integrations for metadata collection, streamlining the deployment process.
Marquez’s compatibility with OpenLineage ensures that it remains aligned with industry standards for metadata collection. The project’s versioning strategy guarantees backward compatibility, allowing users to upgrade without disrupting existing workflows. The compatibility matrix categorizes releases into CURRENT, RECOMMENDED, and MAINTENANCE tiers, providing clear guidance on which versions to use for production environments.
Documentation and community involvement are central to Marquez’s development. The project maintains up-to-date documentation in its repository, with contributions from the community encouraged through pull requests and issue reporting. Users can engage with the project through the official website, GitHub repository, Slack channel, and X account, fostering collaboration and knowledge sharing.