DataHub: A Metadata Platform for Data and AI Assets
DataHub is an open-source metadata platform designed to address the challenges of managing data and AI assets in modern enterprises. Originally developed at LinkedIn to handle hyperscale data requirements, DataHub was open-sourced in 2020 and has since evolved into a widely adopted solution for data discovery, governance, and observability. It serves as a unified metadata graph, integrating real-time streaming or batch ingestion from diverse tools—such as warehouses, lakes, BI platforms, ML systems, and orchestration engines—to provide a centralized view of an organization’s data ecosystem. This approach ensures metadata remains fresh and actionable, supporting both human teams and AI agents.
The platform’s architecture is built on a streaming-first model, leveraging Kafka for real-time metadata updates, which distinguishes it from static catalogs. The platform is licensed under Apache 2.0, ensuring vendor neutrality and community-driven development.
DataHub’s core purpose is to provide data and training integrity by enabling organizations to track lineage across tables, dashboards, pipelines, and models. It integrates governance features such as policy management, tagging, compliance tracking, and audit trails, making it suitable for enterprise environments.
How It Works
DataHub operates by ingesting metadata from various data sources and transforming it into a unified graph representation. This graph captures relationships between data assets, enabling users to trace lineage from source to consumption. The ingestion process can be performed in real-time via streaming or in batch mode, depending on the organization’s requirements. DataHub’s architecture is modular, allowing it to integrate with a wide range of tools and platforms, including Snowflake, BigQuery, dbt, and Kafka.
The platform’s backend, known as the Graph Management Service (GMS), processes and stores metadata, while the frontend provides a React-based UI for visualization and interaction. Elasticsearch is used for indexing and search capabilities, and MySQL or PostgreSQL serves as the primary database for storing metadata.
Use Cases
DataHub is deployed across industries to address challenges related to data fragmentation, governance, and observability. One notable use case is at Netflix, where DataHub enables teams to define and manage metadata through self-service workflows. This approach improves flexibility and governance, particularly as the company expands into new verticals, leading to exponential data growth and complex impact analysis. By building a unified global catalog on DataHub, Netflix achieves cross-domain impact analysis, self-service governance, and a scalable foundation for agentic AI.
Trustpilot utilized DataHub Cloud to centralize its data discovery, ownership, lineage, and quality management. The company’s data estate had grown to over 500,000 assets across AWS and GCP, with no central catalog or consistent ownership model. The platform provided end-to-end visibility of data journeys, anchoring ownership and PII tagging across the organization.
Pinterest faced challenges with 400,000 ungoverned tables and siloed institutional knowledge, which hindered accurate AI-powered analytics. By deploying DataHub as a centralized context platform, Pinterest combined enforced data governance, AI-generated documentation, and intent-indexed query history into a single context layer. This integration powered its Analytics Agent, which became the most-used internal tool, reducing exploration time for data questions from hours to minutes. The platform also made institutional knowledge accessible across the organization.
Optum addressed bottlenecks in data access, governance, and schema standardization by building ‘Data Forge,’ a decentralized data platform powered by DataHub. This solution enabled secure, self-service access to trusted data products, improving time-to-data for consumers and enabling real-time metadata triggers for automation. The platform’s scalability was critical for managing petabyte-scale healthcare data across business units, ensuring decentralized discovery and improved data discoverability.
Super Technologies leveraged DataHub Cloud to scale data quality and discovery across 800+ professionals and four markets. The organization deployed over 600 custom SQL assertions, automated metadata checks, and lineage mapping, integrating these with a conversational discovery layer. This approach reduced KPI lookup times from significant manual efforts to seconds and enabled 200+ employees to access governed, context-rich data independently. DataHub also helped catch a live profit-and-loss incident before it impacted the business.
Key Features and Capabilities
DataHub’s unified metadata platform provides several key features that address the needs of modern data and AI teams. Universal search allows users to find any data asset instantly across the entire stack, while column-level lineage enables tracing data flow from source to consumption. Rich dataset profiles offer schema details, statistics, documentation, and ownership information, enhancing data understanding and governance.
DataHub’s real-time streaming capabilities update metadata within seconds, maintaining accuracy and relevance. It also supports AI-ready integrations, such as the Model Context Protocol (MCP), which allows AI agents like Cursor, Claude, and LangChain to connect directly to DataHub for context management. This integration enables agents to propose metric definitions and business context, which experts can review and refine, improving system reliability.
The platform’s extensibility allows organizations to define custom entity types and integrate with third-party tools, making it adaptable to diverse use cases.
Recent updates to DataHub, such as version 1.6.0.3, focus on security and authorization hardening. These include dependency updates to address vulnerabilities, such as fixes for CVE-2026-69244 in aiohttp and CVE-2026-13346 in pip. The release also introduces opt-in frontend security headers and enforces stricter authorization checks for operations like patching entities and managing user credentials. These improvements align with DataHub’s commitment to providing a secure, scalable, and reliable metadata platform for enterprise environments.