How DataProfiler Works
DataProfiler is a Python library developed by Capital One that automates the profiling of datasets to extract schema, statistical insights, and sensitive entities. It integrates a pre-trained deep learning model to identify personally identifiable information (PII) and non-personally identifiable information (NPI) within structured and unstructured data. The library is designed to classify data before it is used for training or analysis, ensuring compliance with privacy standards.
Users load data from supported formats—CSV, AVRO, Parquet, JSON, text, or URLs—into a Pandas-compatible DataFrame with a single command. The Data class handles file detection and loading, while the Profiler class computes statistics, performs entity recognition, and generates detailed reports.
DataProfiler generates two primary types of profiles: structured and unstructured. Structured profiles analyze tabular data, providing global statistics such as row counts, null ratios, and correlation matrices, alongside column-level metrics like data types, distributions, and statistical summaries. Unstructured profiles focus on text data, identifying entities at the word, character, or post-processed character level.
The library’s pre-trained model efficiently labels PII and NPI in both columns and free text. Users can extend this model by adding new entities or integrating custom pipelines for entity recognition. Reports can be generated in JSON format with configurable output styles, such as compact or pretty-printed formats.
Users can install the full package with machine learning dependencies via pip install DataProfiler[full], a slimmed-down version without ML features via DataProfiler[reports], or a version with only ML dependencies via DataProfiler[ml].
The library’s architecture emphasizes modularity and extensibility. The Data class abstracts file loading, while the Profiler class handles core profiling logic. Profiles are structured as dictionaries, with global statistics capturing dataset-level metrics and column/row-level statistics detailing individual attributes.
DataProfiler’s design ensures compatibility with modern Python environments. As of version 0.14.0, the library requires Python 3.10 or higher and updates dependencies such as Keras to maintain compatibility with evolving machine learning frameworks.
Key Features and Capabilities
DataProfiler excels in automating data profiling tasks, reducing the manual effort required to analyze datasets. Its pre-trained deep learning model enables accurate detection of PII and NPI, even in large-scale datasets.
One of DataProfiler’s standout features is its ability to generate comprehensive reports in JSON format. These reports include detailed statistics for structured data, such as null ratios, data type distributions, and correlation matrices. For unstructured text, the library provides entity-level analysis, including word and character-level breakdowns.
Organizations focused on sensitive data detection can install the full package with ML dependencies, while those prioritizing statistical analysis can opt for the slimmed-down version.
DataProfiler’s support for multiple data formats—CSV, AVRO, Parquet, JSON, text, and URLs—enhances its versatility. The library automatically detects file types and loads data into a Pandas DataFrame, simplifying the preprocessing step. Users can also specify custom delimiters or row limits, such as processing only the first 10,000 rows for large datasets.
The library’s profiling capabilities extend to merging profiles from datasets with identical schemas. This feature enables distributed profiling, where multiple profiles can be combined to analyze larger datasets incrementally. Such functionality is particularly valuable in environments where data is partitioned across systems or processed in batches.
DataProfiler’s architecture is designed for extensibility. Users can customize the pre-trained model by adding new entity types or integrating alternative pipelines for entity recognition. This adaptability ensures that the library can evolve alongside changing data privacy regulations and organizational requirements.
The library’s documentation and release notes highlight ongoing improvements. Recent updates, such as the addition of architecture documentation and refactored release processes, reflect a commitment to transparency and maintainability. Version 0.14.0 introduced Python 3.10 as the minimum requirement, aligning with modern development standards and dependency management practices.
Use Cases
Organizations in finance, healthcare, and government sectors can leverage the library to identify and redact sensitive information before data is used in training models or analytical workflows. By automating PII and NPI detection, DataProfiler reduces the risk of accidental data leaks and ensures adherence to regulations such as GDPR and HIPAA.
In data analysis and machine learning pipelines, DataProfiler serves as a preprocessing tool. It generates detailed profiles that inform feature engineering, data cleaning, and model training. For example, a data scientist preparing a dataset for a predictive model can use DataProfiler to identify columns containing sensitive information, allowing them to apply appropriate redaction or anonymization techniques.
Organizations processing text data—such as customer feedback, legal documents, or social media posts—can use DataProfiler to detect and classify sensitive entities within free text. This capability is particularly useful in industries where text analysis involves handling personally identifiable or confidential information.
The library’s ability to generate JSON reports further enhances interoperability, as these reports can be easily consumed by other tools and systems within a data pipeline.