How it Works
fg-data-profiling automates exploratory data analysis (EDA) for pandas and Spark DataFrames, delivering a concise yet comprehensive report in a single line of code. The tool mimics the simplicity of pandas’ df.describe() function but extends its capabilities to include detailed statistical summaries, visualizations, and data quality alerts. The report is structured into three primary sections: Overview, Alerts, and Reproduction.
The Overview section provides global dataset metrics, including the number of records, variables, missing values, duplicates, and memory usage. The Alerts section flags potential data quality issues, such as high correlation between variables, skewness, uniformity, zeros, missing values, or constant values.
Reports can be generated programmatically or via a command-line interface (CLI) for CSV files, with customizable titles and configurations. The CLI command dataprofiling --title "Example Profiling Report" --configfile default.yaml data.csv report.html demonstrates this functionality.
Installation is straightforward via pip (pip install fg-data-profiling) or conda (conda install -c conda-forge fg-data-profiling). Additional dependencies, such as notebook widgets, Unicode support, or Spark integration, can be installed using extras (e.g., pip install -U fg-data-profiling[notebook,unicode,pyspark]). The package is actively maintained, with recent updates including Python 3.14 support and a rename from ydata-profiling to fg-data-profiling to reflect its transition under the Data-Centric AI Community.
The migration from ydata-profiling to fg-data-profiling involves uninstalling the old package (pip uninstall ydata-profiling), installing the new one (pip install fg-data-profiling), and updating imports in codebases (e.g., replacing import ydataprofiling with import dataprofiling). A one-liner command (grep -r "ydataprofiling" . --include=".py") helps identify affected files.
Use Cases
fg-data-profiling addresses a variety of practical scenarios in data analysis and quality assurance.
The tool is also designed to handle large datasets through Spark integration, enabling scalable profiling for distributed data environments.
For database integration, YData Fabric Data Catalog extends fg-data-profiling’s capabilities by enabling direct profiling of data stored in relational databases (PostgreSQL, Oracle, Azure SQL) and object storages (Google Cloud Storage, AWS S3, Snowflake). This integration supports interactive and guided profiling experiences, reducing the need for manual data extraction and preprocessing.
The profile.towidgets() method renders the report as interactive widgets, while profile.tonotebookiframe() embeds the HTML report directly into a notebook cell.
For example, the command dataprofiling --title "Example Profiling Report" --configfile default.yaml data.csv report.html produces an HTML report from a CSV input, with configurable titles and settings via a YAML file.
Key Features & Capabilities
- Warnings and Alerts: Identifies data quality issues such as missing values, skewness, high correlation, uniformity, zeros, or constant values, with actionable insights for cleanup or transformation.
- Multivariate Analysis: Includes correlation matrices, missing data analysis, duplicate detection, and pairwise variable interactions, revealing relationships and dependencies in the data.
- Dataset Comparison: Enables side-by-side analysis of multiple datasets, highlighting structural or distributional differences in a single report.
- Flexible Output Formats: Supports HTML for sharing, JSON for programmatic use, and Jupyter Notebook widgets for interactive exploration, catering to diverse workflows.
- Database Connectivity: Through YData Fabric Data Catalog, connects to relational (PostgreSQL, Oracle) and object-based (S3, Snowflake) storage systems, reducing manual data extraction steps.
- Unicode and Encoding Support: Offers detailed analysis of text encoding and scripts, with optional extras for enhanced Unicode support at the cost of increased resource usage.