How It Works
The Synthetic Data Generator (SDG) is a Python-based framework developed by the Harbin Institute of Technology Shenzhen (HITSZ-IDS) to produce high-quality structured tabular synthetic data. It achieves this through a combination of statistical methods, generative adversarial networks (GANs), and large language models (LLMs). The framework is designed to ensure synthetic data retains statistical properties of real datasets while eliminating sensitive or personally identifiable information, thus complying with privacy regulations like GDPR and ADPPA.
SDG operates by processing input data through multiple stages: data preprocessing, model training, and post-processing. The Data Processor module, introduced in May 2024, handles format conversions for columns such as datetime, custom preprocessing, null value management, and supports a plugin system for extensibility.
The framework integrates multiple algorithms for synthetic data generation, including CTGAN (Conditional Tabular GAN), GaussianCopula, and LLM-based models. CTGAN, introduced in the initial release (December 2023), supports billion-level data processing with optimized memory usage, avoiding crashes during training.
SDG also incorporates LLM-based synthetic data generation, implemented in February 2024, through the sdgx.models.LLM.singletable.gpt.SingleTableGPTModel. This model introduces two key functionalities: synthetic data generation without requiring real training data (using only metadata) and off-table feature inference, where new columns are inferred based on existing data and LLM knowledge. These capabilities address limitations of traditional GAN or statistical methods, particularly in handling complex relationships or missing data.
Privacy enhancements are central to SDG’s design. The framework supports differential privacy and anonymization techniques to further secure synthetic datasets.
SDG’s architecture is modular and extensible, allowing users to integrate custom models, data processors, or connectors via plugins. The framework also includes evaluation metrics to assess the quality of synthetic data, ensuring it closely mirrors the statistical properties of the original dataset.
Use Cases
SDG is designed for applications requiring structured tabular synthetic data that prioritizes privacy and statistical fidelity. One primary use case is data sharing, where organizations can distribute synthetic datasets without risking privacy breaches. These datasets can be used for collaborative research, third-party analytics, or public access while adhering to regulations like GDPR or ADPPA.
In model training and debugging, SDG provides a safe alternative to real-world data, particularly in domains where sensitive information must be protected.
Another critical application is system development and testing, where synthetic data can simulate real-world scenarios without the risks associated with production data.
SDG’s LLM-based capabilities extend its utility to exploratory data analysis and feature engineering. Researchers can generate synthetic data based solely on metadata, enabling hypothesis testing or prototyping without access to original datasets. The off-table feature inference function allows for the creation of new attributes derived from existing data and LLM knowledge, supporting use cases like data augmentation or synthetic experiment design.
The framework’s support for multi-table synthetic data generation further broadens its applicability.
Key Features & Capabilities
- Statistical and LLM-Based Generators: SDG integrates multiple algorithms, including CTGAN, GaussianCopula, and LLM-driven models. CTGAN is optimized for large-scale data (billions of records) with reduced memory consumption, while GaussianCopula improves efficiency for discrete data. LLM-based models enable data synthesis without training data and off-table feature inference, addressing gaps in traditional methods.
- Privacy Enhancements: The framework incorporates differential privacy and anonymization techniques to ensure synthetic data does not contain sensitive information. This compliance with GDPR and ADPPA makes it suitable for regulated industries such as healthcare, finance, and government.
- This ensures accurate representation of dataset structures and relationships, improving the fidelity of synthetic outputs.
- Extensible Architecture: The framework’s modular design allows users to integrate custom models, data processors, or connectors via plugins. This extensibility supports a wide range of use cases, from small-scale data synthesis to large-scale applications requiring billions of records.
- Evaluation Metrics: SDG includes tools to assess the quality of synthetic data, ensuring it closely mirrors the statistical properties of the original dataset. These metrics help users validate the accuracy and reliability of generated data for their specific applications.
- LLM-Integrated Synthetic Data Generation: SDG’s LLM-based models introduce unique capabilities, such as generating synthetic data from metadata alone and inferring new features based on existing data and LLM knowledge. These features expand the framework’s applicability to scenarios where traditional methods fall short, such as exploratory analysis or feature engineering.