How it Works
ydata-synthetic, now maintained as fg-data-synthetic, generates synthetic tabular and time-series data using generative adversarial network (GAN) architectures. The tool leverages TensorFlow 2.0 for deep learning implementations and provides a modular ecosystem for synthetic data generation.
The tool operates by training a synthesizer model on real-world datasets to learn their statistical properties. Once trained, the model generates synthetic data that mimics the original dataset’s structure without exposing identifiable information. This process ensures privacy compliance while preserving data utility for machine learning applications.
A key feature is the integration of a Streamlit-based user interface, introduced in version 1.0.0, which simplifies the generation of synthetic data through a low-code experience. Users can train models, generate synthetic samples, and profile results directly from the interface, reducing the need for manual scripting.
The package also includes a Gaussian Mixture-based model for faster synthetic data generation, eliminating the dependency on GPU acceleration for quick prototyping.
Use Cases
Synthetic data generated by ydata-synthetic addresses several critical needs in data privacy and machine learning workflows. Its primary applications include:
- Privacy Compliance: Generating synthetic datasets that replicate real-world data structures without containing personally identifiable information (PII). This is essential for sharing data across organizations or for public use while adhering to regulations like GDPR or HIPAA.
- Bias Mitigation: Creating balanced synthetic datasets to reduce biases present in original datasets.
- Dataset Augmentation: Expanding small or imbalanced datasets to improve the performance of machine learning models. Synthetic data can be used to supplement training sets, especially in domains where collecting real data is challenging or expensive.
- Testing and Development: Providing safe environments for developing and testing machine learning models without risking exposure of sensitive information.
The tool’s versatility extends to both structured tabular data and sequential time-series data, making it applicable in finance, healthcare, and other industries where data privacy and model robustness are priorities.
Model Architectures
fg-data-synthetic supports a diverse set of GAN-based architectures tailored for different data types and use cases. For tabular data, the available models include:
- CGAN (Conditional GAN): Generates synthetic data conditioned on specific features, improving the relevance of generated samples.
- CTGAN (Conditional Tabular GAN): Specifically designed for tabular data, CTGAN handles mixed data types (numeric and categorical) and conditional generation, making it ideal for structured datasets.
- Gaussian Mixture: A non-deep-learning alternative that offers fast synthetic data generation without requiring GPU resources, suitable for quick prototyping or resource-constrained environments.
For sequential and time-series data, the tool provides:
Streamlit UI Integration
- Model Training: Users can train synthesizer models directly through the UI, selecting from available architectures and configuring hyperparameters.
- Synthetic Data Generation: After training, the UI enables the generation of synthetic samples with configurable parameters, such as sample size and data distribution constraints.
This low-code approach lowers the barrier to entry for users who may not have deep expertise in machine learning or GANs, making synthetic data generation more accessible.
Example Datasets and Notebooks
The repository includes pre-loaded datasets and Jupyter Notebook examples to demonstrate the tool’s capabilities across different scenarios.
- Tabular Data: Adult Census Income dataset for income prediction tasks, Credit Card Fraud detection, and Cardiovascular Disease analysis.
- Time-Series Data: Stock market data for financial forecasting and FCC MBA dataset for sequential pattern analysis.
These examples provide practical templates for users to adapt to their own datasets, accelerating the implementation process.
Migration and Compatibility
The transition from ydata-synthetic to fg-data-synthetic involved renaming the package and updating import statements.
- Uninstall the old package using pip uninstall ydata-synthetic.
- Install the new package with pip install fg-data-synthetic.
- Update import statements in their codebase from ydatasynthetic to datasynthetic.