How It Works
The Private Data Generation Toolbox is designed to enable machine learning practitioners to generate synthetic datasets with strong differential privacy guarantees. It implements five generative models that incorporate differential privacy mechanisms to ensure that the generated data does not reveal sensitive information from the original dataset. The toolbox supports both continuous and categorical data types, though preprocessing is required to ensure compatibility with the selected model. Users must provide their dataset in CSV format, partitioned into training and testing sets, with the target variable specified for evaluation.
The toolbox operates by training generative models on the provided training data while applying differential privacy techniques such as noisy gradient descent and moments accountants. The models include PATE-GAN, DP-WGAN, RON-GAUSS, Private IMLE, and Private PGM, each tailored to specific data types and privacy requirements. Once trained, the models generate synthetic datasets that can be evaluated using downstream classifiers to assess their utility while maintaining privacy guarantees.
The evaluation process involves training classifiers such as Logistic Regression, Random Forests, and Gradient Boosting on either the original or synthetic data. Performance is measured using metrics like AUC scores on the held-out test data. Users can adjust privacy parameters such as epsilon and delta to balance utility and privacy, with higher epsilon values reducing noise and improving data utility at the cost of weaker privacy guarantees. The toolbox also provides options to normalize data, handle categorical attributes, and configure model-specific hyperparameters to optimize results.
For models like DP-WGAN and IMLE, users can toggle privacy mode on or off, though non-private mode is only available for these specific models. The toolbox assumes the input CSV file adheres to a strict format, with the first row containing column names and the target variable specified during evaluation. Missing values must be preprocessed by the user, with caution advised to ensure any imputation methods used are also differentially private.
The workflow begins with downloading and preprocessing datasets using provided scripts. For example, the Adult Census dataset can be processed with preprocessadult.py after downloading it via downloaddatasets.sh. Once preprocessed, users can train a model such as DP-WGAN and evaluate its performance using the evaluate.py script. The script accepts arguments to specify the target variable, data paths, privacy parameters, and model-specific configurations. Synthetic data can then be saved for further use, with privacy costs accounted for throughout the pipeline.
Use Cases
The toolbox is primarily intended for testing and red teaming in machine learning workflows, particularly in domains where privacy is critical. It is suited for scenarios where synthetic data is needed for third-party sharing, model development, or benchmarking without compromising the privacy of individuals in the original dataset.
One key use case is in financial services, where synthetic data can be used to develop and test credit scoring models without exposing real borrower information. For instance, the Give Me Some Credit dataset allows practitioners to generate synthetic borrower profiles while maintaining differential privacy, enabling secure collaboration with external parties.
The toolbox also supports robustness and stress testing of machine learning models. By generating synthetic datasets with varying levels of privacy noise, practitioners can assess how well their models generalize under adversarial conditions or limited data availability. This is particularly valuable in high-stakes applications where model reliability is paramount. Additionally, the toolbox can be used to benchmark the trade-offs between privacy and utility, helping organizations optimize their synthetic data generation pipelines for specific use cases.
For researchers and developers, the toolbox provides a standardized way to evaluate generative models under differential privacy constraints. The inclusion of multiple models—each with distinct strengths—allows for comparative analysis to determine which approach best suits a given dataset and privacy budget. For example, Private PGM is optimized for categorical data, while RON-GAUSS and DP-WGAN are better suited for continuous attributes. This flexibility ensures that users can tailor their synthetic data generation process to their specific requirements.
Key Features & Capabilities
- Five Generative Models with Differential Privacy: The toolbox implements PATE-GAN, DP-WGAN, RON-GAUSS, Private IMLE, and Private PGM, each designed to handle specific data types and privacy challenges. PATE-GAN uses a teacher-student framework with differential privacy, while DP-WGAN employs noisy gradient descent and Wasserstein distance for stable training. RON-GAUSS focuses on non-interactive private data release, and Private PGM leverages graphical models for categorical data.
- Support for Continuous and Categorical Data: The toolbox supports datasets where all attributes are continuous (PATE-GAN, DP-WGAN, RON-GAUSS, IMLE) or categorical (Private PGM). Mixed data types require preprocessing to discretize continuous values or encode categorical attributes. Users must ensure no missing values remain, as the toolbox does not support them natively.
- Differential Privacy Mechanisms: Privacy is enforced through techniques such as the moments accountant, which tracks privacy loss during training, and noisy gradient descent, which adds calibrated noise to gradients. Users can adjust privacy parameters like epsilon and delta to control the strength of privacy guarantees, with lower epsilon values providing stronger privacy at the cost of reduced data utility.
- Evaluation with Downstream Classifiers: The toolbox evaluates synthetic data quality by training classifiers such as Logistic Regression, Random Forests, and Gradient Boosting on the generated datasets. This approach ensures that synthetic data retains sufficient predictive power for downstream tasks.
- Preprocessing involves handling missing values, normalization, encoding, and splitting data into training and test sets. Users can also integrate their own datasets, provided they meet the format requirements.
- Model-Specific Configurations: Each generative model supports unique hyperparameters to optimize performance. For example, PATE-GAN allows configuration of the number of teacher discriminators and noise scales, while DP-WGAN includes parameters for weight clipping and Gaussian noise variance.
- Privacy Budget Management: The toolbox accounts for privacy costs incurred during preprocessing steps, such as imputation of missing values. Users must ensure that any statistical computations used in preprocessing are also differentially private to avoid compromising the overall privacy guarantees.
- Example commands demonstrate how to train DP-WGAN on the Adult Census dataset and evaluate its performance using classifiers. Synthetic data can be saved for later use, and results are displayed in a structured format, including AUC scores for each classifier.