MOSTLY AI Synthetic Data SDK
The MOSTLY AI Synthetic Data SDK is a Python-based toolkit designed for generating high-fidelity, privacy-safe synthetic datasets from tabular and language data.
The SDK operates in two primary modes: LOCAL and CLIENT. LOCAL mode trains and generates synthetic data on the user’s own compute resources, while CLIENT mode connects to a remote MOSTLY AI platform for processing. This dual-mode approach ensures flexibility, whether users prefer on-premise control or cloud-based scalability.
Generators are trained on existing data assets to produce synthetic records, Synthetic Datasets allow for the generation of custom volumes of synthetic data, and Connectors facilitate integration with organizational data sources such as databases and cloud storage systems.
How It Works
The SDK provides a programmatic interface for creating, managing, and querying synthetic data. Users interact with the toolkit via Python APIs, which include methods for training generators, generating synthetic records, probing generators for sample data, and connecting to external data sources.
For tabular data, the SDK leverages TabularARGN (arXiv:2501.12012), a model architecture optimized for efficiency and quality. TabularARGN achieves synthetic data fidelity comparable to state-of-the-art models while reducing training and generation times by 1–2 orders of magnitude. This efficiency allows users to generate millions of synthetic records within minutes, even on CPU-based systems.
Language data synthesis is supported through two approaches: a lightweight, non-pre-trained LSTM model for short-text scenarios with limited variability, and fine-tuning of pre-trained Hugging Face models for broader textual domains. The latter approach leverages models like AutoModelForCausalLM, which incorporate general world knowledge before adapting to domain-specific training data. GPU acceleration is strongly recommended for language model tasks to ensure performance.
The SDK incorporates differential privacy mechanisms to mitigate re-identification risks, ensuring compliance with privacy regulations. Additionally, the toolkit provides automated quality assurance through metrics for fidelity and privacy, along with interactive HTML reports for visual analysis of synthetic data distributions.
Use Cases
The Synthetic Data SDK is tailored for applications requiring privacy-preserving data access, such as anonymized datasets for machine learning, testing, and analytics.
- Data Privacy Compliance: Generating synthetic datasets that mimic real-world distributions without exposing sensitive information, enabling secure sharing of data for research or development.
- Machine Learning Training: Creating synthetic datasets to augment limited real-world data, particularly in domains with sparse or imbalanced samples.
- Text and Multimodal Data Synthesis: Generating synthetic text, geospatial, or mixed-type data for applications in natural language processing, geospatial analysis, or multimodal AI development.
The SDK’s support for connectors to databases (e.g., PostgreSQL, Snowflake, BigQuery) and cloud storage systems (e.g., S3, Databricks) extends its utility to enterprise environments, where synthetic data can be seamlessly integrated into existing data pipelines.
Broad Data Support
The SDK handles diverse data types, including categorical, numerical, geospatial, and textual data.
Advanced Training and Generation
Training configurations are highly customizable, with options for GPU/CPU acceleration, differential privacy tuning, and progress monitoring. Users can start with small-scale experiments (e.g., training on a subset of data for a limited number of epochs) before scaling up to full datasets.
Automated Quality Assurance
Quality metrics are computed during and after training to evaluate synthetic data fidelity and privacy guarantees.
Flexible Sampling and Conditional Generation
Users can specify sampling parameters to generate synthetic records at scale, including up-sampling to match desired volumes or conditional generation based on column values.
Seamless Integration and Deployment
Local training allows for air-gapped deployment, where users can install the SDK and required dependencies (including Hugging Face models) offline. Docker support provides a consistent runtime environment, while CLIENT mode enables interaction with the MOSTLY AI platform for collaborative or cloud-based use cases.
Performance Optimization
Performance varies based on data characteristics, compute resources, and model configurations. TabularARGN’s efficiency enables rapid generation of large-scale synthetic datasets, even on CPUs. For language models, GPU acceleration is critical, particularly when fine-tuning large pre-trained models. The SDK recommends a phased approach—starting with small-scale experiments—to refine configurations before full-scale deployment.
Technical Implementation
The SDK is implemented in Python and requires Python 3.11 or higher. Installation is managed via pip or uv pip, with optional extras for LOCAL mode (including PyTorch dependencies) or CLIENT mode. Users can install the base package for CLIENT-only operations or opt for mostlyai[local] or mostlyai[local-gpu] for full functionality, including GPU-accelerated training.
For environments without internet access, the SDK provides wheel files and instructions for manual dependency installation. Hugging Face models used in language synthesis must be pre-downloaded and transferred to air-gapped systems via the local cache directory.