How it Works
Detoxify is a Python library designed for toxic comment classification, built using transformer models and PyTorch Lightning. It provides pre-trained models that address three key challenges from Jigsaw: Toxic Comment Classification, Unintended Bias in Toxic Comments, and Multilingual Toxic Comment Classification. The library is structured to offer a user-friendly interface for predicting toxicity in text inputs, making it accessible for both research and practical applications in content moderation.
The library supports three primary models: original, unbiased, and multilingual. Each model is trained on specific datasets and optimized for distinct objectives. The unbiased model, based on RoBERTa, aims to minimize unintended bias related to identity mentions, using data from Civil Comments.
Detoxify is installed via pip and can be integrated into existing workflows with minimal setup. Users can load models directly using the library’s API, specifying the model variant and optionally designating a device (CPU or GPU) for computation.
The library also includes lightweight versions of the original and unbiased models, trained using ALBERT, which reduce computational requirements while maintaining high performance. These smaller models are accessible via the same API by specifying 'original-small' or 'unbiased-small'.
For training custom models, Detoxify requires access to the Jigsaw datasets, which can be downloaded using the Kaggle API. The library’s design emphasizes reproducibility and ease of use, ensuring that users can fine-tune models on domain-specific datasets while adhering to ethical considerations.
Use Cases
Detoxify is primarily intended for research purposes, enabling developers and researchers to experiment with toxicity detection models and evaluate their performance on custom datasets. Its modular design allows for fine-tuning on datasets that reflect real-world demographics, making it suitable for academic studies or internal research projects focused on improving content moderation systems.
In practical applications, Detoxify serves as a baseline toxicity classifier for filtering harmful content in online platforms. It can be integrated into content moderation pipelines to pre-screen comments, flagging toxic or biased language before human review. This reduces the workload for moderators by automating the initial detection phase, though it is recommended to use it in conjunction with human oversight to address its limitations, such as false positives in context-dependent or humorous content.
By supporting seven languages, it enables toxicity detection across diverse linguistic groups, addressing the challenge of bias in multilingual environments.
Detoxify’s unbiased model is specifically designed to mitigate bias in toxicity detection, particularly concerning identity-based mentions. It aligns with ethical guidelines by minimizing unintended bias, though users should remain aware of potential residual biases in real-world applications.
Key Features & Capabilities
Detoxify offers three specialized models, each addressing distinct aspects of toxicity detection:
- Original Model: Trained on the Toxic Comment Classification Challenge dataset from Wikipedia comments, this model detects six types of toxicity: toxic, severe toxic, obscene, threat, insult, and identity hate. It achieves an AUC score of 98.64 on the test set, closely matching the top Kaggle leaderboard score of 98.86.
- Unbiased Model: Developed for the Unintended Bias in Toxicity Classification challenge, this model focuses on reducing bias related to identity mentions. The model achieves an AUC score of 93.74, aligning with the Kaggle leaderboard score of 94.73. It is based on RoBERTa and has been updated to include data from both the first and second Jigsaw challenges.
- The model’s primary evaluation metric is toxicity, though it retains labels from both challenges.
Each model variant returns consistent class names to avoid confusion during use. For example, the original model’s 'identityhate' label has been standardized to 'identityattack' to match the unbiased model’s naming convention.
Detoxify also provides lightweight versions of the original and unbiased models, trained with ALBERT to reduce computational overhead. These models achieve competitive AUC scores of 98.28 and 93.36, respectively, demonstrating that smaller architectures can maintain high performance in toxicity detection tasks.
The library includes tools for both inference and training, with dependencies such as PyTorch Lightning and the Hugging Face Transformers library. For inference, users only need to install the core dependencies, while training requires additional tools like the Kaggle API for dataset access.
The library emphasizes the importance of fine-tuning on diverse datasets to mitigate bias and improve accuracy in real-world scenarios. Users are encouraged to review resources on bias in toxicity detection to ensure responsible deployment.
Detoxify’s performance metrics are benchmarked against Kaggle leaderboard scores, providing a clear reference for its effectiveness. Similarly, the unbiased and multilingual models achieve scores within a few percentage points of the leaderboard, demonstrating their reliability as starting points for further development.
The library’s release history reflects ongoing improvements, including updates to model weights, compatibility fixes for newer versions of PyTorch, and enhancements to the training pipeline. These updates ensure that Detoxify remains functional and efficient across evolving technical landscapes.