How it Works
Top Secret is a Ruby library designed to filter sensitive information from free text before transmitting it to external services or APIs, such as chatbots and large language models (LLMs). It combines regex-based pattern matching with named-entity recognition (NER) to identify and redact sensitive data. By default, it detects and redacts credit card numbers, email addresses, phone numbers, U.S. Social Security numbers, people’s names, and locations.
The library operates by processing text through predefined filters. When text is passed to Top Secret, it scans for sensitive information using both regex patterns and NER. For example, regex filters handle structured data like credit card numbers (e.g., 4242-4242-4242-4242), while NER filters identify unstructured data such as names (e.g., Ralph) or locations (e.g., Boston). Each detected sensitive value is replaced with a placeholder (e.g., [PERSON1], [EMAIL1]), and the original value is stored in a mapping for later reference.
The filter method redacts sensitive information and returns the filtered text along with a mapping of placeholders to original values. This distinction allows developers to choose between full redaction or selective scanning based on their use case.
For instance, a custom regex filter could be added to detect IP addresses or network ranges. These custom filters can be integrated alongside or instead of the default filters, providing flexibility for specialized requirements. Additionally, Top Secret includes methods to query results by category (e.g., emails?, people?), allowing developers to check whether specific types of sensitive data were found in the text.
This is particularly useful when interacting with external services like LLMs, where responses may contain placeholders that need to be replaced with their original values. The restoration process uses the mapping generated during the initial filtering to substitute placeholders back into the text, ensuring consistency and accuracy in communication.
Top Secret depends on the MITIE Ruby library and its underlying NER model (nermodel.dat). While this model is required for NER-based filters (e.g., names, locations), users can disable NER entirely by setting modelpath to nil, which improves performance and eliminates the need for the model file. However, this limits filtering to regex-based patterns only. The model file is large and should not be committed to version control; instead, tools like Trove can be used for deployment in production environments.
Users can add the gem to their project via Bundler by running bundle add topsecret or install it globally with gem install topsecret. Configuration options, such as specifying a custom path for the NER model, can be set using the TopSecret.configure block.
Use Cases
Top Secret is primarily designed for scenarios where free text containing sensitive information must be transmitted to external services without exposing the original data. One key use case is integrating with chatbots and LLMs, where user input may contain personal or confidential details. By filtering sensitive information before sending prompts to these services, organizations can mitigate risks associated with data leaks or unauthorized access.
For example, in a customer support chatbot, user messages might include personal details such as names, email addresses, or phone numbers. The filtered response from the LLM can later be restored using the original mapping, allowing the chatbot to reference the sensitive data indirectly while maintaining security.
Another use case is validating user input before storing it in databases or logging systems. Top Secret can scan text for sensitive data and generate errors if such data is detected, preventing accidental exposure. This proactive approach reduces the risk of data breaches and ensures compliance with privacy laws such as GDPR or CCPA.
The filterall method ensures consistent labeling of sensitive data across multiple messages, which is essential for maintaining audit trails or generating reports without exposing original values.
In addition to redacting sensitive data, Top Secret can be used to monitor and log sensitive information for compliance purposes. By scanning text for specific categories of sensitive data, organizations can track instances where such data appears in logs, APIs, or other systems.
Key Features & Capabilities
Top Secret includes a comprehensive set of default filters for detecting common types of sensitive information. These filters cover credit card numbers, email addresses, phone numbers, Social Security numbers, people’s names, and locations. Each filter uses either regex patterns or NER to identify sensitive data, with NER-based filters relying on the MITIE model for accuracy.
The library provides detailed methods for querying filtered results, allowing developers to check whether specific types of sensitive data were found. Similarly, result.mapping provides a hash of placeholders to original values, enabling precise tracking of redacted data. These methods support both default and custom filters, ensuring flexibility in how sensitive information is managed.
Custom filters can be added to Top Secret to detect specialized types of sensitive data. For example, a custom regex filter could be defined to match IP addresses, network ranges, or proprietary identifiers. These filters are integrated into the scanning process alongside default filters, allowing users to extend the tool’s capabilities without modifying its core functionality. Custom labels for filters generate corresponding query methods, such as result.ipaddresses or result.networkmappings, maintaining consistency in the API design.
Batch processing is a key feature of Top Secret, enabling consistent handling of sensitive data across multiple messages. The filterall method processes an array of text inputs, assigning the same placeholder label to identical sensitive values across all messages. This ensures that references to the same data (e.g., an email address appearing in multiple messages) are consistently redacted, simplifying downstream processing. The global mapping provided by filterall includes all detected sensitive data, while individual message mappings contain only the data relevant to that specific message.
Top Secret supports the restoration of filtered text, which is essential for maintaining context in interactions with external services. When an LLM or chatbot returns a response containing placeholders (e.g., [PERSON1]), the FilteredText.restore method replaces these placeholders with their original values using the provided mapping. By restoring filtered text, organizations can ensure that sensitive data is handled securely while preserving the integrity of the conversation.
The tool’s dependency on the MITIE Ruby library and its NER model introduces some considerations for deployment. The nermodel.dat file, required for NER-based filters, is large and should not be included in version control. Instead, users are advised to use tools like Trove for deploying the model in production environments. Alternatively, NER filtering can be disabled entirely by setting modelpath to nil, which optimizes performance for applications that rely solely on regex-based filters.