How LangKit Works
LangKit is an open-source Python toolkit designed to extract and monitor text-based signals from language model inputs and outputs. It integrates with the whylogs library, an open-source data logging framework, to provide structured observability for large language models (LLMs) in production environments. The toolkit processes text data to generate metrics that help assess model behavior, security risks, and performance drift.
When a schema is instantiated with LangKit modules, the system logs both default whylogs metrics and custom LangKit metrics for prompts and responses. These metrics can then be visualized in the WhyLabs platform or analyzed independently for further insights.
The toolkit’s architecture is modular, allowing users to selectively enable metrics based on their specific needs. For example, a user monitoring an LLM for security risks might prioritize jailbreak detection and prompt injection similarity scores, while another focused on text quality could emphasize readability and sentiment analysis.
LangKit’s integration with whylogs ensures that logged data remains privacy-preserving and compatible with existing observability pipelines. The logged profiles can be used to track deviations in model behavior over time, such as shifts in toxicity levels, changes in response consistency, or unexpected patterns in user inputs.
Benchmarks on AWS instances show that lighter metric sets achieve higher throughput, with up to 2,492 chats per second on a g4dn.xlarge instance when using only light metrics. In contrast, enabling all metrics reduces throughput to 1.79 chats per second on the same instance, reflecting the computational cost of comprehensive monitoring.
LangKit’s design addresses the challenges of monitoring unstructured text in production. By transforming raw text into quantifiable signals, it provides visibility into model behavior that would otherwise remain obscured. This is particularly critical for identifying risks such as prompt injection attacks, jailbreak attempts, or unintended data leaks, all of which can have serious operational and security consequences.
The toolkit’s compatibility with whylogs ensures that logged data adheres to a standardized format, facilitating interoperability with other tools in the AI observability ecosystem.
Key Metrics and Capabilities
LangKit provides a comprehensive set of metrics categorized into four primary domains: text quality, text relevance, security and privacy, and sentiment and toxicity. Each category addresses specific aspects of LLM behavior that are critical for production monitoring.
Under text quality, LangKit evaluates readability, complexity, and grade-level scores for both prompts and responses.
The text relevance category includes similarity scores between prompts and responses, as well as comparisons against user-defined themes. These metrics are useful for detecting hallucinations, where the model’s output diverges significantly from the input or expected context.
In the security and privacy domain, LangKit offers several critical features. Additionally, it provides similarity scores for known jailbreak attempts and prompt injection attacks, enabling proactive monitoring for exploitation vectors.
The sentiment and toxicity metrics analyze the emotional tone and harmful content in text. These metrics are essential for maintaining safe and ethical interactions with LLMs, particularly in customer-facing or public applications.
LangKit’s modular approach allows users to enable only the metrics relevant to their use case. For instance, a team focused on security might prioritize jailbreak and prompt injection detection, while another concerned with user experience could emphasize sentiment and readability. This flexibility ensures that monitoring remains efficient and targeted.
The toolkit’s ability to log these metrics into whylogs profiles enables long-term tracking of model behavior. Over time, users can identify trends such as increasing toxicity in responses, growing instances of prompt injection attempts, or declining readability scores. These insights support proactive interventions, such as model retraining or input filtering, to maintain performance and safety.
Use Cases for LangKit in Production
LangKit is primarily designed for monitoring LLMs in production environments, where the unstructured nature of text inputs and outputs poses significant observability challenges. Its metrics provide actionable insights for teams responsible for deploying and maintaining AI systems, particularly in scenarios where model behavior must align with security, ethical, and performance expectations.
One key use case is security monitoring, where LangKit helps detect and mitigate risks such as prompt injection and jailbreak attempts. For example, if an LLM begins generating responses that closely match known jailbreak prompts, LangKit can trigger alerts for further investigation or automated mitigation.
Another critical application is content moderation, where sentiment and toxicity analysis ensure that model outputs remain appropriate for their intended audience.
LangKit also supports performance and drift monitoring by tracking changes in text quality, relevance, and consistency over time. For instance, if an LLM’s readability scores decline suddenly, it may indicate a degradation in model performance or a shift in input distribution. Similarly, inconsistencies in responses to the same prompt can signal hallucinations or other reliability issues that warrant attention.
In privacy-preserving applications, LangKit’s PII detection capabilities help prevent unintended exposure of sensitive information. By scanning prompts and responses for patterns matching user-defined regex groups, the toolkit can identify potential data leaks or compliance violations.
For teams deploying LLMs in multi-turn conversations, LangKit’s consistency checks and theme similarity scores provide visibility into how well the model maintains context across interactions. This is useful for applications like chatbots or virtual assistants, where maintaining coherence and relevance is essential for user satisfaction.
Technical Implementation and Performance
LangKit is implemented as a Python package and is distributed via PyPI, making it accessible for integration into existing data pipelines. The toolkit’s design ensures compatibility with whylogs, allowing users to log custom schemas that incorporate LangKit’s UDFs alongside default text metrics.
The core functionality of LangKit revolves around its modular UDFs, which are automatically wired into whylogs’ string feature collection. When a schema is initialized with LangKit modules, such as llmmetrics.init(), the system generates a profile containing both standard and custom metrics. This profile can then be visualized in the WhyLabs platform or exported for further analysis.
On an AWS g4dn.xlarge instance, light metrics achieve up to 2,492 chats per second, while enabling all metrics reduces throughput to 1.79 chats per second.
LangKit’s reliance on whylogs for data logging ensures that logged profiles adhere to a standardized format, facilitating interoperability with other tools in the AI observability ecosystem. Users can extend LangKit’s functionality by defining custom UDFs or integrating additional regex patterns for security checks. This flexibility allows teams to tailor monitoring to their specific requirements without sacrificing compatibility.
The toolkit’s design also prioritizes privacy, as whylogs enables privacy-preserving logging by default. This is particularly important for organizations handling sensitive data, as it ensures that monitoring does not compromise user privacy or violate regulatory requirements.
Integration with WhyLabs and Open-Source Contributions
LangKit was developed by WhyLabs as part of its broader initiative to advance AI observability through open-source tools. While WhyLabs has discontinued operations, the toolkit remains available under the Apache 2.0 license, ensuring continued access for the AI community. The open-source nature of LangKit aligns with WhyLabs’ mission to support responsible AI adoption by providing standardized, privacy-preserving monitoring solutions.
The toolkit’s integration with whylogs, another open-source project from WhyLabs, ensures that logged data can be seamlessly incorporated into existing observability pipelines. The standardized format of whylogs profiles also enables collaboration across teams, as metrics can be shared and compared without loss of context.
WhyLabs’ decision to open-source its platform, including LangKit and whylogs, reflects a commitment to fostering innovation in the AI observability space. By making these tools freely available, the organization has enabled researchers, developers, and enterprises to build upon its work, contributing to the broader goal of responsible AI deployment. The open-source community can now extend, modify, or integrate LangKit into their own projects, further advancing the field.
These contributions ensure that LangKit remains a reliable tool for monitoring LLMs as the technology evolves.
The toolkit’s open-source status also encourages community-driven enhancements, such as custom UDFs or additional metric modules.