OWASP LLM Top 10 — Mapped to Scanner Rules
Compliance · 2019-02-12 · 7 min read · FilterPrompt Team
Every item in the OWASP LLM Top 10, mapped to the validators and rule defaults that mitigate it out of the box.
The OWASP LLM Top 10 is the closest thing we have to a shared checklist for production LLM apps. Here's how each item maps to scanner rules and validators that ship enabled by default.
LLM01 — Prompt Injection
Default rules: 'instruction override', 'role hijack', 'context smuggling'. Engine layer: an injection scorer evaluating every request.
LLM02 — Insecure Output Handling
Output-side validators block markdown images to non-allowlisted hosts and strip JS-protocol links before the response reaches your client.
LLM06 — Sensitive Information Disclosure
Built-in PII detectors and secret scanners on both directions. Add custom rules from the Rules page for your internal IDs and document classifications.
LLM10 — Model Theft
Per-tenant rate limits and monthly quotas, plus an output-side rule that detects 'reveal your system prompt' style probes and rate-limits the offending key.
LLM03 — Training Data Poisoning: Protecting the Foundation
Training Data Poisoning (LLM03) is a subtle yet potent attack vector, compromising the integrity of an LLM's knowledge base. Attackers inject malicious, biased, or incorrect data into the training dataset, influencing the model's behavior, outputs, and even its ethical alignment. While often a pre-deployment concern, continuous learning systems and fine-tuning pipelines are perpetually vulnerable. For scanner rules, monitoring the provenance and integrity of fine-tuning datasets is paramount. We implement checksum validation, anomaly detection for data distributions in new training batches, and enforce strict access controls on data lakes feeding into model training.
Mitigation also extends to the operational phase. Imagine a scenario where a user, through an indirect prompt injection, causes the model to generate content that's subtly biased against a particular group. If this generated content is then, inadvertently, cycled back into a reinforcement learning from human feedback (RLHF) loop or a self-supervised learning process, it can gradually poison the model's future behavior. Our platform includes output-side validators that cross-reference generated text against known bias dictionaries and sentiment analysis models, flagging outputs that deviate significantly from expected ethical boundaries. These flags can then trigger human review or automatic suppression from feeding into any subsequent training loops, effectively creating a quarantine zone.
LLM04 — Denial of Service (DoS): Resource Starvation Attacks
LLM04, Denial of Service, specifically targets the operational availability of your LLM service. Attackers exploit resource-intensive operations inherent to LLMs, such as large context windows, complex retrieval augmented generation (RAG) queries, or repetitive generation of extremely long responses. Without proper controls, a malicious actor can drain compute resources, leading to slow responses, service outages, and increased infrastructure costs. Our scanner rules proactively address this through multiple layers of defense, extending beyond simple rate limiting.
Sophisticated DoS vectors involve crafting prompts that trigger extraordinarily long processing times. For instance, recursive self-referential prompts or prompts demanding mathematically complex computations for little practical output. Our system employs a real-time token-per-second (TPS) budgeting across all stages of LLM interaction – input parsing, internal reasoning, and output generation. If a user's prompt consistently triggers significantly higher TPS consumption than their quota or historical average, it's flagged and throttled. Furthermore, output length caps are enforced dynamically, preventing the generation of multi-megabyte responses that tie up GPU inference time and network bandwidth. This combination offers granular protection against both traditional volumetric DoS and more nuanced algorithmic DoS.
LLM05 — Supply Chain Vulnerabilities: Trusting Upstream Reliance
LLM05 underscores the inherent risks in the supply chain of AI development. This isn't just about Python packages; it extends to pre-trained models, fine-tuning datasets from third parties, open-source frameworks, and even specialized hardware accelerators. A vulnerability or malicious dependency introduced at any point upstream can compromise the entire LLM application. For instance, a malicious actor could inject a backdoor into a Hugging Face model or a PyTorch dependency, allowing arbitrary code execution or data exfiltration when loaded for inference.
Our scanner rules integrate with secure MLOps practices. For every model artifact, we enforce strict software bill of materials (SBOM) generation and validation, ensuring that all components, from foundational models to custom layers, are explicitly declared and their hashes verified against trusted registries. Furthermore, for models pulled from external sources, we can deploy a sandboxing policy that restricts network access and file system interactions during initial load and even during inference, limiting the blast radius of a compromised model. This holistic approach prevents malicious code from ever reaching the production environment or executing with elevated privileges.
LLM07 — Insecure Plugin Design: Extending Capabilities Safely
Plugins and external tools significantly augment LLM capabilities, enabling web browsing, code execution, and database interactions. However, LLM07 highlights that improperly designed or secured plugins are a critical attack surface. A malformed prompt could trick an LLM into invoking a plugin with elevated privileges or unintended arguments, leading to arbitrary code execution, data manipulation, or information disclosure. Traditional prompt injection protections alone are insufficient here; the focus shifts to the interface between the LLM and its tools.
Our scanner implements strict validation for plugin arguments and return types. For example, if a plugin expects a URL, the validator ensures it's a well-formed URL pointing to an allow-listed domain, preventing SSRF attacks. For plugins that execute code (e.g., a Python interpreter plugin), a custom sandboxing environment is enforced, defining the exact resources and functions available. Before any plugin call, the LLM's generated arguments are subjected to a security policy check that validates not just syntax, but also semantic intent against defined allow-lists of operations. This 'least privilege' philosophy is applied to every tool invocation, ensuring the LLM can only interact with its environment in predefined, safe ways.
LLM08 — Excessive Agency: Limiting Autonomous Actions
LLM08, Excessive Agency, addresses the dangers of granting LLMs too much autonomy without adequate human oversight or safety mechanisms. As LLMs evolve into AI agents capable of planning and executing multi-step tasks, their ability to perform actions in the real world (e.g., send emails, make financial transactions, deploy code) introduces new risks. An attacker exploiting Prompt Injection combined with Excessive Agency could orchestrate a chain of malicious actions, turning the LLM into a powerful weapon.
Our mitigation strategy for Excessive Agency involves a multi-tiered approval system and dynamic permissioning. While an LLM might generate a plan involving sensitive actions, these actions are not executed directly. Instead, they are routed through a human-in-the-loop (HITL) review process, where a human operator must explicitly approve based on a clear summary of the LLM's proposed action and its potential impact. For less sensitive or routine actions, an automated 'guardrail' system can apply rules based on predefined thresholds or patterns (e.g., blocking any email containing specific keywords or preventing transactions above a certain monetary value) before human escalation. This ensures that while LLMs can propose ambitious actions, critical execution remains under controlled human supervision or strictly delimited automated safety policies.
- Implement a mandatory Human-in-the-Loop (HITL) for all high-impact actions (e.g., financial transactions, system modifications).
- Dynamic permissioning: LLM's current permissions are contextually adapted, minimizing available actions to only what is strictly necessary for the immediate task.
- Automated safety policies and guardrails: Pre-veto rules based on content, value, or recipient for specific categories of actions.
- Clear audit trails for all LLM-initiated actions, regardless of human approval status.
- User confirmation prompts for sensitive LLM-generated actions, even if pre-approved by internal policies.
LLM09 — Over-reliance: Understanding LLM Limitations
LLM09 tackles the human element of security: over-reliance on LLM outputs without critical verification. Users, both internal and external, may implicitly trust LLM-generated content, especially if it's presented convincingly, even when it's inaccurate, biased, or even malicious (a phenomenon known as 'hallucination'). This over-reliance can lead to poor decision-making, adoption of incorrect information, or unknowingly acting on adversarial prompts.
Our scanner helps mitigate over-reliance not just by filtering malicious content, but by injecting metadata and confidence scores directly into the LLM's output when appropriate. For instance, responses derived from RAG are tagged with source citations. If an answer involves a high degree of inference or generalization, a 'low confidence' marker can be programmatically added. We also offer rules to automatically append disclaimers for specific types of generated content (e.g., 'This financial advice is AI-generated and should not replace professional consultation'). This moves beyond simply blocking bad outputs and educates the user, fostering a more critical engagement with AI interactions. Furthermore, our platform can integrate with internal knowledge bases to flag outputs that contradict verified facts, helping to curb factual errors and hallucinations before they mislead users.
Implementation Patterns: Integrating Security Frameworks
Adopting the OWASP LLM Top 10 as a compliance baseline requires a structured approach to integration. Organizations often start by mapping each vulnerability to existing security controls or identifying gaps that new tooling must address. A common pattern involves a multi-layered defense incorporating API gateways, specialized LLM vulnerability scanners, and continuous integration/continuous deployment (CI/CD) pipeline hooks for validation. For instance, an API gateway can enforce rate limits and geo-blocking, while an LLM vulnerability scanner inspects prompt and response payloads for malicious patterns, PII, or policy violations. Integrating these controls early in the development lifecycle, through mandatory security gates, ensures that security isn't an afterthought but a foundational element of LLM application design.
Effective implementation also necessitates clear ownership. Security teams are typically responsible for defining policies and evaluating tools, while development teams integrate the safeguards within their applications. A critical implementation pattern involves using configuration-as-code for security policies, allowing version control, automated deployment, and auditing of all security rules. This enables a consistent and repeatable application of the OWASP LLM Top 10 guidelines across an organization's entire LLM portfolio, from prototype to production.
Regulatory Context: EU AI Act and NIST AI RMF Alignment
The operationalization of the OWASP LLM Top 10 must also consider the evolving regulatory landscape. The EU AI Act, for example, classifies AI systems by risk level and imposes stringent requirements for high-risk AI, including robust risk management systems, data governance, cybersecurity, and human oversight. Similarly, the NIST AI Risk Management Framework (RMF) provides a structured approach to managing risks throughout the AI lifecycle, emphasizing governance, mapping, measuring, and managing. Compliance with OWASP LLM Top 10 directly contributes to satisfying sections of these frameworks, particularly those related to cybersecurity (e.g., integrity, confidentiality) and robust, secure system design.
Specifically, the Act's requirements for 'Robustness, security and accuracy' align well with mitigating vulnerabilities like Prompt Injection (LLM01) by ensuring predictable and safe behavior, and 'Cybersecurity' which directly maps to all items concerning data integrity, confidentiality, and system availability. Our platform's capabilities for PII detection (LLM06) and access controls align with requirements for 'Data governance', ensuring data quality and protection. Therefore, robust OWASP LLM Top 10 mitigation is not just good security practice, but a critical component of achieving broader AI regulatory compliance.
Real-World Examples: Preventing Costly Breaches
Consider a financial institution using an LLM for customer service. A prompt injection attack (LLM01) could trick the LLM into disclosing sensitive account information, leading to devastating reputational damage and regulatory fines. Our 'instruction override' and 'role hijack' validators, combined with PII detection (LLM06), would immediately flag and block such attempts. Another scenario involves an e-commerce platform using an LLM for product recommendations. An attacker leveraging insecure output handling (LLM02) could inject malicious JavaScript into product descriptions, leading to cross-site scripting (XSS) attacks on end-users. Output-side validators that strip JS-protocol links and analyze content for suspicious embeds prevent this vector.
Without these specific protections, organizations face not only the direct costs of a breach (forensics, remediation, legal fees) but also the intangible costs of lost customer trust and brand erosion. For example, a major cloud provider recently disclosed an incident where sensitive customer data was accidentally exposed via an LLM due to insufficient output filtering, underscoring the critical need for comprehensive validation mechanisms that extend beyond traditional security perimeters.
Decision Frameworks: Prioritizing Mitigation Strategies
When faced with implementing mitigations for the OWASP LLM Top 10, a structured decision framework helps prioritize actions. We recommend a risk-based approach that evaluates the likelihood and impact of each vulnerability in the context of your specific LLM application. A simple matrix mapping each OWASP item against 'Impact on Business Operations', 'Data Confidentiality', 'Data Integrity', and 'System Availability' can highlight critical areas. For instance, Prompt Injection (LLM01) often has a high impact across all categories, making its mitigation a top priority. Conversely, certain aspects of Supply Chain Vulnerabilities (LLM05) might have varied impact depending on the reliance on third-party models or data providers.
Our platform's ability to quickly deploy and customize rules for each OWASP category allows organizations to adapt their defenses as new threats emerge or as the LLM application evolves. This agility is crucial, as the threat landscape for LLMs is highly dynamic. Prioritization should also consider the effort required for mitigation; some defenses, like basic PII detection, are relatively low-effort/high-impact, while others might require architectural changes or extensive fine-tuning of models.
- Assess likelihood and impact for each OWASP LLM Top 10 item.
- Prioritize high-impact, high-likelihood vulnerabilities first.
- Consider existing security controls versus required new investments.
- Factor in regulatory compliance requirements (e.g., EU AI Act, NIST AI RMF).
- Leverage tooling that offers flexible rule customization and rapid deployment.
Vendor Comparisons: Distinguishing Robust LLM Security
Evaluating LLM security vendors requires looking beyond superficial feature lists. True differentiation lies in the depth of coverage for each OWASP LLM Top 10 category, the flexibility of policy enforcement, and the ability to integrate seamlessly into diverse tech stacks. Many vendors offer basic prompt filtering, but few provide granular control over output validation, dynamic behavioral analysis for prompt injection, or automated discovery and classification of sensitive data in real-time. For instance, when comparing solutions for LLM01 (Prompt Injection), assess whether the vendor uses static pattern matching, semantic analysis, or a combination including anomaly detection against baseline model behavior. Static rules are easily bypassed; advanced solutions use machine learning to detect novel injection techniques.
Another critical differentiator is the ability to handle both pre-deployment (e.g., training data validation for LLM03) and post-deployment (e.g., real-time inference monitoring for LLM01, LLM02, LLM06) security. Our platform excels in offering comprehensive protection across the entire LLM lifecycle, with customizable rules that address specific enterprise needs, from PII redaction to custom instruction injection patterns unique to a given application. This comprehensive approach simplifies compliance efforts and reduces the total cost of ownership for LLM security.