What is an LLM vulnerability scanner?

An LLM vulnerability scanner sends batteries of adversarial probes — jailbreaks, prompt injections, PII extraction attempts, harmful-content requests — at a target LLM and grades the responses. The output is a vulnerability + optimization report with per-probe evidence, severity, and a prioritized fix list.

Which LLMs can I scan with FilterPrompt?

OpenAI, Anthropic, Google Gemini, Azure OpenAI, plus any OpenAI-compatible endpoint — Ollama, Groq, Mistral, Together AI, OpenRouter, Perplexity, Hugging Face, vLLM, or your own custom endpoint. Bring your own keys per tenant.

What kinds of vulnerabilities does FilterPrompt test for?

Jailbreaks (DAN, role hijack, translation smuggling), direct and indirect prompt injection, system-prompt extraction, harmful-content compliance, PII / secret leakage, bias & fairness, RAG poisoning, agent/tool abuse, output quality, and robustness — categories map to the OWASP LLM Top 10.

How are probes graded?

Each probe declares an evaluator: regex match, refusal-check, contains-check, or an AI judge (Gemini 3 Flash). Pass/fail comes with severity, category, the exact prompt sent, the model's full response, and the evaluator's reason — fully auditable.

How much does a scan cost?

1 credit per probe executed. New accounts get 1 welcome credit on signup. Pay-as-you-go credit packs after that — credits never expire. Connecting LLMs and creating tenants is free.

NIST AI RMF, OWASP LLM Top 10 and MITRE ATLAS: The Technical Backbone Behind Every AI Law

Frameworks · 2024-11-05 · 10 min read · FilterPrompt Security Research

Laws set the obligation; frameworks set the how. Here's how NIST AI RMF, OWASP LLM Top 10 (2025) and MITRE ATLAS stack together into a single continuous LLM testing program.

Laws like the EU AI Act, U.S. executive orders and India's DPDP Act create the legal obligation to secure AI systems. They rarely specify how. That gap is filled by technical frameworks — NIST AI RMF, OWASP's Top 10 for LLM Applications, and MITRE ATLAS — which auditors, regulators and security teams treat as the de facto checklist for demonstrating compliance.

OWASP LLM Top 10 (2025) — the categories every scanner must cover

MITRE ATLAS — the attacker's playbook

ATLAS is structured like ATT&CK: tactics (reconnaissance, initial access, ML model access, execution, persistence, defence evasion, discovery, collection, exfiltration, impact) with techniques underneath. For LLM teams, the relevant technique groups include AML.T0051 (LLM Prompt Injection), AML.T0054 (LLM Jailbreak), AML.T0057 (LLM Data Leakage), and AML.T0053 (LLM Plugin Compromise). A serious LLM vulnerability scanner reproduces these techniques as executable probes, not just descriptions.

Why continuous scanning is the practical requirement

AI systems change constantly — foundation models are updated, system prompts are edited, RAG content is refreshed weekly, and new agent tools or MCP servers are connected. Each change can silently reopen a previously closed vulnerability. A one-time penetration test cannot certify a system that mutates weekly. This is the technical justification for continuous, automated scanning against the OWASP LLM Top 10 categories, with findings mapped to MITRE ATLAS techniques and reported against the NIST AI RMF's four functions.

How the frameworks stack together in practice

Govern (NIST) — write policy that references OWASP LLM Top 10 as the minimum bar.
Map (NIST) — inventory every LLM, agent, MCP server and RAG pipeline; map each to ATLAS tactics.
Measure (NIST) — run continuous probes covering LLM01–LLM10 and report by ATLAS technique.
Manage (NIST) — block, redact or roll back on regression; retain audit logs per your governing law.