What is an LLM vulnerability scanner?

An LLM vulnerability scanner sends batteries of adversarial probes — jailbreaks, prompt injections, PII extraction attempts, harmful-content requests — at a target LLM and grades the responses. The output is a vulnerability + optimization report with per-probe evidence, severity, and a prioritized fix list.

Which LLMs can I scan with FilterPrompt?

OpenAI, Anthropic, Google Gemini, Azure OpenAI, plus any OpenAI-compatible endpoint — Ollama, Groq, Mistral, Together AI, OpenRouter, Perplexity, Hugging Face, vLLM, or your own custom endpoint. Bring your own keys per tenant.

What kinds of vulnerabilities does FilterPrompt test for?

Jailbreaks (DAN, role hijack, translation smuggling), direct and indirect prompt injection, system-prompt extraction, harmful-content compliance, PII / secret leakage, bias & fairness, RAG poisoning, agent/tool abuse, output quality, and robustness — categories map to the OWASP LLM Top 10.

How are probes graded?

Each probe declares an evaluator: regex match, refusal-check, contains-check, or an AI judge (Gemini 3 Flash). Pass/fail comes with severity, category, the exact prompt sent, the model's full response, and the evaluator's reason — fully auditable.

How much does a scan cost?

1 credit per probe executed. New accounts get 1 welcome credit on signup. Pay-as-you-go credit packs after that — credits never expire. Connecting LLMs and creating tenants is free.

Prompt Injection Testing: How to Test Your LLM for OWASP LLM01

Guide · 2026-07-10 · 8 min read · FilterPrompt Security Team

A hands-on guide to prompt injection testing — how to test for prompt injection with open source and managed tools, what to look for, and how to score results.

What prompt injection testing actually is

Prompt injection testing is the LLM equivalent of a pentest. You send a curated library of adversarial prompts to your model and check whether it follows the attacker's instructions or holds to yours. The output is a report: which attacks worked, which failed, and what evidence you have.

It is different from unit testing your prompts for quality. Quality tests ask 'did the model answer well?'. Prompt injection testing asks 'did the model refuse to break its rules under adversarial pressure?'.

How to test for prompt injection: the 5-step workflow

Define your policy — what must the model never do? (leak system prompt, call a dangerous tool, disclose PII, insult users)
Pick a probe library — Garak, PyRIT, Promptfoo, or a managed scanner like FilterPrompt.
Point the tool at your target — the actual production endpoint, not the raw model.
Run and judge — for each probe, capture the response and let a judge model decide success/failure against your policy.
Report and remediate — export a report, add firewall rules or system prompt hardening, re-run.

The 8 prompt injection techniques you must test

Choosing a prompt injection testing tool

Open source scanners (Garak, PyRIT, Promptfoo) give you full control but require engineering to run at scale and produce compliance-ready reports. Managed scanners (FilterPrompt, Lakera Red, NeuralTrust) trade some flexibility for a one-click experience: connect your endpoint, pick a probe set, get a report with evidence.

Scoring: what pass/fail really means

Do not accept a single 'pass rate' number. A credible prompt injection testing report breaks results down by technique, severity, and remediation status. It also shows the exact prompt sent, the response returned, and the judge's reasoning — without that evidence you cannot fix anything or defend the report to an auditor.

How often to test

Before every production release of a new prompt, model, or retrieval index.
Weekly as a regression scan — catches provider model updates that quietly change behaviour.
After any security incident involving the LLM path.
Quarterly with a broader red-team scope for compliance evidence.