What is an LLM vulnerability scanner?

An LLM vulnerability scanner sends batteries of adversarial probes — jailbreaks, prompt injections, PII extraction attempts, harmful-content requests — at a target LLM and grades the responses. The output is a vulnerability + optimization report with per-probe evidence, severity, and a prioritized fix list.

Which LLMs can I scan with FilterPrompt?

OpenAI, Anthropic, Google Gemini, Azure OpenAI, plus any OpenAI-compatible endpoint — Ollama, Groq, Mistral, Together AI, OpenRouter, Perplexity, Hugging Face, vLLM, or your own custom endpoint. Bring your own keys per tenant.

What kinds of vulnerabilities does FilterPrompt test for?

Jailbreaks (DAN, role hijack, translation smuggling), direct and indirect prompt injection, system-prompt extraction, harmful-content compliance, PII / secret leakage, bias & fairness, RAG poisoning, agent/tool abuse, output quality, and robustness — categories map to the OWASP LLM Top 10.

How are probes graded?

Each probe declares an evaluator: regex match, refusal-check, contains-check, or an AI judge (Gemini 3 Flash). Pass/fail comes with severity, category, the exact prompt sent, the model's full response, and the evaluator's reason — fully auditable.

How much does a scan cost?

1 credit per probe executed. New accounts get 1 welcome credit on signup. Pay-as-you-go credit packs after that — credits never expire. Connecting LLMs and creating tenants is free.

LLM Security Testing: The Complete 2026 Methodology

Guide · 2026-07-17 · 11 min read · FilterPrompt Security Team

A practical LLM security testing methodology — threat modelling, probe design, judge configuration, evidence collection, and mapping results to OWASP LLM Top 10.

Why LLM security testing is different

Traditional application security testing looks for known code-level vulnerabilities: SQL injection, XSS, IDOR. LLM security testing looks for behavioural vulnerabilities: does the model follow its instructions under adversarial pressure? Does it leak data? Does it call tools it shouldn't?

Behavioural testing is probabilistic. The same prompt can succeed and fail on consecutive runs. Your methodology must account for stochasticity — repeat probes, aggregate results, and report confidence intervals rather than binary verdicts.

Phase 1: Threat modelling

Identify sensitive assets the model can access — data, tools, downstream systems.
Enumerate abuse cases per OWASP LLM Top 10 category.
Rank by business impact — data exfiltration and RCE via tool abuse beat mild output issues.
Document a testable policy: what must the model never do?

Phase 2: Probe generation

A probe is an adversarial input designed to elicit a policy violation. Good probes are diverse (technique, domain, modality), targeted (aligned to your policy), and reproducible (deterministic where possible).

Phase 3: Execute and judge

For each probe, capture the full request/response including tool calls. Then run a judge: an LLM configured with your policy and rubric that decides pass/fail with a rationale. The judge is the highest-leverage part of the pipeline.

Phase 4: Report and remediate

The report is the deliverable. It must contain per-probe evidence (prompt, response, judge rationale), aggregated metrics (pass rate by category), OWASP LLM Top 10 mapping, and prioritised remediation actions. Anything less is a demo, not a security test.

Group findings by OWASP LLM Top 10 category.
Assign severity based on impact × exploitability.
Propose a control for each — system prompt patch, firewall rule, retrieval filter, or code change.
Re-run the affected probes after remediation and record the delta.

Continuous LLM security testing

One-shot testing is worthless within 30 days — providers update models, retrieval indexes drift, and attackers publish new techniques. Wire the same probe set into a weekly scheduled scan and alert on regression.