What is an LLM vulnerability scanner?

An LLM vulnerability scanner sends batteries of adversarial probes — jailbreaks, prompt injections, PII extraction attempts, harmful-content requests — at a target LLM and grades the responses. The output is a vulnerability + optimization report with per-probe evidence, severity, and a prioritized fix list.

Which LLMs can I scan with FilterPrompt?

OpenAI, Anthropic, Google Gemini, Azure OpenAI, plus any OpenAI-compatible endpoint — Ollama, Groq, Mistral, Together AI, OpenRouter, Perplexity, Hugging Face, vLLM, or your own custom endpoint. Bring your own keys per tenant.

What kinds of vulnerabilities does FilterPrompt test for?

Jailbreaks (DAN, role hijack, translation smuggling), direct and indirect prompt injection, system-prompt extraction, harmful-content compliance, PII / secret leakage, bias & fairness, RAG poisoning, agent/tool abuse, output quality, and robustness — categories map to the OWASP LLM Top 10.

How are probes graded?

Each probe declares an evaluator: regex match, refusal-check, contains-check, or an AI judge (Gemini 3 Flash). Pass/fail comes with severity, category, the exact prompt sent, the model's full response, and the evaluator's reason — fully auditable.

How much does a scan cost?

1 credit per probe executed. New accounts get 1 welcome credit on signup. Pay-as-you-go credit packs after that — credits never expire. Connecting LLMs and creating tenants is free.

AI Penetration Testing — The Complete 2026 Guide

Guide · 2026-01-24 · 13 min read · FilterPrompt Security Team

What AI penetration testing is, how it differs from traditional pentesting, the 8 attack classes to test, and the tools that make it repeatable.

'AI penetration testing' is the highest-CPC AI security query on Semrush at $25.72, and for good reason — buyers who type it are staring at a launch date for a customer-facing LLM feature and need proof that the model won't leak a customer's data on day one. This guide answers the four things they actually want to know: what AI pentesting includes, how it differs from a normal pentest, what tools to use, and what a good report looks like.

What is AI penetration testing?

AI penetration testing is the discipline of simulating attacks against an AI system to find vulnerabilities before real adversaries exploit them. The scope is broader than 'prompt attacks' — it includes the model, the system prompt, the tools/functions the model can call, the vector store powering RAG, and the training or fine-tuning data. A good AI pentest reports specific attack chains, not just probe pass/fail counts.

The industry sometimes uses 'AI red teaming' interchangeably. In practice: red teaming implies open-ended human adversarial creativity; pentesting implies a structured, repeatable methodology mapped to a framework (OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF).

How does AI pentesting differ from traditional pentesting?

The probabilistic nature of LLMs is the biggest operational difference. A traditional pentester replays a payload and gets a repeatable result; an AI pentester runs each probe N times and reports a probability of failure. That's why scanner tools matter — running 1,000 probes 5 times each by hand is impossible.

The 8 attack classes to cover

Prompt injection (direct + indirect) — LLM01. The most-exploited class.
Jailbreak — bypassing safety alignment (DAN, DUDE, role-play).
Sensitive data disclosure — LLM06. System-prompt leakage, PII leakage, secret exfiltration.
Insecure output handling — LLM02. LLM output executed downstream as SQL, HTML, or shell.
Model denial-of-service — LLM04. Context-window exhaustion, expensive generations.
Supply-chain — LLM05. Compromised model weights, poisoned fine-tuning data, unsafe plugins.
Excessive agency — LLM08. Agents given too many tools; unauthorized calendar/email/DB actions.
Training-data poisoning + RAG poisoning — LLM03 + LLM07. Injected instructions in retrieved chunks.

The 6-step AI pentest methodology

Threat model — inventory models, prompts, tools, data flows, and trust boundaries.
Baseline scan — run an automated OWASP LLM Top 10 probe battery to catch low-hanging fruit.
Targeted probing — hand-craft attacks specific to your system prompt and tools.
Exploit chaining — combine probes into realistic attack scenarios (e.g., indirect injection → unauthorized email send).
Impact analysis — quantify blast radius per finding (data exposed, dollars at risk).
Report + retest — deliver evidence, remediation guidance, and re-scan after fixes.

Automated vs human-driven AI pentesting

Automated pentesting (scanners) delivers breadth: 1,000+ probes covering every OWASP LLM Top 10 category, repeatable weekly, and cheap enough to run on every deploy. Human pentesting delivers depth: novel multi-turn attack chains, culturally-specific jailbreaks, and adversarial creativity a probe library can't reach. The right split for most teams: automated scans on every model change, human engagement once or twice a year.

Tooling landscape 2026

What a good AI pentest report contains

The three sections auditors check: (1) executive summary with risk scoring mapped to OWASP LLM Top 10 or NIST AI RMF, (2) per-finding evidence — the full prompt and response chain that triggered the vulnerability, and (3) prioritized remediation with owners and target dates. Anything less makes the report un-actionable and hard to use for a SOC-2 or ISO-42001 audit.

Start today

The cheapest way to close the biggest 60% of your LLM risk: run an automated scan against your model tomorrow. FilterPrompt Scanner ships the OWASP LLM Top 10 probe battery, agentic tool-abuse tests, and audit-ready PDF reports — you can be scanning in five minutes.