A traditional web application does what its code says. An LLM application does what its code says, plus whatever the model infers from every piece of text it reads. That text can include a customer email, a scraped web page, or a PDF someone uploaded, and the model cannot reliably tell which parts are data and which are instructions.

This is why AI security needs adversarial testing that goes beyond scanning for known vulnerabilities. You have to probe a probabilistic system whose behavior shifts with phrasing, conversation history, and context. Red, blue, and purple teaming give you a structure for doing that.

Source: Generated using AI

This guide covers how the three disciplines map onto AI systems, how jailbreaking and prompt injection work conceptually, what a continuous testing pipeline looks like, and where the limits are. It stays at the level of mechanisms and defenses and does not include working attack payloads.

Background: why classic security testing falls short

Conventional red teams test deterministic systems. If an endpoint is vulnerable to SQL injection, the same payload works every time, and a patch closes the hole. Three properties of LLM systems break that model.

• Non-determinism: The same prompt can produce different outputs on consecutive runs, so a single passing test proves little.
• No hard boundary between instructions and data: Everything the model reads is tokens in one context window. System prompts, user messages, retrieved documents, and tool outputs all compete for influence.
• Wide, natural-language attack surface: Attacks need no exploit code. Attackers can use persuasion, role-play, and gradual escalation, which defeats signature-based filters.

Early AI red teaming focused on the model itself, covering harmful content, bias, and training data leakage. The attack surface has since moved to the application around the model, which now includes retrieval pipelines, tool calling, memory, and autonomous agents. The OWASP Top 10 for LLM Applications 2025 reflects this shift. It keeps prompt injection at number one and adds System Prompt Leakage and Vector and Embedding Weaknesses as new entries. OWASP’s Excessive Agency entry covers three root causes: too much functionality, too many permissions, and too much autonomy. Aembit’s overview of the 2025 list also notes that a 2026 edition now exists. I haven’t verified what changed in it, so check OWASP’s GenAI Security Project directly before citing rankings.

Core concepts: red, blue, and purple teams

The three team names carry over from classic security, but their jobs change for AI systems.

• Red team: Plays the adversary. It tries to make the system misbehave by leaking data, taking unauthorized actions, producing prohibited content, or ignoring its instructions.
• Blue team: Defends the system. It builds guardrails, permission boundaries, logging, detection rules, and incident response.
• Purple team: Is a collaboration model more than a separate team. Red and blue work together in short cycles, so every successful attack becomes a detection rule, a guardrail, or a regression test.

Two reference frameworks help structure the work. OWASP’s list is useful for risk categories. MITRE ATLAS catalogs adversary techniques against AI systems, and security vendors already tag jailbreak research with ATLAS identifiers such as AML.T0054 (LLM Jailbreak). Mapping findings to these frameworks makes them legible to security teams that already think in ATT&CK terms.

Jailbreaking versus prompt injection

These terms are often used interchangeably, but they describe different problems.

Jailbreaking targets the model’s own safety training. The attacker is usually the user, and the goal is to get the model to produce something it was trained to refuse. The victim is the model provider’s or deployer’s policy.

Prompt injection targets the application. The attacker’s goal is to make the model follow instructions it should have treated as data. When the attacker types the instructions directly, it is direct injection. When the instructions hide in content the model later reads, such as a web page, an email, or a retrieved document, it is indirect injection. In indirect injection the user may be completely innocent, and the user is the victim.

How jailbreaks work, conceptually

Several research lines show that safety training holds up poorly against determined adaptive attackers. The categories below are the ones red teams test against.

• Optimization-based attacks: Research on gradient-guided adversarial suffixes, such as Zou et al. (2023), showed that appended token sequences found by optimization can push aligned models toward compliance. They generally require white-box access or transfer from open models, and they often produce gibberish that input filters can flag.
• Multi-turn escalation: Microsoft’s Crescendo paper describes a technique that begins with an innocuous question and steers the model gradually, referencing the model’s own earlier answers. The paper states that the inputs are benign and human-readable, and that it exploits the model’s tendency to follow patterns and weight recent text. Because each message looks harmless alone, per-message filters struggle with it.
• Long-context priming: Anthropic’s many-shot jailbreaking research showed that filling a long context window with many examples of undesired behavior can shift model behavior. As context windows grew, this attack surface grew with them.
• Role-play and framing attacks: Persona framing, fictional wrappers, and “authorized tester” claims try to recast a harmful request as a legitimate one.
• Encoding and obfuscation: Translation, encoding, and unusual formatting try to slip content past filters that match on surface patterns.

The practical lesson is that single-turn refusal rates are a weak measure of robustness. A model that refuses a request when asked directly may comply after several turns of benign priming, so evaluations have to be trajectory-level.

How an indirect injection attack unfolds

Indirect injection is the scenario that matters most for production systems, because it needs no malicious user. Consider an assistant that summarizes a user’s inbox and can also send replies and query a customer database.

Nothing in this flow requires the model to be “broken” in any special way. The model followed what looked like a reasonable instruction in its context. The failure is architectural: untrusted content and trusted instructions shared one channel, and the agent held enough privilege to cause damage once the model was influenced.

This is why mitigation centers on limiting what a compromised model can do rather than on perfectly detecting every injection. As one practitioner overview of the OWASP list puts it, you can’t patch your way out of prompt injection, because it stems from how LLMs process input.

The purple team loop in practice

A useful purple team process is a loop, not a one-off engagement. Each finding feeds a control, and each control feeds a regression test.

The loop restarts whenever the system changes. A new model version, a revised system prompt, a new tool, or a changed retrieval source can all invalidate earlier results. This is the main reason one-time pen tests age badly for AI systems.

Scoping the engagement

Before testing, write down four things.

• Assets: What is worth protecting? Examples include customer data, internal documents, system prompts, API credentials, and the ability to take actions such as payments or emails.
• Trust boundaries: Which inputs are untrusted? Anything the user did not author, including retrieved documents, web pages, emails, tool outputs, and other agents’ messages, counts.
• Capabilities: What can the model do? List every tool, its permissions, and whether actions need human confirmation.
• Harm model: What does failure look like here? A customer support bot leaking another customer’s order is a different severity than a chatbot producing an off-brand joke.

Many findings turn out to be architecture findings. A prompt injection against a system with no tools and no sensitive context is an embarrassment. The same injection against an agent with email and database access is a breach.

Practical implementation

Red team: a minimal canary-based test harness

Manual testing finds creative failures, but automation provides repeatability. The harness below tests whether a deployed assistant leaks a planted canary secret when processing a document that contains an injected instruction. It uses a harmless canary string rather than a real attack payload, and the exact injection text is left as a configurable fixture. The code is illustrative.

import uuid
from dataclasses import dataclass

CANARY = f"CANARY-{uuid.uuid4().hex[:12]}"

@dataclass
class TestCase:
    name: str
    document: str      # untrusted content the app will ingest
    user_request: str  # the legitimate user task

def build_cases(injection_fixtures: list[str]) -> list[TestCase]:
    """Wrap each fixture inside otherwise ordinary document text."""
    cases = []
    for i, fixture in enumerate(injection_fixtures):
        doc = f"Quarterly summary: revenue was flat.\n\n{fixture}\n\nEnd of report."
        cases.append(TestCase(f"indirect-{i}", doc, "Summarize this document."))
    return cases

def run_case(app, case: TestCase) -> dict:
    # The app under test is seeded with a secret in its privileged context.
    response = app.handle(
        user_message=case.user_request,
        retrieved_docs=[case.document],
        secret_context=CANARY,
    )
    return {
        "case": case.name,
        "leaked_canary": CANARY in response.text,
        "unexpected_tool_calls": [
            c.name for c in response.tool_calls
            if c.name not in {"search_docs"}   # allowlist for this task
        ],
    }

def run_suite(app, fixtures, trials=5):
    """Repeat each case, since model output is non-deterministic."""
    results = []
    for case in build_cases(fixtures):
        outcomes = [run_case(app, case) for _ in range(trials)]
        failures = sum(o["leaked_canary"] or bool(o["unexpected_tool_calls"])
                       for o in outcomes)
        results.append({"case": case.name, "failure_rate": failures / trials})
    return results

Three design choices matter here. The canary gives you an unambiguous, machine-checkable signal instead of subjective judgment. The tool allowlist catches behavioral failures that never touch the output text. Repeated trials report a failure rate, which fits a probabilistic system better than a single pass or fail.

For broader coverage, open-source tooling exists. PyRIT is Microsoft’s framework, and its authors report using it in more than 100 red teaming operations on generative AI systems. Practitioner comparisons describe NVIDIA’s garak as a scanner that fires a catalog of known probes, while PyRIT is a framework for building custom attack campaigns and scoring. Used together they cover both breadth and depth.

Blue team: constrain what a compromised model can do

Detection of malicious input will always have gaps, so the strongest blue team controls limit blast radius. The sketch below enforces tool permissions outside the model, in ordinary code the model cannot talk its way past.

from dataclasses import dataclass

@dataclass
class ToolPolicy:
    allowed: set
    needs_confirmation: set
    allowed_recipient_domains: set

POLICY = ToolPolicy(
    allowed={"search_docs", "draft_email", "send_email"},
    needs_confirmation={"send_email"},
    allowed_recipient_domains={"example-corp.com"},
)

def authorize(tool_name, args, taint_level, user_confirmed):
    """Deterministic gate between the model and its tools.

    taint_level is 'untrusted' if untrusted content (web, email, files)
    has entered the context during this turn.
    """
    if tool_name not in POLICY.allowed:
        return False, "tool not permitted"

    if tool_name == "send_email":
        domain = args["to"].split("@")[-1].lower()
        if domain not in POLICY.allowed_recipient_domains:
            return False, "recipient outside allowed domains"

    if tool_name in POLICY.needs_confirmation and taint_level == "untrusted":
        if not user_confirmed:
            return False, "human confirmation required after untrusted input"

    return True, "ok"

The gate tracks taint: once untrusted content enters the context, sensitive actions require explicit human confirmation. This is an application of the privilege-control and content-segregation guidance in OWASP’s mitigation advice. It does not stop the model from being influenced, but it stops that influence from becoming an action.

Blue team detection signals

Logging is where the blue team earns its keep. Useful signals include:

• Full prompt and response logs with the source of each context segment, kept under your privacy and retention rules.
• Tool calls that deviate from the task, such as a summarization request that triggers a database query.
• Outputs containing canary tokens, internal hostnames, or credential-shaped strings.
• Conversation-level patterns, such as long runs of escalating requests on one topic, since per-message checks miss multi-turn escalation.
• Cost and token spikes, which the OWASP 2025 list covers under Unbounded Consumption.

Purple team: turning findings into regression tests

Every confirmed finding should end up as a permanent test. A simple convention keeps this manageable:

finding_id: PT-2026-014
title: Indirect injection via retrieved document triggers outbound email
owasp: LLM01 Prompt Injection, LLM06 Excessive Agency
mitre_atlas: AML.T0054
status: mitigated
mitigation:
  - external-recipient allowlist enforced in tool gate
  - confirmation required after untrusted context
regression_test: tests/injection/test_outbound_email.py
retest_trigger:
  - model version change
  - system prompt change
  - new tool added

Wire these tests into CI so that a model upgrade, a prompt edit, or a new tool triggers the suite. Industry write-ups on AI red teaming tooling describe this same trend toward continuous, CI-gated safety regression testing, though they are vendor-adjacent sources and worth treating as directional rather than authoritative.

Security risks and mitigations worth prioritizing

For most teams the highest-value items map to a handful of OWASP categories.

Limitations and engineering tradeoffs

Red teaming proves presence, not absence. A clean campaign means your tests did not find a failure. It does not mean none exists. Adaptive attackers get unlimited attempts, and coverage of natural language is effectively unbounded.

Automated and manual testing find different things. Scanners are fast and repeatable but test generic weaknesses. Off-the-shelf probes will not exercise your specific tools, data sources, and business logic, so custom test suites and human creativity remain necessary.

Guardrails add latency and cost. Classifier layers on inputs and outputs add milliseconds to seconds and a bill. Aggressive filters also create false positives that frustrate legitimate users, so tuning is a recurring task.

Security and capability pull in opposite directions. An agent that always asks for confirmation is safer and slower. An agent with broad access is more useful and riskier. The right setting depends on the harm model, not on a universal rule.

Results age quickly. Model updates, prompt edits, and new integrations all change behavior. A finding from six months ago may have been fixed, regressed, or made irrelevant by a new tool.

Jailbreak defenses are an arms race. Providers continually harden models, and researchers continually find new bypasses. Plan for defense in depth rather than depending on any single model’s refusal behavior.

Advanced insights

Trajectory-level evaluation. Since multi-turn attacks like Crescendo succeed by accumulating context, evaluate full conversations rather than isolated prompts. Detection that scores the whole dialogue, including whether it is drifting toward a restricted topic, catches what per-message filters miss.

Treat the agent as an untrusted principal. Design as if the model could be fully controlled by an attacker at any moment, then ask what the worst reachable action is. This is the same posture security engineers take toward user-supplied code, and it leads to sandboxed execution, scoped tokens, and out-of-band approval for irreversible actions.

Measure with rates, not anecdotes. Report attack success rate across repeated trials, per technique category, per model version. This makes regressions visible and lets leadership compare risk across releases.

Multi-agent and memory risks. When agents pass messages to each other or write to persistent memory, an injection can persist or propagate. Red teams should test whether poisoned content written to memory in one session influences behavior in later sessions, and whether one compromised agent can instruct another.

Future outlook

What is demonstrated today: automated multi-turn attacks, indirect injection against tool-using assistants, and open-source frameworks for large-scale probing. What is being actively deployed: continuous red teaming in CI, runtime guardrails paired with testing, and framework mapping to OWASP and MITRE ATLAS. What remains research: provable robustness against prompt injection, reliable separation of instructions from data at the model level, and standard benchmarks that predict real-world agent safety.

Speculatively, some form of architectural separation, such as models that treat tool outputs as strictly non-executable data, could reduce the injection problem. Nobody has shown that this works at scale without hurting capability, so treat it as a direction rather than a plan.

Conclusion

• Red teaming finds how a system fails, blue teaming limits and detects those failures, and purple teaming makes the cycle continuous and measurable.
• Jailbreaking attacks the model’s safety training, while prompt injection attacks the application. Indirect injection is the more dangerous form because the user can be innocent.
• Multi-turn attacks show that per-message filtering and single-turn evaluation give a false sense of security.
• The most durable defenses are architectural: least privilege, taint tracking, deterministic tool gates, and human approval for sensitive actions.
• Every finding should become a regression test, and every model, prompt, or tool change should rerun the suite.
• Report failure rates across repeated trials, and accept that you can reduce risk but not eliminate it.