Over the last year, most conversations we have had with security teams about threat modeling have ended up at the same question: can we just have an LLM do it? It is a reasonable question. Threat modeling is slow, it depends on a small number of people, and the systems it is supposed to cover now change faster than those people can review them. At the same time, the systems themselves increasingly contain LLMs and agents, which introduce threats the classic methods were not designed for. AI threat modeling is the term that gets attached to all of this. It covers three things: threat modeling systems that contain LLMs or agents, using an LLM to produce or maintain a threat model, and threat modeling a development process in which coding agents write most of the code.
In this post we look at all three. We cover what changes when a system contains an LLM or an agent, and which of the new frameworks are worth using. We look at what the 2025 and 2026 research says about letting an LLM produce the threat model. And we describe what threat modeling in the AI era looks like when it has to keep up with agent-generated change.
#What "AI threat modeling" means
It helps to separate the three before going further.
The first is threat modeling systems that contain AI. A product gains an LLM feature, a retrieval pipeline, an agent with tools or an MCP server, and some of the assumptions behind STRIDE no longer hold. This is where most of the new frameworks are aimed.
The second is using AI to produce or maintain the threat model. Tools such as STRIDE GPT, AWS Threat Designer and the various vendor copilots fall into this category. The question here is whether the output is reliable enough to act on, and the answer depends almost entirely on how the model is grounded and reviewed.
The third is threat modeling the software development lifecycle itself, now that a large share of code is written by coding agents, a practice often called vibe coding. Securing vibe coding is a question of where the review happens, which this post comes back to. Veracode's October 2025 update of its GenAI Code Security Report found that AI-generated code introduced security flaws in 45% of tests, with no improvement from larger or newer models. Design review and code review were sized for human output, and neither keeps up with the volume an agent produces. Whatever threat model exists has to be applied at the points where that change enters the system, or it will be bypassed by default.
Most writing on the topic covers the first and touches the second. We think the third decides whether the other two matter, so we come back to it at the end.
#What changes when a system contains an LLM
Microsoft published guidance on threat modeling AI applications in February 2026 which names three properties of AI components that break classic assumptions. We find it a useful starting point.
The first is nondeterminism. The same input does not reliably produce the same output, so a threat model has to reason about ranges of behaviour rather than single outcomes. A test that shows the model refused a malicious request once is not evidence of a control.
The second is that instructions and data arrive on the same channel. A model treats conversation and instruction as one input stream, so any text it reads, including an email, an issue body or a tool description, can be interpreted as something to do. Microsoft's phrasing is that text can be read as "executable intent". In practice this means that wherever an LLM reads content an outsider can influence, filtering that content is not a sufficient control. The control has to be on what the model is able to do afterwards.
The third is that the system expands through tools and memory. An agent with a GitHub token, a file system and a memory store has a much larger reach than the chat interface suggests. Microsoft's conclusion is that AI threat modeling "focuses on limiting blast radius rather than enforcing perfect behavior", which matches our experience.
Two heuristics have emerged for reasoning about that blast radius quickly.
Simon Willison's lethal trifecta describes the combination of an agent that reads untrusted content, has access to private data and can communicate externally. Any agent with all three properties can be turned into an exfiltration channel by a suitably crafted input. Meta's Agents Rule of Two turns this into a design rule. Within a single session, an agent should have at most two of:
- untrusted input
- access to sensitive systems or private data
- the ability to change state or communicate externally
An agent that needs all three should not run unsupervised.
Willison points out a gap: the rule allows an agent with untrusted input and the ability to change state, and that pairing alone can still do damage. We treat the rule as a cheap check that every agent design should pass, rather than as a complete review.
#Four AI agent and MCP security incidents
The incidents below are the ones we use when explaining why this matters. None of them involved a code-level bug that a scanner would have found. In each case the problem was a design decision about what could reach the agent, what it could read and what it was allowed to do. The Amazon Q case is the vendor side of the AI tooling supply chain; we covered the developer side in an earlier post on what coding assistants ship into repositories.
| Incident | What happened | The design question that would have caught it |
|---|---|---|
| EchoLeak, CVE-2025-32711, Microsoft 365 Copilot, disclosed June 2025 | A single crafted email triggered zero-click data exfiltration by chaining bypasses of the prompt-injection classifier, link redaction and the content security policy | Where does untrusted external content meet internal data, and what can the model emit afterwards? |
| GitHub MCP exploit, Invariant Labs, May 2025 | A malicious issue in a public repository hijacked an agent whose GitHub credential could also read private repositories, and private repository data was leaked into a public pull request | What can the agent's credential reach, and does the agent read anything the public can write? |
| MCP tool poisoning, Invariant Labs, April 2025 | Instructions hidden in tool descriptions caused Cursor to exfiltrate SSH keys and MCP configuration files | Is tool metadata on the data-flow diagram as an untrusted input? |
| Amazon Q Developer extension, CVE-2025-8217, July 2025 | An over-scoped GitHub token in CI allowed an attacker to ship a "wiper" prompt in a release to an extension with nearly a million marketplace installs. The payload failed on a syntax error | Is the AI tooling supply chain, including its CI permissions, in the model at all? |
The GitHub case is worth looking at in detail because nothing in it malfunctioned. The model, the MCP server and GitHub each behaved as designed. The flaw was the combination of a public write surface (issues) feeding an agent with a private read scope and a public write scope (pull requests). That combination is visible on a data-flow diagram, and it would have been visible months before the exploit was published.
#Which frameworks to use for threat modeling AI agents
The last two years have produced a lot of frameworks for threat modeling AI agents and LLM applications. The problem is no longer finding one but choosing a small enough set that teams will actually use. The table summarises what each one is for.
| Framework | What it is | When to use it |
|---|---|---|
| Four questions (Threat Modeling Manifesto) | What are we working on, what can go wrong, what are we going to do about it, did we do a good job | Always, as the structure for everything else. Adam Shostack describes it as the technique "holding up best under the disruption" |
| STRIDE | Classic threat taxonomy | The conventional components: APIs, authentication, storage, infrastructure. It does not cover goal hijack or agent cascades |
| OWASP Top 10 for LLM Applications 2025 | Risk catalogue for LLM features | A coverage check for chat, retrieval and completion features. Prompt injection is LLM01 |
| OWASP Top 10 for Agentic Applications 2026 | Risk catalogue from ASI01 Agent Goal Hijack to ASI10 Rogue Agents, published December 2025 | Any agent that calls tools or holds memory. It introduces the principle of "least agency" |
| CSA MAESTRO | Seven-layer method from foundation model to agent ecosystem, with cross-layer analysis | Complex multi-agent platforms. It is heavy for a single feature |
| PHANTOM-B | Shostack's elicitation list: prompt injection, hallucination, anthropomorphising, non-explainability, training issues, overreliance, missing security engineering, bias | A short LLM-specific pass during design. It is intended to sit alongside STRIDE |
| MITRE ATLAS | Adversary tactics and techniques against AI systems | Validating coverage and planning red team work. It is not a design method |
| Lethal trifecta, Rule of Two | Design heuristics | A quick gate on every agent design and every agent plan |
Our recommendation:
- The four questions plus STRIDE for the conventional parts of the system.
- The Agentic Top 10 for anything that calls tools or holds memory.
- PHANTOM-B or the LLM Top 10 for pure LLM features.
- The trifecta check on every agent.
- MAESTRO only for genuinely multi-agent platforms, where its layered method earns its overhead.
In all cases we would rather see the AI-specific prompts built into one living model than run as a separate exercise, because separate exercises get skipped.
#Using an LLM to produce the threat model
Using an LLM to produce the threat model is the part of the question we get asked about most, so it deserves evidence rather than opinion. This is LLM threat modeling in the second sense above. The research from 2025 and 2026 is still early, but it points in one direction from several angles.
An October 2026 preprint, From Requirements to Attack Trees, compared an architecture-informed agent with a single-shot LLM on two synthetic systems. On the stronger of the two, the agent scored 0.97 on coverage, 0.84 on grounded precision, 1.00 on traceability and 0.91 on actionability, against 0.44, 0.24, 0.00 and 0.06 for the single-shot LLM. The second system shows the same ordering with a smaller gap.
On a Microsoft reference threat model, grounded coverage was 0.83 for the agent, 0.17 for the single-shot LLM and 0.11 for single-shot STRIDE. This is one preprint on two small cases, so we would not lean on the exact figures, but the gap is large enough to take seriously.
A Carleton University study had LLMs classify 5G threats into STRIDE categories and found 63% accuracy zero-shot and 71% few-shot with web search, with a ceiling of 72%. The authors note hallucination and missed second-order threats, meaning threats that only appear when two components interact, as the main failure modes.
A banking-system study reported that STRIDE GPT struggles with consistent categorisation and produces unstable results. Fine-tuning combined with prompt engineering raised mitigation accuracy on an 8B model from 0.36 to 0.69, which gives some idea of how much the grounding contributes relative to the model.
Practitioner experience is consistent with this. Adam Shostack wrote up lessons from his LLM threat modeling course in April 2026. The same ChatGPT prompt produced 20 threats on one run and 44 on another. Some outputs ran past fifty pages, and as he puts it, "no one wants to review the output". One of his students remarked that before the course they would have accepted the output as a good threat model.
Finally, an ESEM 2026 study applied three GenAI-aware threat modeling methods, with human analysts, to a real LLM-augmented business system. None of the methods covered supply chain, improper output handling, excessive agency, system prompt leakage or misinformation from the OWASP LLM list. The developers surveyed preferred the tool-supported method. One of them asked for something we hear regularly, to "just connect it to our tickets", adding that this alone would make a huge difference.
#Why single-prompt threat models fall short
Reading the studies together, the failures of "give me a threat model for this" are not really about model capability. Three things go wrong.
The prompt rarely contains the architecture. Without the data flows, trust boundaries, token scopes and deployment details, the model produces the generic STRIDE list for a web application with a database. The grounded precision of 0.24 in the benchmark above is a measurement of that effect.
The output is unstable and too long to review. If the same system produces 20 threats one day and 44 the next, nobody can tell whether the system changed or the sampling did. A fifty-page document is not reviewed in practice, and an unreviewed threat model that has been signed off is misleading, because it records a review that did not take place.
Nobody owns the result. A threat model is a set of decisions about which risks are accepted, which controls are built and who verifies them. An LLM cannot accept risk on an organisation's behalf. Unless the output becomes a requirement that a named person is accountable for, it has no effect on what is shipped.
The variable that matters is therefore grounding, scope and review rather than whether an LLM is involved. Narrow tasks tied to real artifacts, traced back to a source and checked by a person work well in both the research and our own use. Broad prompts do not. That is an argument about where to put the LLM in the process, not an argument against using one.
#Reykur's approach to keeping the model current and enforced
Our approach, which we call continuous threat modeling, rests on three ideas. The model describes design intent, and the implementation is verified against it. The model is maintained as change happens, from whichever source the change comes from. And the model understands each system in the context of the organisation around it, so it can make the connections between systems and their components that single-system diagrams miss.
#Intent, verified against implementation
A threat model is a statement of intent, usually produced in a design review: these are the threats we expect, and these are the controls we have decided to build against them. It does not by itself say anything about what was shipped.
We therefore verify the implementation against the model when the code lands, in both directions. Were the controls we decided on built, or were some missed? And did the implementation introduce threats the model did not anticipate, for example because a dependency, a token scope or a data flow differs from the design? Both questions apply to human-written code. They apply more often to agent-written code, because the implementation is produced faster than it can be read.
#Updated from every source of change
A model that is accurate on the day of the review and not updated afterwards describes a system that no longer exists. We update the model from the events that change the system: a new story in the tracker, an agent's plan for a feature, a merged pull request, and a dependency on another team's service being added or removed. Each of these triggers a check of the intent against the new state.
#Context across systems
A model of one system in isolation does not hold the facts that matter. In the GitHub MCP case the facts were about the edges between systems: the issue tracker is publicly writable, the agent's credential can read private repositories as well as the public one, and a pull request is a public write. The model is therefore a graph of the organisation's systems and the connections between them and their components, not a set of per-system diagrams.
This context is what makes shared services manageable. A platform team typically runs the identity provider, the API gateway, the secrets manager and the deployment pipeline, yet every product model still receives the full set of authentication threats and every team re-mitigates or accepts them. In our model a provider declares which threats it handles for the systems that depend on it, as a deterministic policy rather than a judgement call. Those systems are covered automatically, can opt out where the dependency does not apply, and lose the coverage if the provider's controls stop being verified.
The organisation's security policy sits alongside: which systems hold crown-jewel data, how often each tier is reviewed, which categories of risk may never be accepted, and the approval required for the rest.
The graph and the policy are what the analysis and plan-review agents receive as context. Two instructions go with them. Do not propose that a system re-implement what a provider already handles. Do emit the threats in the integration itself, such as how the system validates the tokens it receives.
#Where it is enforced
These three ideas only have an effect if the model is checked at the points where change enters the system. With coding agents, those points occur earlier in the lifecycle than they used to.
The first point is the ticket. When a story is created, the model already knows which systems it touches and which threats a provider already covers, so the intended controls are written into the issue as acceptance criteria. This is also what the developers in the ESEM study asked for.
The second is the agent's plan. The plan is a design document, and it is the point at which a design flaw can still be corrected before any code is written. An agent asked to build a GitHub issue triage bot will typically propose authenticating the MCP server with whatever organisation token is available, reading issue bodies and opening pull requests. That reproduces the Invariant Labs exploit, and a review against the graph identifies it from facts the graph already holds. We describe this plan-time review for secure vibe coding in more detail on the use-case page.
The third is the pull request, where the implementation is verified against the intent: the controls from the ticket are confirmed as built, and the change is checked for threats the design did not anticipate. The same verification keeps shared-service coverage accurate, since a provider's declared controls have to remain verified for its consumers to remain covered.
The fourth is the evidence record: each requirement, its origin in the model, the ticket it was attached to and the pull request that verified it. For the Cyber Resilience Act, SOC 2 or ISO 27001, this is the material an auditor asks for, and it is produced by the process rather than assembled separately.
This is the loop behind continuous threat modeling at Reykur. The method does not depend on a product. What tooling adds is that the check runs on every ticket, every plan and every pull request, and that coverage from shared services is kept accurate as providers change.
#The Cyber Resilience Act and continuous threat modeling
The regulatory direction is consistent with the engineering argument. The EU Cyber Resilience Act requires manufacturers to document a cybersecurity risk assessment and to take it into account during planning, design, development, production, delivery and maintenance. The reporting obligations have applied since 11 September 2026, including for products already on the market, and the main obligations apply from 11 December 2027. Commission guidance ties the concept of a "substantial modification" to a change that alters risk in a way the manufacturer had not already considered in its risk assessment.
With coding agents in the picture, that clause has a practical consequence. If the risk assessment is a document from the last design review, every agent-generated change that touches a trust boundary is a change that was not considered. A model that is updated from the ticket and verified at the pull request is the most straightforward way to demonstrate that it was.
#What threat modeling in the AI era requires
AI has not made threat modeling obsolete. It has made the once-a-year workshop version insufficient, because the system changes too often for a static document to describe it, and it has made the single-prompt LLM version unreliable, because the output is ungrounded, unstable and unowned. What holds up is the four-question method, applied by people who own the system. It is extended with the agent-specific threat catalogues where LLMs and tools are involved, and it is kept current and enforced at the ticket, the plan and the pull request. The research so far supports using LLMs for the grounding and upkeep of that model. It does not support using them as the source of truth.
#FAQ
#What is AI threat modeling?
The term covers three things. The first is threat modeling systems that contain AI components, which requires additional threat categories for prompt injection, goal hijack, tool misuse and agent-to-agent trust. The second is using AI to draft or maintain a threat model, which works well when the model is grounded in the real architecture and reviewed by people, and poorly when it is a single prompt. The third is threat modeling a development process in which coding agents produce most of the change, which requires the model to be checked at the ticket, the plan and the pull request.
#Can ChatGPT do a threat model?
It can produce a document that resembles one. The available studies and practitioner reports show that single-prompt threat models vary substantially between runs, score low on precision against the real architecture and have no traceability to source artifacts. Used as a grounded assistant on a narrow, reviewed task, an LLM is useful. Used as the sole source of the model, it is not.
#MAESTRO vs MITRE ATLAS: which one do I need?
MAESTRO is a seven-layer method for agentic AI threat modeling on complex multi-agent systems. ATLAS is a knowledge base of adversary techniques observed against AI systems. Use MAESTRO to structure a design review of a multi-agent platform and ATLAS to validate coverage and plan red teaming. For a single agent feature, the OWASP Agentic Top 10 together with the trifecta check is usually sufficient.
#How do you threat model an MCP server?
MCP threat modeling starts by treating the server as a trust boundary and its tool descriptions as untrusted input. Establish what the server's credential can reach, which of its tools write or communicate externally, whether tool definitions can change after approval, and whether anything the agent reads through it can be written by outsiders. Then apply least agency: separate read and write tokens, scope them to the smallest resource, and require human approval for destructive actions.
#References
- Microsoft Security, Threat modeling AI applications (February 2026)
- OWASP GenAI Security Project, Top 10 for Agentic Applications 2026 and Top 10 for LLM Applications 2025
- Ken Huang, Cloud Security Alliance, Agentic AI Threat Modeling Framework: MAESTRO (February 2025)
- Adam Shostack, PHANTOM-B talk summary and Lessons from Threat Modeling Intensive Using LLMs (2026)
- Simon Willison, The lethal trifecta for AI agents (June 2025) and notes on the Rule of Two (November 2025)
- Meta AI, Agents Rule of Two: A Practical Approach to AI Agent Security (October 2025)
- Threat Modeling Manifesto
- Reddy and Gujral, EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System, arXiv 2509.10540
- Invariant Labs, GitHub MCP exploited and MCP tool poisoning attacks (2025)
- AWS Security Bulletin AWS-2025-015, Amazon Q Developer extension
- From Requirements to Attack Trees: Grounded LLM Agents for Design-Time Security Review, arXiv 2610.03820 (October 2026, preprint)
- LLMs' Suitability for Network Security: A Case Study of STRIDE Threat Modeling, arXiv 2505.04101
- ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking System, arXiv 2411.17058
- Díaz Ferreyra et al., Emerging Challenges in Threat Modeling for GenAI-Augmented Systems: A View from the Trenches, ESEM 2026, arXiv 2607.28431
- Veracode, GenAI Code Security Report (October 2025 update)
- European Commission, Cyber Resilience Act and Guidelines on the Cyber Resilience Act (July 2026)