Risk Management & Threat Modeling Basics (Now with 73% More Drama)

"Risk is what happens when reality refuses to follow your slide deck." — every CISO ever, probably

---

Why You're Here (and Not Rewriting Your Policy Doc)

We already danced with the frameworks: NIST CSF and ISO/IEC 27001 gave us the vibe-check for governance, and MITRE ATT&CK + defense-in-depth showed us how attackers party hop from initial access to exfil. Today we marry all that energy into something practical: actually deciding what to protect and how worried to be.

Welcome to risk management and threat modeling — the part where we turn "vibes" into "priorities" and use structured paranoia to keep systems (and careers) alive. Also: AI-driven threats are that new chaotic friend who is both helpful and terrifying. So yes, we’ll make space for their… personality.

---

Quick Glossary That Saves Meetings

• Asset: The thing you’d be sad about losing. Data, models, services, brand reputation, uptime.
• Threat: A thing that can go wrong on purpose. Attackers, malware, insider risk, AI misuse.
• Vulnerability: The bug/weakness that lets the bad thing happen.
• Likelihood: Chance that the bad thing tries and succeeds.
• Impact: How hard it hits when it lands.
• Risk: The combo meal. Often simplified as: Likelihood × Impact.
• Control: What you do to make the bad thing less likely or less painful.

Pro move: Distinguish inherent risk (no controls) vs residual risk (after controls). AI often lowers effort for attackers → higher inherent risk.

---

Threat Modeling: The Structured Paranoia Toolkit

Threat modeling = systematically asking: What are we building? What can go wrong? What are we going to do about it? Did we actually do it?

Popular Approaches (aka Choose Your Fighter)

• STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) — simple, mnemonic, great for app-by-app analysis.
• PASTA — 7-stage, risk-driven, aligns well with business impact. More pasta, more process.
• Attack Trees — visualize how someone could own you. Root: attacker goal; branches: paths.
• DREAD/OWASP Risk Rating — quick threat scoring.
• FAIR — if you like quantifying in dollars and stress.

Here’s STRIDE with an AI twist:

STRIDE	What It Means	Classic Example	AI-Driven Flavor
Spoofing	Pretending to be someone else	Stolen creds	Synthetic voice deepfake to bypass voice auth
Tampering	Messing with data	Config change	Data poisoning in training set
Repudiation	Denying you did the thing	No logs	LLM agent executes tasks without solid audit trail
Info Disclosure	Leaking secrets	S3 misconfig	Prompt injection exfiltrates secrets via LLM
DoS	Making it unavailable	SYN flood	Model-serving saturation with adversarial prompts
EoP	Getting extra powers	Local root	Prompt escalation to invoke hidden LLM tools

---

The 7-Step Flow (Now With Framework Callbacks)

1) Define Scope & Crown Jewels

• Systems, data, models, APIs, third parties.
• Tie to NIST CSF Identify (ID.AM, ID.RA) and ISO 27001 asset management.
• Question: If it vanished at 3 a.m., who screams?

2) Decompose the System

• Draw a quick data flow diagram (DFD): users → front-end → API → model → DB → logs.
• Trust boundaries: where auth changes, where data leaves your control (e.g., third-party LLM API).

3) Identify Threats (Use Multiple Lenses)

• STRIDE across each DFD element.
• MITRE ATT&CK to check for realistic TTPs (phishing, credential dumping, cloud persistence).
• AI-specific: prompt injection, prompt leakage, model inversion, data poisoning, model theft, adversarial examples, automated spear phishing.

4) Rate Risk (Be Consistent!)

• Simple: 1–5 Likelihood × 1–5 Impact = 1–25 score.
• Add business context: safety, legal/regulatory, brand, confidentiality, integrity, availability.
• FAIR if you want dollars; DREAD/OWASP if you want quick triage.

function risk_score(likelihood, impact, detection, control_strength):
    inherent = likelihood * impact
    adjusted = inherent * (1 - control_strength) * (1 - detection)
    return round(adjusted)

// Example: LIK=4, IMP=5, DET=0.3, CTRL=0.4 → residual ≈ 8

5) Choose Controls (Defense-in-Depth Remix)

• Map to NIST CSF: Protect (PR), Detect (DE), Respond (RS), Recover (RC).
• Map to ISO/IEC 27001 Annex A for control families.
• Example controls for AI risks below.

6) Document It (Risk Register)

• Keep it lightweight but real. You’ll live here.

ID	Threat	Asset	Likelihood	Impact	Residual Risk	Owner	Control(s)	Status
R-01	Prompt injection exfiltrates secrets	LLM agent	4	5	8	AppSec	Output filters, secret redaction, allowlist tools	In progress
R-02	Data poisoning via supplier dataset	Model	3	5	9	ML Eng	Data provenance, signed datasets, anomaly detection	Planned
R-03	Credential theft via AI-crafted phish	Workforce	5	4	10	SecOps	Phishing-resistant MFA, awareness, mailbox rules	Implemented

7) Validate & Iterate

• Tabletop exercises, purple team, adversarial ML testing, chaos drills for incident response.
• Feed lessons back into frameworks (NIST CSF RS/RC, ISO continuous improvement).

---

AI-Driven Threats: The New Boss Level

• Prompt Injection: Tricking the model to ignore instructions, leak data, or call tools. Controls: strict tool allowlists, output validation, content filters, system prompt hardening, separation of sensitive context.
• Data Poisoning: Contaminated training sets cause biased or backdoored behavior. Controls: data lineage, signed datasets, differential data checks, adversarial data tests.
• Model Inversion & Membership Inference: Reconstructing or confirming training data. Controls: differential privacy, regularization, rate limiting, query monitoring.
• Model Theft: Copying a model via query APIs. Controls: watermarking, rate limiting, anomaly detection, IP allowlists, TOS/legal.
• Adversarial Examples: Inputs crafted to mislead models. Controls: adversarial training, input preprocessing, ensemble checks.
• Automated Social Engineering: Scalable, personalized phishing and deepfakes. Controls: phishing-resistant MFA, verification out-of-band, training with real-ish simulations.

Pro tip: Treat LLMs as powerful but gullible interns. Never give them production keys without a chaperone.

---

Mini Walkthrough: The FinBot Fable

You run FinBot, a fintech support assistant using an LLM to answer account questions.

1. Scope & Assets

• Assets: customer PII, transaction data, model prompts/responses, API keys.
• Dependencies: third-party LLM API, CRM, auth provider.

2. DFD Snapshot

• User → Web → API Gateway → Policy Engine → LLM Service ↔ Vector Store → CRM/Accounts DB → Logs.
• Trust boundary at LLM vendor and vector store.

3. Threat Hunt

• STRIDE:
  • S: Deepfake caller → voicebot? If yes, high.
  • T: Prompt injection to alter refund rules.
  • I: Model inversion reveals PII from embeddings.
  • D: Flood of long prompts → DoS on LLM credits.
  • E: LLM triggers admin-only refund tool.
• ATT&CK lens: initial access via phishing; credential abuse; cloud persistence; exfil via API.

4. Risk Ratings (abridged)

• Prompt injection causing tool misuse: Likelihood 4 × Impact 5 → Inherent 20. With allowlist, output validation, RBAC, and rate limits → Residual ~8.
• Data poisoning via supplier feed: L3 × I5 → 15. With dataset signing, schema checks → 9.
• Model inversion on vector store: L3 × I4 → 12. With encryption-at-rest, KMS, access monitoring → 6.

5. Controls (mapped)

• NIST CSF PR: secret management, least privilege, content filters, secure SDLC with ML abuse tests.
• NIST CSF DE: anomaly detection on prompts/responses, query fingerprinting, data drift alerts.
• ISO 27001 Annex A: A.8 (asset management), A.9 (access control), A.12 (ops security), A.14 (system acquisition, secure dev), A.15 (supplier relationships).

6. Validate

• Red team runs prompt-injection playbook; purple team maps findings to ATT&CK tactics. Capture lessons, update risk register, adjust controls.

If you can’t simulate it, you can’t claim you’re ready for it.

---

Fast Comparisons You’ll Thank Later

Method	Good For	Weakness	Use With
STRIDE	Design-time review	Doesn’t quantify money	OWASP/ATT&CK
PASTA	Business-aligned risk	Heavier process	ISO 27001 risk treatments
FAIR	Dollar estimates	Data hungry	Board reporting
Attack Trees	Visualizing attacker paths	Maintenance	MITRE ATT&CK

---

Common Facepalms (So You Don’t Repeat Them)

• Treating AI like magic instead of software with inputs/outputs and threat surfaces.
• Skipping supply chain checks (datasets, models, prompts as code, packages).
• Logging zero context around model actions; then being shocked you can’t investigate.
• Assuming MFA solves phishing when tokens are phishable; use phishing-resistant methods.
• Risk matrices with everything red; congratulations, you’ve prioritized nothing.

---

TL;DR (but make it useful)

• Risk = structured prioritization: Likelihood × Impact, adjusted by real control strength and detection.
• Threat modeling turns frameworks into street smarts; mix STRIDE, ATT&CK, and AI-specific checks.
• Defense-in-depth still rules: preventive, detective, and responsive layers mapped to NIST CSF and ISO 27001.
• AI adds new classes of threats (prompt injection, poisoning, inversion) — treat models as sensitive assets with governance, monitoring, and guardrails.
• Keep a living risk register, validate with exercises, and iterate. Boring? Maybe. Effective? Absolutely.

Final thought: Good security isn’t about fear — it’s about focus. Threat modeling is how you choose what to care about before the internet chooses for you.