Test Case Design — The Scientific Method, but for Prompts (with Sass)

"If you can't break your prompt, you don't really understand it."

You're already practicing outline-first strategies, hypothesis testing, and verification-first prompting from the previous module. Great. Now we turn that lab notebook into a set of repeatable experiments. Welcome to Test Case Design: the art of making your prompts fail fast and learn faster.

---

Why test cases matter (and why your brain is bad at it)

Humans love success stories. Models, too. But both of us are lousy at finding the quiet, tiny failure modes that become pandemics in production. Test cases force you to: specify expectations, surface blindspots, and guard against regressions when you iterate on prompts or change model parameters.

This builds directly on:

• Chain-of-Thought Considerations: when you expect internal steps, you also need tests that verify each step (not just the final answer).
• Eliminating Irrelevant Paths: design negative tests that tempt the model down those irrelevant alleys.
• Socratic Questioning Prompts: unit-test the model's internal reasoning by asking it to justify steps.

---

The test-case taxonomy — know your weapons

Test Type	Purpose	Example Input	What it exposes
Positive (Happy path)	Confirms spec compliance	A clean, typical prompt	Baseline performance
Negative (Invalid / malformed)	Checks failure modes	Missing fields / nonsense data	Robustness to garbage
Edge / Boundary	Tests extremes	Long text, empty string, max tokens	Tokenization / truncation bugs
Adversarial	Traps the model	Ambiguous or leading wording	Hallucination, bias, prompt injection
Stateful / Regression	Ensures no regressions after changes	Previous production examples	Broken behavior after tweaks
Stepwise / Intermediate Assertions	Verifies internal reasoning steps	Ask for chain-of-thought + justification	Faulty chains, skipped steps

---

A 7-step recipe for designing test cases (follow like a cult)

1. Define the contract — what exactly should the model do? Format, tone, correctness criteria. Be surgical.
2. List success metrics — exact match? F1? BLEU? human-rated plausibility? confidence thresholds? (Use multiple.)
3. Create 3–5 positive examples — typical inputs that should pass easily.
4. Create adversarial/negative examples — exploit likely hallucinations or misinterpretations. Make them weird.
5. Add edge cases — empty strings, huge inputs, unicode, multiple languages.
6. Design intermediate-step checks — require explanations, numbered steps, or verification prompts to confirm reasoning.
7. Automate and iterate — run tests whenever you change the prompt or model hyperparameters.

Ask yourself at each step: "What did my earlier outline-first/hypothesis testing steps assume? Which assumption will break silently?" If you can’t answer, design a test for it.

---

Prompt Test Templates (copy-pasteable and glorious)

1) Summarization (abstractive)

• Contract: 2–3 sentence summary, preserves named entities, neutral tone.
• Positive case: a 400-word news paragraph.
• Edge case: text with quoted dialogue and dates.
• Negative case: input is a shopping list — should return "Input not summarizable." or a brief clarification question.

Prompt template (to test):

Task: Summarize the following text in 2-3 sentences. Preserve named entities. If the text is not an article, respond: "UNSUMMARIZABLE".

Text: "{input}"

Answer:

Test asserts: output length 2–3 sentences, contains entity names if present, or EXACT "UNSUMMARIZABLE".

2) Code generation (small function)

• Contract: Return a Python function that passes unit tests.
• Positive case: simple function spec with constraints.
• Adversarial case: intentionally ambiguous spec (e.g., "sort data") to see assumptions.

Intermediate-step check: ask model to provide test cases it thinks are necessary for the function.

3) Multi-step reasoning (math problem)

• Contract: Show chain-of-thought, then a final numeric answer.
• Test case: a word problem requiring 3 steps.
• Negative case: trick wording (double negation) that historically causes arithmetic slip.

Prompt: "Show your chain-of-thought step-by-step, then write 'Answer:' and final number." Then assert both the chain and the final result.

---

Automatable harness — pseudocode

for case in test_suite:
    response = run_prompt(prompt_template, case.input)
    if case.expect_format:
        assert matches_format(response, case.expect_format)
    if case.expect_value:
        assert metric(response, case.expect_value) >= case.threshold
    if case.expect_chain:
        assert verify_steps(response.chain, case.expected_steps)
    log_result(case.id, response, pass/fail)

Add randomness to test multiple seeds and temperature settings. That shows sensitivity.

---

Debugging a failed test — triage checklist

1. Re-run with deterministic settings (temperature=0) to see if nondeterminism is to blame.
2. Ask for chain-of-thought — does the model show the specific step that broke?
3. Try the Socratic approach: ask the model why it chose that wording or why it ignored a constraint.
4. Simplify the input until it passes; the point of bisection is to isolate the failure dimension (length? punctuation? tokenization?).
5. Patch the prompt: add guardrails (explicit fail responses, validation steps), then rerun test suite.
6. Write regression tests for the bug and add to CI.

Pro tip: Logging the full conversation, model config, and random seed is your future self's hero.

---

Quick adversarial examples to steal and adapt

• Confusable entity: "Apple bought 1,000 shares of Orange Inc." (Does it mix companies?)
• Implausible date: "Event occurred on February 30th." (Does it hallucinate plausible fixes?)
• Instruction conflict: "Summarize in one sentence. Write at least three sentences." (How does it prioritize?)
• Prompt injection style: embed a secondary instruction in quotes to see if it obeys the main instruction.

---

Closing — bring the chaos into order

Design test cases like you design experiments: state hypotheses, define success criteria, and try to falsify the claim that "this prompt works." Use positive, negative, edge, adversarial, and stepwise tests. Automate them. When a test fails, debug by asking for the model's chain-of-thought, bisecting inputs, and writing a regression test so the failure doesn't come back to haunt you.

Final thought: If your tests never fail, your tests are probably lying.

Go forth. Break your prompts responsibly.

---

Version name: Test Cases: The Scientific Method with Sass