7.3 Baselines and Reference Models — The Unsexy but Critical Benchmarking Ritual

"If you don't know where you're starting from, every finish line looks like progress." — Your future angry evaluator

---

Welcome back. We already covered how to evaluate (7.1 Evaluation Protocols) and how to slice-and-dice your validation data (7.2 Validation Set Design and Splits). Now: the thing that makes your metrics mean anything — baselines and reference models. Think of them as the control group in your experiment. No control group? Congratulations, you just produced a very expensive, very pretty hallucination.

This section tells you which baselines matter, how to pick them, how to run them reproducibly (especially when you're distributed with DeepSpeed / FSDP / ZeRO), and how to monitor comparative performance over time so your model doesn’t sneakily regress while you sleep.

---

Why baselines are non-negotiable

• Context: Raw numbers are meaningless without a frame of reference. A 2% accuracy bump can be the holy grail — or worse-than-random — depending on the baseline. (Remember 7.1: your evaluation protocol only helps if your baseline is sane.)
• Cost-efficiency: You tuned hyperparameters and burned GPU-hours. Baselines let you decide if that burn was worth it.
• Reproducibility & fairness: Comparing to a documented baseline means others can replicate and compare.

---

The Baseline Hierarchy (aka the “Who to Invite to the Party”)

1. Null / trivial baseline — e.g., majority class, random, or simple heuristic. Cheap, fast sanity check.
2. Pretrained checkpoint (zero-shot) — run the original pretraining checkpoint on your validation tasks. Measures what fine-tuning actually bought you.
3. Existing production model — the current deployed model or service you aim to beat.
4. Parameter-efficient variants — LoRA, adapters, prompt tuning. Compare compute & perf trade-offs.
5. Full fine-tune baseline — the dense, full-parameter fine-tuned model (if feasible).
6. External SOTA / Leaderboard — public benchmarks or published baselines, if applicable.
7. Human baseline — for subjective tasks or high-stakes cases.

Tip: Always include at least one trivial baseline and one realistic production baseline.

---

Quick comparison table

Baseline type	Why include it	Cost	What it tells you
Trivial (random/majority)	Sanity check	Very low	If you can't beat random, stop here
Pretrained (zero-shot)	True value of fine-tuning	Low	How much fine-tuning shifts capability
Production model	Real-world comparison	Medium	Are we actually better in practice
PEFT (LoRA/Adapters)	Efficiency trade-off	Low–medium	Perf per compute/latency win
Full fine-tune	Upper bound (maybe)	High	Performance ceiling
SOTA / external	Benchmarking	Variable	Where you sit in the community

---

Practical checklist: choosing baselines for cost-effective fine-tuning

• Include a trivial baseline (majority/random heuristic).
• Evaluate the raw pretrained checkpoint on your validation split from 7.2.
• Include your current production model (if there is one).
• Try at least one PEFT method (LoRA or adapters) — they might meet requirements at a fraction of the cost.
• If budget allows, include a dense full-finetune or an earlier tuned checkpoint as a high-water mark.
• If you publish, compare to public leaderboards where available.

---

Reproducible baseline evaluation in distributed settings

Remember your scaling tools (DeepSpeed, FSDP, ZeRO). They reduce memory pain, but they introduce nondeterminism if you’re not careful.

• Pin RNG seeds globally (torch, numpy, random). Test across multiple seeds and report mean ± std.
• Keep architecture/IO/config differences consistent between baseline and candidate models (same tokenizer, same batch sizes for eval, same precision where possible).
• Use the same hardware profile or normalize for latency/cost when comparing runtime or cost-efficiency.
• If you use activation/checkpoint sharding, document it — it can change numerical results slightly.

Pseudocode for a baseline-comparison evaluation loop:

for model in [trivial, pretrained, peft, candidate, full_finetune]:
    for seed in seeds:
        set_seed(seed)
        load_model(model, distributed=True)
        perf = evaluate_on(validation_split)
        log(model.name, seed, perf, cost, latency)
# compute mean, CI using bootstrap or paired t-test

---

Statistical rigour: small differences are sneaky

• Use paired tests where possible (same examples across models). Paired bootstrap or paired t-tests are common.
• Report confidence intervals (95%) and effect sizes, not just p-values.
• When comparing multiple models, correct for multiple comparisons or construct a pre-registered comparison matrix.

Example: paired bootstrap for metric delta — resample validation examples with replacement, compute metric difference candidate-baseline on each resample, build CI.

---

Slicing, fairness, and the per-application baseline

A global average can hide tragedies. Compare baselines across slices from 7.2:

• By demographic or domain slice
• By difficulty (easy/ambiguous)
• By latency or resource constraints

If a PEFT baseline has the same global score as a dense fine-tune but completely fails on a demographic slice, you need to know that.

---

Monitoring baselines in production — the “keeps-the-C-suite-happy” bit

• Keep a rolling baseline snapshot: every week/month, evaluate the deployed model against the reference baseline on a buffered holdout.
• Canary & A/B: roll new model to a small fraction of traffic and compare live metrics to the existing baseline.
• Drift detection: monitor per-slice performance deltas vs. baseline. Trigger retraining if degradation crosses a threshold.
• Latency & cost baseline: track inference latency, memory, and cost-per-query alongside quality metrics.

Quote to remember:

"Performance is not just accuracy — it's accuracy minus surprise." — Production ML Engineer, 3am

---

Common traps and how to avoid them

• Trap: Using a single seed. Fix: run multiple seeds, report mean & CI.
• Trap: Comparing models with different tokenizers or preprocessing. Fix: standardize eval pipeline.
• Trap: Only measuring aggregate metrics. Fix: slice-level checks & fairness tests.
• Trap: Confusing small numeric gains with practical wins. Fix: pair metrics with cost/latency and user impact.

---

Closing — Concrete next steps

1. Choose your baseline set (use the checklist).
2. Run deterministic, multi-seed evaluations using the same distributed config you’ll use in production (DeepSpeed/FSDP notes documented).
3. Report mean, CI, and cost per query; visualize Pareto frontiers (accuracy vs cost/latency).
4. Build automated monitoring to compare live traffic to baseline slices and trigger retrains.

Key takeaways:

• Baselines are your truth serum. No baseline, no trust.
• Always include trivial and pretrained checkpoints; consider PEFT methods as often the most cost-effective option.
• Be rigorous: multi-seed, paired stats, slice-level checks, and production monitoring.

Ready to be the kind of researcher/product person who can honestly say, "We improved the model" — backed by numbers, cost analysis, and no surprises? Good. Bring snacks. We’re re-running evaluations.

---

Version: "Baseline But Make It Practical + Sass"