AI For Everyone > AI Fundamentals for Everyone

Framing Problems for AI: From Vibes to Variables

You can’t fix a problem you never actually framed. That’s not AI, that’s astrology with Wi‑Fi.

We’ve seen where AI sneaks into products (Position 10: yep, it’s ranking your feed and autocorrecting your chaotic 3 a.m. texts), and we zipped through the AI lifecycle (Position 9: data → train → evaluate → deploy → monitor → oops → iterate). Today is the glue between those: how to turn a squishy idea like “make it smarter” into a precise, testable AI problem. This is Step Zero. Frame it wrong, and everything else is a very expensive detour.

---

Why Framing Matters (a.k.a., Don’t Bring a Poem to a Math Fight)

• AI doesn’t solve “vibes.” It solves well-specified input → output mappings under constraints.
• The same user need can be framed multiple ways. Pick the wrong frame, and you’ll optimize the wrong thing beautifully.
• Framing sits between product reality and the AI lifecycle. It decides your data needs, model type, evaluation metrics, and whether you should even use AI.

Good framing sounds boring: “Given X, predict/generate Y to optimize Z under constraints C, measured by M.” Boring is beautiful. Boring ships.

---

The Problem Framing Canvas

Use this as your cheat sheet. If you can’t fill these, you don’t have an AI problem yet — you have a wish.

1. Outcome and User Story

• Who benefits, in what scenario, and why? What product metric moves? Revenue, retention, time saved, risk reduced?
• Example: “As a support agent, I want suggested replies so I can resolve tickets faster (reduce average handle time by 20%).”

2. Decision in the Loop

• What decision will this model inform or automate? Who is the final decision-maker: human, model, or hybrid?
• Clarify the tolerance for error. If a mistake costs $1M, we’re in ‘assistive’ territory, not ‘fully automated.’

3. Inputs and Context

• What data do we have at decision time? Text, images, logs, tabular history, geolocation?
• Beware time leakage: do not use future data to predict the present. Regression models are not time travelers.

4. Output Type (Choose Your Fighter)

Frame	Input → Output	Common Metrics	Example	Classic Failure Mode
Classification	X → category	Accuracy, F1, ROC-AUC	Spam/not spam	Class imbalance hides pain
Regression	X → number	MAE, RMSE, MAPE	ETA prediction	Outliers wreck averages
Ranking	X → ordered list	NDCG, MAP@K, CTR	Search results	Position bias, clickbait wins
Generation	X → text/image/code	BLEU/ROUGE, human ratings, toxicity	Support replies	Hallucinations, tone mismatches
Extraction	X → structured fields	F1, exact match	Pull invoice totals	Boundary errors
Clustering	X → groups	Silhouette score, utility studies	Customer segments	Meaningless clusters
Anomaly detection	Time series/logs → flag	Precision@k, recall	Fraud spike alerts	Alert fatigue
Forecasting	History → future series	sMAPE, WAPE	Demand planning	Seasonality drift
Recommendation	User+items → top-N	Recall@K, NDCG, diversity	Movie suggestions	Filter bubble
RL/Policy	State → action	Reward	Bidding strategy	Reward hacking

5. Success Metrics (Offline and Online)

• Offline: how the model behaves on holdout data (F1, RMSE, BLEU). Choose metrics that reflect costs of mistakes.
• Online: impact in the wild (conversion, time saved, revenue, safety flags). If your offline metric moves but users don’t, you optimized a lab trick.

6. Constraints (Reality’s Gentle Hug)

• Latency budget? Mobile vs server? Cost per inference?
• Safety, privacy, fairness, explainability. Hard constraints trump accuracy.
• Example: “Replies must generate in under 1s, cost <$0.002/call, and pass safety filters.”

7. Data Picture

• What labeled data exists? How much? How skewed? Any PII? Licensing limits?
• Labeling plan if you don’t have it. Can you bootstrap with weak labels or heuristics?
• Drift risk: will the world change under your feet?

8. Baselines and Alternatives

• What’s the non-ML baseline? If rules get you 90% there, maybe do rules.
• Establish dumb baselines (majority class, pasting top template). Beat them first, then celebrate with snacks.

9. Risks and Harms

• Misclassification costs, unfair outcomes, feedback loops (“bad recommendations beget bad clicks beget worse recs”), adversarial abuse, hallucinations.
• Mitigation plan: human-in-the-loop, thresholds, monitoring, red-teaming, guardrails.

10. Lifecycle Hooks (Tie-back to Position 9)

• Define eval sets, deployment gate criteria, monitoring metrics, rollback triggers.
• If you can’t say how you’ll know it’s working post-launch, you’re not done framing.

---

Mini Case Studies (Because Examples > Platitudes)

1. Spam Detection (Classification)

• Outcome: Protect inboxes; reduce spam complaints by 50%.
• Input: Email text, sender history.
• Output: spam/not spam + confidence.
• Metrics: Precision prioritized (false positives hurt trust), F1 overall.
• Constraints: Sub-100ms latency; comply with privacy rules.
• Baseline: Block known bad domains; keyword filters.
• Risk: Adversarial adaptation; constantly retrain on new spam styles.

2. ETA Prediction (Regression/Forecasting)

• Outcome: Reduce user cancellations by improving ETA accuracy.
• Input: GPS traces, traffic, road features.
• Output: Minutes until arrival.
• Metrics: Median absolute error; calibration (do 10 min ETAs actually average ~10?).
• Constraint: Updates every 30s; max 200ms per prediction on device.
• Baseline: Historical average by route/time-of-day.
• Risk: Weather anomalies; add uncertainty bands (“10–12 min”).

3. Product Search (Ranking + Retrieval)

• Outcome: Increase add-to-cart by 5%.
• Input: Query text, item catalog, user signals.
• Output: Ordered results.
• Metrics: NDCG@10, CTR uplift, revenue per session.
• Constraints: 300ms budget; safe/appropriate results.
• Baseline: BM25 or lexical search.
• Risk: Clickbait trap; add diversity and business rules.

4. Support Chat Assistant (Generation with Retrieval)

• Outcome: Cut agent handle time by 20% with suggested replies.
• Input: Ticket text + retrieved docs (RAG).
• Output: Draft response, editable.
• Metrics: Human-rated helpfulness, edit distance from final, deflection rate.
• Constraints: Must cite sources; zero PII leakage; <1s latency.
• Baseline: Canned macros.
• Risk: Hallucinations; enforce tool-use, citations, and response policies.

---

The One-Sentence Spec (Tattoo This on Your PM Notebook)

Given [available inputs at decision time], predict/generate [the output] to optimize [business/user outcome] under [hard constraints], evaluated by [offline metrics] and validated by [online metrics].

Example:

Given ticket text and relevant knowledge-base snippets retrieved at runtime,
generate a draft reply to optimize agent handle time (-20%) and CSAT (+5pt)
under 1s latency, <$0.002/inference, and strict safety/citation constraints,
evaluated offline by human-rated helpfulness ≥4/5 and online by AHT and CSAT uplift.

Fill this out before you train anything. Your future self will send you a fruit basket.

---

The “Should We Even Use AI?” Flow

• Do we have a repeatable decision that happens at scale? If not, rules or UX might beat AI.
• Is there data at the moment of decision? If not, first solve data collection.
• Are mistakes cheap and correctable? If mistakes are catastrophic, keep a human in the loop.
• Can a simple heuristic get us most of the impact? Start there; layer AI later.

Shipping a simple solution now beats shipping a fancy model never. Your roadmap is not a Kaggle competition.

---

Common Framing Faceplants (Spot Them Like Red Flags on a Dating App)

• Vibe-based objectives: “Make it smarter” (smarter at what?).
• Metric mismatch: Optimizing accuracy when costs are asymmetric. Use precision/recall, cost-weighted loss, or calibrated thresholds.
• Data leakage: Accidentally using post-event info to predict pre-event outcomes. Split by time; be paranoid.
• One-metric tyranny: Great F1, terrible user experience. Track latency, stability, and cost too.
• Baseline amnesia: No baseline, no progress. Always compare.

---

Quick Framing Worksheet (Use in Kickoff Meetings)

• Problem statement: [your one-sentence spec]
• User story + outcome metric:
• Decision type: assist, recommend, or automate?
• Inputs at decision time: data schema, freshness, privacy flags
• Output format: classification/regression/ranking/generation/…
• Offline metrics (with target thresholds):
• Online metrics (primary + guardrails):
• Constraints: latency, cost, safety, fairness, explainability
• Baselines: rule-based, random, majority, heuristic
• Risks + mitigations: human-in-loop, thresholds, monitoring plan
• Lifecycle hooks: eval set, deployment gates, rollback triggers, retraining cadence

Copy that into your doc template. Your team will thank you. Eventually.

---

Tying Back to the Lifecycle and Product Context

• From Position 10 (Where AI shows up): Map user moments to decision types. Each “AI moment” is a candidate for a framed problem.
• From Position 9 (Lifecycle): Your framing defines the datasets you need, the metrics gating deployment, and the monitors you’ll watch in production.
• Framing isn’t a one-off. As data drifts and goals evolve, you’ll reframe, re-metric, and re-baseline. That’s not failure; that’s maintenance.

---

TL;DR and Marching Orders

• Say the quiet part loud: “Given X, predict Y to optimize Z under C, measured by M.”
• Choose an output type that fits the decision. The table is your menu, not a buffet.
• Pick metrics that match costs and user impact. Offline is for sanity; online is for truth.
• Respect constraints: latency, cost, safety, fairness. Constraints are product features.
• Always establish baselines and risks. If you can’t beat simple, don’t ship fancy.

Powerful insight to pocket: Framing is product strategy written in math. Get the frame right, and the rest of the lifecycle becomes execution instead of excavation.