Learn · Module 14 · ML system design · Lesson 1 of 5
A framework for system design
Designing an ML system rewards a calm, structured approach far more than clever ideas. Here is one that works for any ML problem.
- Clarify, frame, data, model, evaluation, serving, risks: in that order.
- Spend the first minutes asking questions that change the design.
- Start with a simple baseline, use numbers, and name trade-offs and risks.
In plain words
Most ML systems start from an open brief, like 'build a feed' or 'build a support assistant'. What matters is how you think it through: do you clarify, make sensible choices, explain trade-offs and know where things break? A fixed set of steps keeps you on track.
The seven steps
| step | what to cover |
|---|---|
| 1. Clarify | Users, scale, what success means, constraints (speed, cost, privacy) |
| 2. Frame the ML problem | What's predicted or generated, from which inputs; is ML even needed? |
| 3. Data | Where it comes from, labels, how fresh, privacy |
| 4. Model | Start simple; a baseline; then the stronger approach and why |
| 5. Evaluation | Offline metrics, online metrics, how you'd test a change |
| 6. Serving | Latency, cost, caching, fallbacks, scale |
| 7. Risks and next steps | Failure modes, monitoring, what you'd improve |
Worked example: the first questions
Prompt: 'Design a system that answers employees' HR questions.' Before drawing anything, ask questions that change the design.
- 1Who and how many?
5,000 employees, maybe 2,000 questions a day. Small scale, so cost isn't the main worry.
- 2Where are the answers?
A few hundred HR policy documents, updated monthly. That points to retrieval, not fine-tuning.
- 3What does good mean?
Correct, with a link to the policy, and passes personal cases to HR staff. That defines your evaluation.
- 4What must never happen?
Revealing someone else's salary or leave details. That shapes permissions and guardrails.
- 5Say your plan out loud
'So: a RAG assistant over HR policies, with citations, a hand-off to HR for personal cases, and strict access control. Let me go through data first.'
Your manager says 'Build a recommendation system for our app.' What should you do first?
What to remember
Always state a simple baseline before a complex model. Give numbers where you can (requests a day, latency budget, cost per request). Name trade-offs explicitly, and finish by naming what could go wrong and how you'd know. The next four lessons apply this framework to common questions.
Common mistakes
- Jumping straight to model architecture without clarifying users, scale or success.
- Never stating numbers, so the design can't be judged on cost or latency.
- Running out of time before discussing evaluation, monitoring and risks.
Answer in your own words, out loud or on paper, before you open the model answer.
Walk me through how you'd approach an ML system design question.
I follow a fixed structure so nothing important is skipped. First, clarify: who the users are, the scale, what success means and the constraints on latency, cost and privacy. Second, frame it as an ML problem: what we predict or generate from which inputs, and whether ML is needed at all. Third, data: sources, labels, freshness. Fourth, the model: a simple baseline first, then a stronger approach and why. Fifth, evaluation: offline metrics and how I'd test online. Sixth, serving: latency, cost, caching, fallbacks. Finally, risks and monitoring. Throughout, I state assumptions and rough numbers out loud and decide which parts deserve the most depth.
- Design an assistant that answers employees' HR questions.
- How do you decide whether a problem needs ML at all?
Go further
- ArticleWhat We've Learned From A Year of Building with LLMsYan, Bischof, Frye, Husain, Liu & Shankar
- ArticlePatterns for Building LLM-based Systems & ProductsEugene Yan
- BookDesigning Machine Learning SystemsChip Huyen (O'Reilly)