epochs

Learn · Module 14 · ML system design · Lesson 1 of 5

A framework for system design

Designing an ML system rewards a calm, structured approach far more than clever ideas. Here is one that works for any ML problem.

3 minute read

In short
  • Clarify, frame, data, model, evaluation, serving, risks: in that order.
  • Spend the first minutes asking questions that change the design.
  • Start with a simple baseline, use numbers, and name trade-offs and risks.

In plain words

Most ML systems start from an open brief, like 'build a feed' or 'build a support assistant'. What matters is how you think it through: do you clarify, make sensible choices, explain trade-offs and know where things break? A fixed set of steps keeps you on track.

The seven steps

stepwhat to cover
1. ClarifyUsers, scale, what success means, constraints (speed, cost, privacy)
2. Frame the ML problemWhat's predicted or generated, from which inputs; is ML even needed?
3. DataWhere it comes from, labels, how fresh, privacy
4. ModelStart simple; a baseline; then the stronger approach and why
5. EvaluationOffline metrics, online metrics, how you'd test a change
6. ServingLatency, cost, caching, fallbacks, scale
7. Risks and next stepsFailure modes, monitoring, what you'd improve

Worked example: the first questions

Prompt: 'Design a system that answers employees' HR questions.' Before drawing anything, ask questions that change the design.

  1. 1
    Who and how many?

    5,000 employees, maybe 2,000 questions a day. Small scale, so cost isn't the main worry.

  2. 2
    Where are the answers?

    A few hundred HR policy documents, updated monthly. That points to retrieval, not fine-tuning.

  3. 3
    What does good mean?

    Correct, with a link to the policy, and passes personal cases to HR staff. That defines your evaluation.

  4. 4
    What must never happen?

    Revealing someone else's salary or leave details. That shapes permissions and guardrails.

  5. 5
    Say your plan out loud

    'So: a RAG assistant over HR policies, with citations, a hand-off to HR for personal cases, and strict access control. Let me go through data first.'

Your turn

Your manager says 'Build a recommendation system for our app.' What should you do first?

What to remember

Always state a simple baseline before a complex model. Give numbers where you can (requests a day, latency budget, cost per request). Name trade-offs explicitly, and finish by naming what could go wrong and how you'd know. The next four lessons apply this framework to common questions.

Common mistakes

  • Jumping straight to model architecture without clarifying users, scale or success.
  • Never stating numbers, so the design can't be judged on cost or latency.
  • Running out of time before discussing evaluation, monitoring and risks.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Walk me through how you'd approach an ML system design question.

I follow a fixed structure so nothing important is skipped. First, clarify: who the users are, the scale, what success means and the constraints on latency, cost and privacy. Second, frame it as an ML problem: what we predict or generate from which inputs, and whether ML is needed at all. Third, data: sources, labels, freshness. Fourth, the model: a simple baseline first, then a stronger approach and why. Fifth, evaluation: offline metrics and how I'd test online. Sixth, serving: latency, cost, caching, fallbacks. Finally, risks and monitoring. Throughout, I state assumptions and rough numbers out loud and decide which parts deserve the most depth.

More to think about
  • Design an assistant that answers employees' HR questions.
  • How do you decide whether a problem needs ML at all?

Go further

Was this lesson clear?