Learn · Module 14 · ML system design · Lesson 5 of 5
Designing an LLM product
Put everything together: a customer support assistant, designed end to end.
3 minute read
- Start from clarifying questions and a simple workflow, not a free-running agent.
- Give concrete numbers for cost and scale; plan evaluation before building.
- Name risks with specific mitigations: citations, permissions, masking, fallbacks.
The brief
'Design an AI assistant that answers customer support chats for an online store. 20,000 chats a day.'
Worked example: the design, step by step
- 1Clarify
Which questions? Order status, returns, product questions. Can it take actions (refunds) or only answer? What should go to a human? Success: resolved without a human, correctly, with good satisfaction.
- 2Frame
A workflow, not a free agent: classify the chat, then answer with retrieval and read-only tools, and hand off to a human for refunds over a limit, complaints or low confidence.
- 3Data
Help-centre articles and policies for retrieval; order system through tools; past chats to build an evaluation set and find common questions.
- 4Model
A small model for classification and routing; a stronger model for answers; structured outputs for tool calls. Baseline: retrieval plus a good prompt, no fine-tuning.
- 5Evaluation
300 real past chats with known good outcomes. Measure correctness, faithfulness to policy, hand-off accuracy. Online: resolution rate, satisfaction, escalations.
- 6Serving and cost
Each turn sends about 2,500 input tokens and 300 output tokens. At $3 / $15 per million, that's about $0.012 a turn. 20,000 chats × 4 turns ≈ $960 a day, so prompt caching and routing simple questions to a small model matter.
$0.012 × 80,000 turns ≈ $960 / day
- 7Risks
Wrong policy answers (cite sources, evaluate), prompt injection (read-only tools, limits in code), personal data in logs (masking), outages (fallback model, then 'connecting you to a person').
The assistant resolves 70% of chats. Leadership wants 90%. What's the best next step?
What to remember
A strong design is concrete: real numbers, a simple baseline, a clear evaluation plan, and honest risks. It uses everything in Part 2 of this course: prompting, retrieval, tools, workflows, evaluation, cost, fallbacks and guardrails. If you can reason through this design comfortably, you can design most LLM products.
Common mistakes
- Building a fully autonomous agent with refund powers on day one.
- Launching without an evaluation set built from real past conversations.
- Not planning the hand-off to human agents.
Answer in your own words, out loud or on paper, before you open the model answer.
Design an AI customer support assistant for an online store.
I'd clarify scope first: order status, returns and product questions, whether it can take actions, and when to involve a person. Then a workflow rather than a free agent: classify the message, answer with retrieval over help articles and policies, and use read-only tools for order lookups, handing off to a human for refunds above a limit, complaints or low confidence. I'd build an evaluation set from a few hundred real past chats and measure correctness, policy faithfulness and hand-off accuracy before launch. For cost, I'd estimate tokens per turn, cache the shared prompt and route simple questions to a small model. Risks include wrong policy answers, prompt injection and personal data in logs.
- How would you measure whether the assistant is good enough to launch?
- How would you keep the cost under control as traffic grows?
Go further
- ArticleWhat We Learned from a Year of Building with LLMsYan, Bischof, Frye, Husain, Liu & Shankar
- ArticlePatterns for Building LLM-based Systems & ProductsEugene Yan
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.