epochs

Learn · Module 08 · Prompting and context · Lesson 2 of 6

Structured outputs

People read prose. Programs read JSON. To put a model inside a product, its output must be something code can rely on.

5 minute read

In short
  • Structured output makes a model's reply something code can parse and trust.
  • Constrained decoding removes tokens the schema doesn't allow, so the shape is always valid.
  • A valid shape isn't a correct answer: still validate, and give the model a way to say 'unknown'.

In plain words

When a person reads the answer, 'The priority looks high' is fine. When code reads it, you need exactly "priority": "high", every time. Getting a model to reply in a fixed, machine-readable shape is called structured output.

There are three levels of strictness. You can ask politely in the prompt. You can turn on a JSON mode that guarantees valid JSON. Or you can give a schema, a precise description of every field, and have the system force the output to match it.

Worked example: an email into fields

Input
Hi, our whole team has been locked out since this morning and we have a client demo at 3pm. Please help! - Meera, Acme Corp
Schema
{
  "company":  text,
  "problem":  text, one sentence,
  "priority": one of "low", "medium", "high"
}
Output
{
  "company": "Acme Corp",
  "problem": "The whole team is locked out of their accounts.",
  "priority": "high"
}

If you only ask politely, three things go wrong sooner or later: the model adds a sentence before the JSON, leaves out a field, or invents a value such as "urgent" that your code doesn't expect.

Worked example: how the system forces a valid answer

Strict structured output works at the sampling step from the Transformers module. When the model reaches the value of priority, it gives these probabilities for the next token.

tokenprobabilityallowed by schema?
high0.50yes
urgent0.30no
medium0.15yes
low0.05yes
  1. 1
    Remove tokens the schema doesn't allow

    'urgent' isn't one of the three allowed values, so its probability is set to zero.

    high 0.50   medium 0.15   low 0.05   (total 0.70)
  2. 2
    Rescale what's left to add up to 1

    Divide each by 0.70.

    high 0.714   medium 0.214   low 0.071
  3. 3
    Sample as usual

    Whatever is picked, it's a valid value. The model cannot output 'urgent'.

This is called constrained decoding. It guarantees the shape of the answer. It does not guarantee the answer is right: the model can still pick 'low' for an email that is clearly urgent.

Your turn

With strict structured output turned on, which problem can still happen?

What to remember in practice

Always validate the output in your own code, even with strict mode, and decide what happens when validation fails: usually one retry, then a safe default or a human.

Give the model an honest way out. If a field might be unknown, allow null or an 'unknown' value. Otherwise the schema forces it to make something up.

Field order matters, because the model writes left to right. Put a short 'reasoning' field before the final answer field if you want it to think first. If the answer comes first, the reasoning is written afterwards to justify it.

Common mistakes

  • Parsing free text with regular expressions when the provider supports schema-constrained output.
  • Trusting that valid JSON means correct content; you still need to validate the values.
  • Making every field required, which pushes the model to invent values when it doesn't know.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How do you get reliable JSON out of an LLM?

The most reliable way is to use the provider's structured output or tool-calling feature with a JSON schema, so decoding is constrained and the response always parses. Then I validate it in code with a schema library like Zod or Pydantic, checking types, allowed values and ranges, because valid JSON can still contain wrong content. I keep the schema simple, give fields clear names and descriptions, and allow null or an 'unknown' value so the model isn't forced to invent things. If validation fails, I retry once with the error message, and if it still fails I fall back safely. I also track the failure rate in production so I notice when a model change breaks things.

More to think about
  • How does constrained decoding work?
  • Your extraction returns valid JSON with wrong values. What do you do?

Go further

Was this lesson clear?