Learn · Module 08 · Prompting and context · Lesson 2 of 6
Structured outputs
People read prose. Programs read JSON. To put a model inside a product, its output must be something code can rely on.
- Structured output makes a model's reply something code can parse and trust.
- Constrained decoding removes tokens the schema doesn't allow, so the shape is always valid.
- A valid shape isn't a correct answer: still validate, and give the model a way to say 'unknown'.
In plain words
When a person reads the answer, 'The priority looks high' is fine. When code reads it, you need exactly "priority": "high", every time. Getting a model to reply in a fixed, machine-readable shape is called structured output.
There are three levels of strictness. You can ask politely in the prompt. You can turn on a JSON mode that guarantees valid JSON. Or you can give a schema, a precise description of every field, and have the system force the output to match it.
Worked example: an email into fields
Hi, our whole team has been locked out since this morning and we have a client demo at 3pm. Please help! - Meera, Acme Corp
{
"company": text,
"problem": text, one sentence,
"priority": one of "low", "medium", "high"
}{
"company": "Acme Corp",
"problem": "The whole team is locked out of their accounts.",
"priority": "high"
}If you only ask politely, three things go wrong sooner or later: the model adds a sentence before the JSON, leaves out a field, or invents a value such as "urgent" that your code doesn't expect.
Worked example: how the system forces a valid answer
Strict structured output works at the sampling step from the Transformers module. When the model reaches the value of priority, it gives these probabilities for the next token.
| token | probability | allowed by schema? |
|---|---|---|
| high | 0.50 | yes |
| urgent | 0.30 | no |
| medium | 0.15 | yes |
| low | 0.05 | yes |
- 1Remove tokens the schema doesn't allow
'urgent' isn't one of the three allowed values, so its probability is set to zero.
high 0.50 medium 0.15 low 0.05 (total 0.70)
- 2Rescale what's left to add up to 1
Divide each by 0.70.
high 0.714 medium 0.214 low 0.071
- 3Sample as usual
Whatever is picked, it's a valid value. The model cannot output 'urgent'.
This is called constrained decoding. It guarantees the shape of the answer. It does not guarantee the answer is right: the model can still pick 'low' for an email that is clearly urgent.
With strict structured output turned on, which problem can still happen?
What to remember in practice
Always validate the output in your own code, even with strict mode, and decide what happens when validation fails: usually one retry, then a safe default or a human.
Give the model an honest way out. If a field might be unknown, allow null or an 'unknown' value. Otherwise the schema forces it to make something up.
Field order matters, because the model writes left to right. Put a short 'reasoning' field before the final answer field if you want it to think first. If the answer comes first, the reasoning is written afterwards to justify it.
Common mistakes
- Parsing free text with regular expressions when the provider supports schema-constrained output.
- Trusting that valid JSON means correct content; you still need to validate the values.
- Making every field required, which pushes the model to invent values when it doesn't know.
Answer in your own words, out loud or on paper, before you open the model answer.
How do you get reliable JSON out of an LLM?
The most reliable way is to use the provider's structured output or tool-calling feature with a JSON schema, so decoding is constrained and the response always parses. Then I validate it in code with a schema library like Zod or Pydantic, checking types, allowed values and ranges, because valid JSON can still contain wrong content. I keep the schema simple, give fields clear names and descriptions, and allow null or an 'unknown' value so the model isn't forced to invent things. If validation fails, I retry once with the error message, and if it still fails I fall back safely. I also track the failure rate in production so I notice when a model change breaks things.
- How does constrained decoding work?
- Your extraction returns valid JSON with wrong values. What do you do?
Go further
- DocsStructured outputsOpenAI
- PaperEfficient Guided Generation for LLMs (Outlines)Willard & Louf, 2023
- PaperLet Me Speak Freely? Format restrictions and LLM performanceTam et al., 2024