The context window is the model's whole world for one request. Deciding what to put in it, and where, is most of the engineering.
4 minute read
In short
The context window must hold instructions, tools, history, documents and the answer.
Reserve room for the answer, then budget the rest by what matters.
Put stable parts first, key instructions early, and the question last.
In plain words
Everything the model can use must be in the context window: instructions, tools, chat history, documents, and room for the answer. The window has a fixed size, and every token costs money and time. So the job is choosing the right tokens, not the most tokens. This is often called context engineering.
Worked example: budgeting an 8,000-token window
A support assistant runs on a model with an 8,000-token window. Here's one sensible split.
Retrieved documents3000 tokens
Chat history2500 tokens
Room for the answer1000 tokens
Tool descriptions900 tokens
System prompt600 tokens
1
Reserve the answer first
If the input fills the window, the reply gets cut off. Keep space for the longest answer you expect.
8,000 − 1,000 = 7,000 for input
2
Fixed parts are fixed
System prompt and tools are the same every request.
7,000 − 600 − 900 = 5,500
3
Split what's left
Documents and history compete for the remaining space. When the chat grows long, summarise older turns instead of dropping them blindly.
documents 3,000 + history 2,500 = 5,500
Order matters too
Models use information at the start and end of a long context more reliably than information buried in the middle. Put the instructions that matter most at the start, put the question last, and put the most relevant document closest to the question.
Also put the parts that never change at the very start. That lets the provider's prompt cache reuse them across requests, as the KV cache lesson explained.
Your turn
A long chat starts giving worse answers, and old instructions seem to be ignored. What's the best first fix?
What to remember
For each piece of context, ask whether the answer would be worse without it. If not, it's costing money and diluting attention. Measure: try the same test set with less context and see if quality drops.
Common mistakes
Sending the entire conversation every time until it hits the limit, which wastes money and slows responses.
Summarising so aggressively that key facts, like the user's order number, are lost.
Putting the most important information in the middle of a very long context, where models use it least.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
How do you manage a long conversation that exceeds the context window?
I'd keep the context focused rather than sending everything. Usually that means a sliding window of the most recent turns, plus a running summary of older parts of the conversation that's updated every few turns. Important facts, like the user's name, goals or IDs, go into a small structured memory that's always included, so they aren't lost in summarisation. If the user refers back to something specific, I can retrieve the relevant earlier messages by search instead of keeping them all. I'd place the key instructions and the current question at the start and end of the prompt, since models use the middle least, and I'd test that answers stay correct in long conversations.
More to think about
Does a longer context window remove the need for retrieval?
Where in the prompt should the most important information go?