Context windows explained: what fits and what gets lost

Paste a 200-page contract into an AI assistant and ask about one exception clause. Whether the answer is right depends on whether the contract fits in the model's context window, what else already occupies that space, and whether the model actually uses the clause once it is there.

Vendors now advertise windows of a million tokens or more. That number tells you what a request can hold. It does not tell you what the model will reliably use.

What a context window is

A large language model processes text as tokens: small units that can be a word, part of a word, punctuation or a single character. OpenAI's rough guide for English is that one token is about four characters, or three-quarters of a word, and it stresses that these are estimates that vary by model and language.[1]

The context window is the limit on how many tokens a model can work with in one request. Anthropic describes it as everything the model "can reference when generating a response, including the response itself," and calls it a kind of working memory, separate from the data the model was trained on.[2]

The window is temporary, since nothing in it changes the model's learned weights. It is also shared: every token spent on one thing is unavailable for another.

What counts toward the window

The window is not just "my prompt." Several things compete for the same space.

• The system prompt. Instructions set by the app or developer, often invisible to you.
• Conversation history. In a typical API conversation, every earlier message and reply is sent again with each new turn, so the input grows as the chat goes on.[2]
• Files, images and tool results. Uploaded documents, search results, tool definitions and the outputs of tools all count.[2]
• The output. The answer the model writes counts too, and so do the hidden reasoning tokens that reasoning models generate before their visible answer.[1][2]

Anthropic's documentation puts it plainly: "Everything in the request counts toward the context window."[2] Google says its newest Gemini window is "shared across input, thinking, and output tokens."[3] OpenAI notes that a short visible answer can therefore use more tokens than its text suggests.[1] Our reasoning models explainer covers why that extra work happens.

Chat apps add another layer. Anthropic notes that interfaces such as claude.ai can manage the window on a rolling "first in, first out" basis, so the oldest parts of a long conversation can drop out.[2] That is one reason an assistant may seem to forget an early instruction.

AI Chronicle

How large are windows now?

Window sizes change with each release, so treat these as a snapshot from official documentation:

These are API figures. A consumer app built on the same model may apply its own limits or trimming.

Fitting is not the same as using

The best-known evidence is Lost in the Middle (2023) by Nelson Liu and colleagues. They tested models on multi-document question answering and key-value retrieval while moving the relevant information to different positions in the input.

"…performance is often highest when relevant information occurs at the beginning or end of the input context."

— Liu and colleagues, Lost in the Middle: How Language Models Use Long Contexts (2023).[5]

The same abstract reports that performance "significantly degrades" when models must use information in the middle of long contexts, "even for explicitly long-context models."[5] These were 2023 models on specific tasks, not an error rate for today's systems, but the finding shows why position deserves testing.

Later work points the same way. The RULER benchmark, from NVIDIA researchers, found that models scoring nearly perfectly on simple "needle in a haystack" retrieval showed large performance drops as context length grew on harder tasks; only half of the 17 models tested kept satisfactory performance at 32K tokens.[6]

Vendors acknowledge the limit too. Anthropic's documentation says "more context isn't automatically better" and describes accuracy and recall degrading as token count grows.[2] Google's long-context guide notes that when you look for several pieces of information at once, rather than one, the model "does not perform with the same accuracy."[3]

Practical tactics

1. Send what the task needs. Google's guidance is direct: if tokens are not needed, avoid passing them.[3] Cut boilerplate, duplicates and irrelevant appendices.
2. Put the question after the material. Both Google and Anthropic recommend placing long documents first and the query at the end.[3][7]
3. Label each document. Anthropic suggests wrapping documents in tags with their source, so the model can tell them apart.[7]
4. Ask for quotes first. Asking the model to extract relevant passages before answering helps it focus, and gives you something to check.[7]
5. Start fresh for a new task. A long chat carries old turns, files and tool results into every request. A clean session with a short summary is often leaner.
6. Reserve room for the answer. If the input nearly fills the window, the output can be cut short.[2]

When to use RAG instead

Retrieval-augmented generation keeps source material outside the model and fetches only the passages relevant to each question.[8] Long context and retrieval solve different problems.

Use a long window when the material is bounded, such as one contract or one codebase, and the task needs all of it at once. Consider retrieval when the material is larger than any window, changes often, needs per-user access controls, or would otherwise be resent in full with every question. Many systems combine both; agents, covered in our AI agents explainer, often manage context step by step.

A worked example (hypothetical)

This scenario is illustrative. No product was tested.

An operations manager uploads a 180-page supplier agreement and asks, "Can we terminate early without penalty?" The termination clause is on page 92; an exception sits in a schedule on page 171.

A risky approach: paste the agreement mid-way through a long chat about other topics, with the question at the top. A better approach: start a new session, place the agreement first, then ask the question, and require the answer to quote every clause it relies on, with page numbers. The reviewer then checks those quotes against the document and looks specifically for schedules and exceptions the answer did not cite.

If the team needs answers across 400 supplier agreements, that is a retrieval problem, not a bigger-window problem.

Questions readers commonly ask

Is the context window the same as the model's memory?

No. It is working space for one request. Some apps add separate memory features that save notes and re-insert them into later requests, but those still have to fit in the window.

Why does a chatbot forget what I said earlier?

The conversation may have exceeded the window, or the app may be trimming older turns. Even within the window, details in a long middle section can be used less reliably.[2][5]

Does a bigger window mean better answers?

Not by itself. Research and vendor guidance both say recall can degrade as context grows.[2][6] Test with your own documents.

Do tokens mean the same thing for every model?

No. Different tokenizers split the same text differently, so the same document can use a different number of tokens on different models.[1]

The next step

Long context is one way to give a model the right information; retrieval is the other. Read What is RAG? Retrieval-augmented generation explained to see how retrieval works and where it fails. When an answer cites a document, our guide to verifying AI-generated research shows how to check it, and our hallucinations explainer covers why confident answers can still be unsupported.

Related: Claude vs. ChatGPT for long-document work · What AI benchmarks actually measure

About these sources

All pages accessed 15 October 2026.

Related reading

• Claude vs. ChatGPT for long-document work
• What AI benchmarks actually measure