What is a context window? LLM limits and how to stay inside them

The short version

• A context window, or context length, is the maximum number of tokens an LLM can handle in one request. Input and output together must fit.
• The system prompt, conversation history, tool definitions, retrieved documents and the output all share that space. Long conversations use more of it and cost more input.
• Past the limit you get an error, or the output stops early. Accuracy can drop before that, as the window fills.
• Trim old turns, summarize, send less, and cache the stable part. If that is not enough, pick a model with a larger window.

Requests that suddenly start failing, answers cut off halfway and a model that forgets its first instructions usually share this one cause.

What a context window is

The context window is the range of tokens a model can see while it generates a response. Anything outside it does not exist for the model.

The size is set per model. Anthropic's documentation lists 1 million tokens for many models, including Claude Opus 5.5 and Claude Sonnet 5.5, and 200,000 for the others (as of October 2026). DeepSeek V4.1 Flash on FastMetal also takes 1 million.

What uses the window

Everything in the request, plus the output still to come, shares the same space:

Why the window shrinks as you talk

LLM APIs are generally stateless. The server keeps nothing between calls, so the client resends everything from the system prompt to the last reply each time. An input of about 500 tokens on turn one can pass 8,000 by turn ten (illustrative numbers).

Agents fill it faster

A coding agent calls the model dozens or hundreds of times per task and adds file contents and command output to the history on each call. That is why input tokens dominate agent costs.

Measuring tokens

The response's usage field reports input and output tokens. To estimate before you send, FastMetal's Anthropic-compatible API has a token counting endpoint that returns the count and generates nothing.

What happens at the limit

The input alone is too big

The API refuses the request with an error. Catch it, reduce the history and send again.

The output hits the limit

The response stops early. On OpenAI-compatible APIs finish_reason is length, and on Anthropic-compatible APIs stop_reason tells you why. No error is raised, so check it on every response.

Inside the limit, quality can still fall

Anthropic calls the loss of precision and recall as tokens accumulate context rot. Research has also found that information in the middle of a long input is easier to miss. What you send matters as much as how much fits.

Long context or RAG

There are two ways to give a model a large body of material: put all of it in the window, or use retrieval to send only the relevant parts.

RAG versus long context compares cost and accuracy in detail.

What a long context costs

LLM APIs usually bill per 1 million tokens, separately for input and output, and resent history is billed as input every time. A large window fits nearly anything, so input can grow unnoticed unless you manage the history.

On supported models, prompt caching reuses an unchanged prefix at a lower input rate. It uses the same amount of the window and changes only what you pay (Anthropic, OpenAI). Prompt caching works on Claude models through FastMetal (checked October 2026).

Per-model rates are on the pricing page, and the cost calculator gives an estimate for your workload.

How to stay inside the window

Drop old turns

Keep the system prompt and cut the history to the last few exchanges. This is the simplest fix.

Replace old turns with a summary

Compress what you cut into a short summary that keeps decisions, numbers and open questions. Use a cheap model for this, and summarize in batches so that the prefix changes less often and caching keeps working.

Shrink tool output and documents

Extract only the lines you need, keep the head and tail of long output, and replace old tool results with a note. In RAG, cap both the result count and the length.

Put stable content first

Keep the system prompt, tool definitions and fixed reference material at the top, and everything that changes after them. Caching depends on that order.

Reserve room with max_tokens

The context length minus max_tokens is roughly what you have for input.

Choose a model by its window

For whole contracts or large repositories, start with a larger window. The model catalog lists the context length of each model.

Summarize as you go, in code

This summarizes once the history passes ten exchanges and keeps the last four verbatim:

from openai import OpenAI

client = OpenAI(
    api_key="<FASTMETAL_API_KEY>",
    base_url="https://api.fastmetal.ai/v1",
)

MAX_TURNS = 10
KEEP_TURNS = 4
history = []
summary = ""


def compact():
    global history, summary
    old, history = history[:-KEEP_TURNS * 2], history[-KEEP_TURNS * 2:]
    text = "\n".join(f'{m["role"]}: {m["content"]}' for m in old)
    resp = client.chat.completions.create(
        model="deepseek-v4.1-flash",
        messages=[{"role": "user", "content": (
            f"Summary so far:\n{summary}\n\nNew conversation:\n{text}\n\n"
            "Summarize in under 100 words. Keep decisions, numbers and open questions."
        )}],
        max_tokens=800,
    )
    summary = resp.choices[0].message.content


def ask(question: str) -> str:
    if len(history) >= MAX_TURNS * 2:
        compact()
    system = "Answer concisely."
    if summary:
        system += f"\n\nSummary of the conversation so far:\n{summary}"
    history.append({"role": "user", "content": question})
    resp = client.chat.completions.create(
        model="anthropic-claude-sonnet-5",
        messages=[{"role": "system", "content": system}] + history,
        max_tokens=1024,
    )
    choice = resp.choices[0]
    if choice.finish_reason == "length":
        print("Output hit the limit. Check max_tokens and the history size.")
    print("Input tokens:", resp.usage.prompt_tokens)
    history.append({"role": "assistant", "content": choice.message.content})
    return choice.message.content

The summarizer and the main model differ only by the string passed as model.

FAQ

What is the difference between the context window and max_tokens?

The context window is the combined limit for input and output. max_tokens is how much of it the output may use. Input plus max_tokens has to fit.

How large is a typical context window?

As of October 2026, models that handle around 1 million tokens are common. A short chat needs far less. Agents and long documents benefit from the larger sizes.

Is a bigger context window always better?

It holds more. Accuracy can still drop as you fill it, and more input costs more, so send only what the task needs.

Wrapping up

Count what fills the window, keep the history short, and check finish_reason on every response.

FastMetal gives you models with different window sizes behind one key, so you can switch with one string. Create a free account to try them, or see the pricing page.