Context Engineering for Long‑Horizon Agents

A field playbook for managing the context window across long-running, multi-turn agents — what breaks, why, and the levers that actually work.

Working notes on managing the context window for agents that run for many turns. The focus is on the substance: what the problem actually is, what works in production, and the best practices to apply. Metrics are kept as reported in the underlying research.

1What context engineering is (and is not)

Prompt engineering is about how to write good instructions — mostly the system prompt. Context engineering is broader: it owns the entire token lifecycle of a multi-turn agent, from the first system-prompt token to the last compacted summary. In practice it means curating and maintaining the optimal set of tokens available to the model during inference, including everything that lands in the window beyond the prompt itself.

The right mental model is to think in context: consider the full state the model sees at each step and what behavior that state is likely to produce — not just whether the system prompt reads well. Operationally that covers which retrieved documents to include and when to drop them, which tool outputs to keep versus clear, when to summarize a growing history, how to route subtasks to isolated subagents, and how to keep persistent notes outside the window.

Why it matters: production agents fail at the context layer far more often than at the prompt-writing layer. Agents are not one-shot chatbots — they run for hundreds of turns and accumulate tool outputs. The canonical components that belong in the window are: task description, few-shot examples, retrieved knowledge (RAG), relevant multimodal data, tools, state, history, and compaction.

What it is not — common misconceptions

• It is not prompt engineering. Prompt engineering asks how to word effective instructions for a largely one-shot exchange. Context engineering owns the full token lifecycle of a sustained run: what to add, what to remove, when to compress, what to retrieve just-in-time, how to route to subagents, and what to persist. Good wording does not save an agent that drowns in accumulated tokens.
• It is not “use the biggest context window.” A larger window is not a fix. Performance still degrades as the window fills, so more capacity mostly buys the ability to fail more slowly. The goal is keeping the right tokens, not cramming in more.
• It is not RAG alone. Retrieval is one lever (Select). On its own it does nothing about history bloat, tool overload, or conflicting information across turns.
• It is not just compaction. Compaction is the most visible lever but only addresses distraction and token spend. Teams that stop there still hit confusion (too many tools) and clash (multi-turn accumulation).
• It is not a one-time setup. It is a per-step discipline over the whole run — context has to be curated continuously, not configured once at the start.
• It is not primarily about model choice. Token discipline drives reliability far more than picking a bigger model; the same model swings widely depending on how its context is managed.
• More context is not more grounding. Extra tokens that are not high-signal add cost and reduce performance — they distract, confuse, or poison rather than help.

2The empirical evidence: “context rot”

The core finding is that model performance degrades as the input token count grows, across every major model tested. Larger context windows move where the degradation starts; they do not remove it. This was shown by extending the classic Needle-in-a-Haystack test — which only checks direct lexical matching — with semantic matching and a conversational QA evaluation, to separate context-length effects from raw task difficulty.

Why it happens is architectural:

• Attention cost scales quadratically — every token attends to every other token (n² pairwise relationships), so representational capacity gets stretched as context grows.
• Models are trained mostly on shorter sequences, so fewer parameters specialize in long, context-wide dependencies.
• Position-encoding interpolation extends sequence length but degrades token-position understanding.

Retrieval correctness for a large 405B-class model started falling around the 32k-token mark, and earlier for smaller models. Smaller models hit their distraction ceiling well before their window is full.

3Failure taxonomy: four ways agents break

Under context pressure, long-horizon agents fail in four distinct ways. Each maps to a primary fix and has its own detection signal.

Two notes. Poisoning and distraction often appear in the same agent — they are two expressions of the same root cause: unmanaged context accumulation. And context clash is not a small-model problem; it hits frontier models too. When prompt information is spread across multi-turn exchanges (exactly how agents operate), accuracy can drop sharply versus the same information given in one shot.

4The four levers: Write, Select, Compress, Isolate

These levers are layered, not competing — most production agents need all four running at different points in the context lifecycle.

• Write — persist information outside the window: structured note-taking, external memory, checkpoints. Turns multi-hour runs into resumable processes instead of brittle stateless sessions.
• Select — pull context just-in-time. Keep lightweight identifiers (file paths, stored queries, links) and load data only when needed; use file search (glob/grep) for on-demand access. Applying RAG to tool descriptions (not just documents) can improve tool-selection accuracy by roughly 3×.
• Compress — summarize or compact history and clear processed tool outputs to cut token spend and reduce distraction.
• Isolate — route subtasks to subagents with their own windows; each returns a condensed summary so the lead's context stays clean.

5Compaction, in depth

Compaction is the primary lever for long-horizon runs — hundreds of turns, hours of wall-clock time, or workflows that would otherwise exhaust the window. Reported effectiveness is strong: in a 100-turn web-search evaluation, context editing reduced token consumption by ~84% while enabling completion of workflows that would otherwise fail from context exhaustion. Context editing alone gave a ~29% performance lift; combined with a memory tool it reached ~39%.

The real question is not whether to compact, but what to preserve. Off-the-shelf summarization does not reliably keep key decisions for long-running tasks — a fine-tuned compaction model is more reliable in production.

On triggers: a practical auto-compact threshold is ~95% of context-window usage. For programmatic pipelines, prefer an automatic context-editing mechanism that clears stale tool calls/results (finer control) over the manual, product-layer auto-compact meant for interactive sessions.

6Token budget by agent type

Recommended ceilings per component slot, assuming a 200k total budget at steady state. The ratios are more stable than the absolute numbers — scale proportionally for other window sizes.

Steady-state rule of thumb for a 200k budget: system prompt ~3k, tool defs ~5k, RAG ~20k (the largest variable slot — size it to the task), message history ~12k (compress before it grows past ~15k), and keep ~160k of headroom for output and tool responses. Filling the window is itself the failure mode.

7Multi-agent architecture: isolation as strategy

In a detailed research-agent case study, token usage explained ~80% of performance variance on a browsing benchmark, with tool-call count and model choice as the other two factors. Translation: context management is the primary performance lever, not model selection.

How it works: give each subagent its own isolated window; subagents return condensed summaries (~1k–2k tokens) to the lead rather than their full context. The lead accumulates distilled insights, not raw trails — which prevents context clash from parallel workstreams merging into one confused window.

• Simple fact-finding: 1 agent, ~3–10 tool calls.
• Direct comparisons: 2–4 subagents, ~10–15 calls each, run in parallel.
• Complex research: 10+ subagents with clearly divided responsibilities.

Spawning 3–5 subagents in parallel (versus sequentially) reportedly cut research time by up to ~90% on complex queries. And tool descriptions are themselves a context lever: rewriting flawed descriptions cut task-completion time by ~40% for later agents. Poorly written descriptions waste tokens explaining themselves and confuse the model — auditing them is a high-ROI, low-effort win.

8System-prompt “altitude”: the Goldilocks zone

Two failure modes bracket the optimal range:

• Over-specification — brittle if/else hardcoded logic that breaks when reality deviates from the anticipated case.
• Under-specification — vague, high-level guidance that assumes shared context the model does not have.
• The optimal zone — specific enough to guide behavior, flexible enough to act as strong heuristics.

Best practices: structure prompts for scannability with distinct sections (XML tags or Markdown headers for background, instructions, tool guidance, output description) — this helps both model comprehension and token efficiency. The guiding principle for every context component is the smallest set of high-signal tokens that maximizes the likelihood of the desired outcome. Agent prompts are not chatbot prompts: chatbot prompts tend to be exhaustive to handle diverse one-shot queries, whereas agent system prompts should be concise and defer to dynamically-selected context for specifics. Think “guide a sustained, tool-using process,” not “answer every possible question.”

Key takeaways

• Token discipline — not model choice — is the real reliability lever. A model that scores ~98 on a clean single prompt can drop to ~64 when the same info is spread across a multi-turn run.
• Token accumulation is the primary failure mode for long-horizon agents, and it affects frontier models as much as smaller ones.
• Map each failure to a lever: just-in-time Select prevents confusion (tool overload); Compress/compaction addresses distraction from history buildup; subagent Isolate prevents clash from merging information; structured external Write prevents poisoning from persisted bad state.
• Applying all four levers together is what separates production-grade agents from demo-grade scaffolding.

← Back to Coding