Prompt compression guide
Prompt compression that keeps the answer
Prompt compression is the practical way to reduce oversized LLM requests before they hit an expensive model. Also called smart context compression when the same query-aware policy runs on retrieved dumps, agent memory, or RAG chunks. SuperCompress scores context against the current question and keeps the lines most likely to matter.
Held-out proof (updated 2026-08-01)
Gates: ≥98% gold-answer containment, ≥55% mean cut, ≥55% token-weighted cut. No restore-to-98% floor.
| Suite | Answer kept | Mean cut | TW cut | Gates |
|---|---|---|---|---|
| Real (seed 4242) | 98.4% (61/62) | 57.8% | 64.7% | pass |
| Fresh4 (seed 9091) | 100% (57/57) | 66.1% | 68.8% | pass |
| Fresh5 (seed 6161) | 100% (62/62) | 57.8% | 61.6% | pass |
| Fresh6 cold (seed 3377) | 98.6% (70/71) | 68.2% | 72.5% | pass |
Primary bundle (real + fresh4 + fresh5): 99.4% pooled answer keep (180/181), 65.4% pooled token-weighted cut. Full tables: benchmarks.
Why teams pick SuperCompress over Headroom
- True query-awareness — scores blocks against the question; Headroom leans on content-type heuristics.
- Coding-agent MCP first —
npx supercompress setupfor Cursor, Claude Code, Codex; keep your login. No provider API-key mode required. - Agent + API ready — hosted compress API, open-source library, and MCP install for coding agents — not only a local CLI workflow.
- Held-out answer gates — we publish answer containment on LongBench + OOD mixes, not only “token % saved.”
Why prompt compression matters
Every LLM call processes your full prompt including context that may be irrelevant to the question. In agent loops, RAG pipelines, and coding assistants, context accumulates rapidly.
Without prompt compression, you pay for every token regardless of relevance. A typical RAG query sending 4,000 tokens may only need 400 of them to answer the question.
How prompt compression works
SuperCompress uses a learned query-aware neural policy that scores every block of context against your question, then keeps only the blocks most likely to contain answer-relevant information.
from supercompress import Compressor
compressor = Compressor()
result = compressor.compress(
context=long_context,
query="What caused the failed deployment?"
)
print(f"Removed {result.tokens_removed} tokens")Prompt compression vs summarization
| Factor | Prompt Compression | Summarization |
|---|---|---|
| Method | Selects original lines | Rewrites with LLM |
| Extra cost | ~60ms CPU | Full LLM call |
| Oracle recall | 100% | ~61% |
Cost savings from prompt compression
| Scenario | Daily Calls | GPT-4o Monthly Savings |
|---|---|---|
| Small agent | 100 | ~$20 |
| Medium RAG app | 1,000 | ~$295 |
| Enterprise | 500,000 | ~$98,500 |
Frequently asked questions
Is prompt compression the same as summarization?
No. Summarization rewrites the prompt using another LLM call, changing wording and potentially losing facts. Query-aware compression selects original lines that matter for the current question.
How much can prompt compression save?
SuperCompress averages 82.5% savings on bundled long-context presets.
Does prompt compression add latency?
SuperCompress adds ~60ms on CPU, often compensated by reduced GPU prefill time.