Token Compression for LLMs:
The Complete Guide

What Is Token Compression?

Token compression is the practice of reducing the number of tokens sent to a large language model (LLM) during inference — before the model processes them — while preserving the information needed to answer the current query. It is a middleware optimization that sits between your application and the LLM API.

Every LLM call processes the full prompt you send, including context that may be irrelevant to the question being asked. In agent loops, RAG pipelines, and coding assistants, context accumulates rapidly: conversation history, tool outputs, retrieved documents, and system instructions all get sent on every turn. Token compression removes the low-value tokens before they ever reach the model.

The key distinction: token compression is not summarization. Summarization rewrites context (costing an extra LLM call and risking information loss). Token compression selectively removes lines that score low against the current question, keeping the original text intact.

Why Token Compression Matters

Token compression addresses three converging pressures on every LLM-powered application:

Cost

LLM APIs charge by the token — both input and output. If 65% of your input tokens are irrelevant to the answer, you're paying for compute that doesn't improve your results. For a typical agent making 100 calls/day, that's thousands of wasted tokens daily. At scale (1M+ calls/month), the savings from token compression reach thousands of dollars.

Latency

LLM inference time scales with input length (especially with attention's quadratic complexity in prefill). Compressing tokens before inference means faster responses. In agent loops, where context grows with every turn, token compression keeps latency from ballooning over time.

Environmental Impact

Every token processed by a GPU consumes electricity and cooling water. Data centers used 415 TWh globally in 2024 and are on track to double by 2030. Token compression directly reduces the compute needed per query. At 10M calls/month, SuperCompress avoids ~120 kg CO₂ and ~1,000 L of cooling water.

Token Compression Methods Compared

Not all token compression is equal. Here are the four main approaches, ranked by effectiveness:

Benchmark Results

Compiler benchmarks measure real API-call behavior: original tokens, kept tokens, tokens removed, important context kept, and verifier risk. Fixed-ratio oracle-recall baselines remain useful for research comparisons.

* Summarization latency excludes the extra LLM call cost. SuperCompress requires zero GPU time.

Compiler Mode Results

In compiler mode, SuperCompress removes as many tokens as it safely can for each API call, then reports important context kept and verifier risk:

View full benchmark details →

Cost Savings Analysis

Token compression reduces both token count and the GPU prefill time that billing is based on. Here's what that means in real dollars:

* Estimated based on published API pricing and SuperCompress assumptions. Your savings depend on your average context length and query patterns.

Token compression is most valuable in these scenarios:

• AI agents with multi-turn conversations where context accumulates
• RAG pipelines sending large document chunks with every query
• Coding assistants processing full file contents and conversation history
• Batch processing at scale where marginal savings compound

How to Implement Token Compression

Token compression can be added to any LLM pipeline with minimal code changes. Here's how to integrate SuperCompress:

Python (recommended)

# Install: pip install supercompress
from supercompress import Compressor

compressor = Compressor()

	# Compiler compression — no budget required
	result = compressor.compress(
	    context=long_prompt,
	    query="What caused the production incident?"
	)

print(f"Removed {result.tokens_removed} tokens ({result.savings_pct:.0f}% savings)")
print(f"Compressed text:\n{result.compressed_text}")

REST API

# Quick one-liner (form-encoded POST)
curl -d "context=Your long prompt here...&query=What should the model answer?" \
  https://supercompress.dev/compress \
  -H "X-API-Key: sc_live_your_key"

# Or as JSON (standard)
curl -X POST https://supercompress.dev/api/v1/compress \
  -H "X-API-Key: sc_live_your_key" \
  -H "Content-Type: application/json" \
  -d '{"context":"Your long prompt here...","query":"What should the model answer?"}'

# Or GET with query params
curl "https://supercompress.dev/compress?context=...&query=...&api_key=sc_live_your_key"

Integrate with LangChain

from supercompress import Compressor
from langchain.callbacks import BaseCallbackHandler

class CompressionCallback(BaseCallbackHandler):
    def __init__(self):
        self.compressor = Compressor()

    def on_llm_start(self, serialized, prompts, **kwargs):
        # Compress prompt before sending to LLM
        compressed = self.compressor.compress(
            context=prompts[0].text,
            query="Continue the conversation."
        )
        prompts[0].text = compressed.compressed_text
        return prompts

SuperCompress: Open-Source Token Compression

SuperCompress is an open-source (MIT) learned token compression engine. It uses a query-aware neural policy to score every line of context against your question and keeps only the lines that matter for the answer. It runs entirely on CPU, adds ~60ms latency, and requires zero GPU time.

• Token savings: 82.5% average on bundled long-context presets
• Safety signal: important context kept and compression risk on every compiler result
• Latency: ~60ms on CPU — no GPU or extra LLM calls
• Model-agnostic: Works with OpenAI, Anthropic, local models
• Query-aware: Scores context against the current question, not just position
• Free allowance: 5M tokens/month ($5) on the hosted API

Get an API key See benchmarks

Frequently Asked Questions

Does token compression work with any LLM?

Yes. Token compression operates on the text before it's sent to the model. It works with OpenAI, Anthropic, Google, open-weight models, local models — any LLM that accepts text input.

Does token compression reduce answer quality?

Not when done correctly. SuperCompress achieves 100% oracle recall on the fixed-budget public suite, and ≥98% gold-answer containment on held-out long-context mixes — meaning answer-critical evidence survives compression. The model receives the same signal — just with less noise.

How is token compression different from prompt engineering?

Prompt engineering changes how you write prompts. Token compression reduces how much context you send. They're complementary: good prompt engineering + token compression = best results.

Is token compression the same as KV cache eviction?

No. SuperCompress never touches the model’s KV cache. Token compression happens before inference on the prompt text. KV eviction (if any) happens inside the provider/model during inference. Shorter prompts mean less prefill work as a side effect — that is not the same as compressing “in KV.”

Can I use token compression with streaming?

Yes. SuperCompress compresses the full context before the streaming LLM call begins. The compressed text is sent as the prompt, and the response streams back normally.

Does SuperCompress require a GPU?

No. SuperCompress runs entirely on CPU. The model is only a learned query-aware neural policy — tiny enough to run in <10KB of memory. It compresses before GPU inference, so it doesn't compete for GPU resources.

How much does SuperCompress cost?

The Python library and browser demo are free (MIT license). The hosted API includes $5 free tokens per month (5M), then pay-as-you-go at $1 per 1M tokens.