Best practices
When to use prompt compression
Prompt compression is not always the right answer. Here is a decision framework to help you determine when to compress and when to skip.
Compress when:
- Context exceeds 1,000 tokens, Below this, the savings may not justify the overhead
- Context contains irrelevant information, Standards table with update data for many questions, only some questions touch most standards
- Cost is a primary concern, If you are tracking LLM spending, compression is your highest-ROI optimization
- Long sessions hit the context limit, Trimming large tool outputs before they reach the model leaves more room in the context window for the actual task
Skip when:
- Context is under 500 tokens, Minimal savings, and the compression overhead may not be worth it
- You need the full original text, Some applications, such as legal review or exact audits, must pass the complete source to the model, so compression is the wrong tool
- You are debugging prompt quality, Compression adds a variable that complicates debugging
Frequently asked questions
Should I always compress RAG contexts?
Yes. RAG contexts are the highest-ROI target for compression due to their size and noise level.
Should I compress few-shot examples?
Not usually. Few-shot examples are carefully chosen and typically small. Compression is more valuable on large context blocks.