THE SHORT ANSWER
It dramatically increases cost and latency while degrading the model's ability to retrieve precise facts from the middle of the prompt.
Flashcards
Q1
What is the 'Lost in the Middle' phenomenon?
LLMs are highly accurate at using information at the beginning and end of a long context, but struggle to extract facts from the middle.
Q2
How do system prompts impact token limits?
System prompts consume tokens permanently across the entire session, reducing space available for user input and context.
Q3
Why is retrieval (RAG) better than context stuffing?
RAG injects only the mathematically relevant snippets, keeping prompts cheap, fast, and highly focused.
Related Concepts
Chaos Stack Field Notes FAQs
What are Chaos Stack Field Notes?
Chaos Stack Field Notes are technical flashcards that explain core engineering concepts quickly.
How are these different from topics?
Topics are broad thematic hubs that connect characters, episodes, and environments. Field Notes are short, direct Q&A flashcards for quick technical alignment.
AI Summary
This page covers Token Economics and Context Windows as a technical flashcard. Description: Why is maximizing an LLM context window usually a bad idea?.