Every API call you make has a cost, but what if the biggest cost is one you can't see? It's not on the pricing page. It's not in the SDK's documentation. It's an invisible payload that gets attached to every single request, inflating your bill and slowing your product to a crawl.
This isn't a hypothetical. As Hacker News recently highlighted, a deep-dive analysis showed that Anthropic's Claude Code model sends a massive 33,000 token system prompt before your actual prompt is ever read. For comparison, the open-source OpenCode model sends around 7,000. That's a nearly 5x difference in hidden overhead that you pay for on every single call.
This is more than a rounding error. It’s a fundamental flaw in how we evaluate and integrate third-party AI services. We look at benchmark performance and per-token pricing, but we're missing the quiet part of the equation. This hidden 'token tax' is a ticking time bomb in your application's architecture, and most teams don't even know it's there.
The Math Doesn't Lie: Calculating the Hidden Tax
Let's make this concrete. Thirty-three thousand tokens sounds like an abstract number, so let's translate it into dollars and cents, and more importantly, into product limitations.
We use a lot of AI services at VooStack, both for our internal tools and for products like NutriScan. Cost management is not just an accounting problem, it's a feature-gating problem. If a feature costs too much to run per user, you can't ship it. It's that simple.
Imagine you're building a code completion tool for your internal developer platform using a model like Claude 3 Sonnet. The pricing is $3 per million input tokens and $15 per million output tokens. Your developer types a function signature, and you send a 1,000 token prompt with the surrounding code for context. You expect a 200 token completion.
Here's the cost you think you're paying per call:
(1,000 input tokens / 1,000,000) * $3 = $0.003
(200 output tokens / 1,000,000) * $15 = $0.003
Total Expected Cost: $0.006
Now, here's the cost you're actually paying with the hidden 33,000 token system prompt:
((1,000 + 33,000) input tokens / 1,000,000) * $3 = $0.102
(200 output tokens / 1,000,000) * $15 = $0.003
Total Actual Cost: $0.105
The actual cost is 17.5 times higher than you calculated. Your budget projections are not just wrong, they're off by an order of magnitude. If 100 developers make 100 of these calls a day, you're not spending $60. You're spending $1,050. That's the difference between a cheap internal tool and a line item that gets your project killed in the next budget review.
And it gets worse. This hidden prompt eats your context window. A model might advertise a 200,000 token context window, which sounds amazing for complex tasks. But if 33,000 of those tokens are already reserved by the vendor before you send a single character, your effective context window is suddenly only 167,000 tokens. That's 16.5% of your available context gone. It's like buying a 1TB hard drive and finding out 165GB is filled with unremovable bloatware.
Beyond Cost: The Latency and Context Catastrophe
Money is one thing. Performance is another. For any user-facing AI feature, latency is the silent product killer. No one will use a chatbot that takes eight seconds to respond. No developer will keep a code completion tool that makes them wait longer than it would take to just type the code themselves.
More tokens mean more processing. The model has to read and internalize that 33,000 token system prompt every single time. This directly impacts the Time to First Token (TTFT), which is the critical metric for user-perceived performance. The stream of output tokens might be fast, but the initial pause while the model thinks can be deadly.
Let's say your product is an AI-powered customer support agent built with MailStack. A user asks a question. Your system grabs their last five emails for context (maybe 5,000 tokens) and sends it to the LLM. With the hidden prompt, you're sending 38,000 tokens just to start. The perceived lag can be the difference between a customer feeling helped and feeling ignored. This isn't just an engineering metric. It's a core business metric. A slow AI assistant can directly lead to higher churn and lower customer satisfaction.
This hidden overhead fundamentally changes the kinds of products you can build. An application that requires real-time interaction becomes non-viable if every turn in the conversation is saddled with a massive, invisible latency tax.
Why Do Models Do This? The Black Box Tradeoff
So why would a vendor do this? It's not pure malice. That giant system prompt is doing a lot of heavy lifting. It's likely filled with complex instructions, persona definitions, ethical guardrails, and fine-tuning examples to make the model behave in a very specific, highly-curated way.
In effect, you're paying for a pre-packaged, opinionated product. Anthropic has done a ton of prompt engineering for you to ensure the model is helpful, harmless, and good at coding tasks out of the box. That's a feature.
But it's a feature that comes with a massive tradeoff. You're trading transparency and control for convenience. You get a model that
Building something in this space? AgileStack helps teams ship enterprise-grade software without the consulting-firm overhead. Book a 30-minute call and tell us what you're working on.