There’s a moment that happens in almost every AI startup pitch I’ve seen over the past two years. The founder clicks to a slide showing pricing, notes with visible satisfaction that AI costs have dropped dramatically, and implies, without quite saying it, that the trend will continue indefinitely. Free is coming. Margins will materialize. Just give us time.
They’re not wrong about the direction. Token prices have fallen faster than almost anyone predicted, and the trajectory is genuinely impressive. But there’s a floor no one seems to want to talk about, and it’s made of physics.
Every token a language model generates requires computation. Computation requires electricity. Electricity costs money. And while hardware gets more efficient over time, it does not get infinitely efficient. At some point, the cost of generating a token is not set by market competition or model architecture. It’s set by the laws of thermodynamics.
That floor matters more than it might seem, especially if you’re building on top of AI infrastructure, investing in companies that do, or trying to understand whether any of this actually makes money at scale.
What a token actually costs to produce
Forget the per-token pricing you see on API documentation pages. That’s the retail price, with margin baked in, designed for developers at low volume. The more interesting number is what it actually costs to run the compute underneath.
The simplified version: generating a token requires executing a certain number of floating-point operations, which require a certain amount of electrical energy, multiplied by whatever overhead your data center adds for cooling and power conversion. Put some real numbers to it, and a frontier model running at reasonable scale burns somewhere between 0.3 and 1.0 kilowatt-hours per million output tokens.
At average industrial electricity prices, that’s roughly three to nine cents per million tokens, just for the power. Current API list prices from major providers run into the dollars per million tokens. So today, energy is a rounding error. Maybe two or three percent of the cost stack.
But that math changes dramatically if token prices fall another hundredfold, which is precisely what the trajectory of the past few years suggests they might. At that point, electricity stops being a rounding error and becomes the dominant cost. The bill for the electrons doesn’t compress the way software margins do.
This is not a reason to panic. It is a reason to pay attention to which companies are building as if this future is coming.
Three curves, not one
Most of the conversation around AI costs focuses on a single curve: dollars per token, going down. That framing is useful but incomplete. There are actually three curves that matter, and they move at different speeds.
The first is cost per useful unit of work. Tokens are a means to an end. What you actually want is a task completed, a document drafted, a piece of code that runs. If a model needs 50,000 tokens to reliably complete a task that a better-designed system finishes in 8,000, the raw token price is almost irrelevant. The companies that figure out how to do more with fewer tokens will have a structural advantage that persists long after the commodity players race to zero.
The second is energy per useful unit of work. As token prices compress, this becomes the real efficiency metric. Models running on specialized hardware in regions with cheap renewable power, using techniques like speculative decoding or mixture-of-experts architectures to reduce wasted computation, will have cost structures that differ from their competitors in ways that aren’t visible from pricing pages.
The third is variance. How often does the system produce an output good enough to use without a human reviewing it? Supervision cost, the time your team spends checking, correcting, and re-running AI outputs, almost always dwarfs the raw compute cost in real workflows. A system that costs twice as much per token but requires human review half as often is dramatically cheaper in practice. Almost nobody reports this number. Almost everybody should.
Where the moats are actually forming
I’ve been doing due diligence on AI companies for long enough to notice a pattern. The pitches that excite me are increasingly not about model quality. Model quality is improving everywhere, roughly simultaneously, and access to frontier models is available to anyone with a credit card. A startup’s ability to call GPT-5.n or Claude is not a competitive advantage. It’s table stakes.
What I’m watching for instead is operational intelligence around inference. The companies building durable businesses are the ones obsessed with reducing wasted tokens, not through prompt hacks that break six months later, but through better retrieval, smarter caching, and task decomposition that actually reflects how the underlying work should be structured.
Prompt caching is underappreciated here. If a company has built workflows where large portions of their inputs are repeated across requests, and they’ve architected to take advantage of that, their effective cost per task is a fraction of what it appears. That’s a real moat, and it compounds as their data grows.
Routing is another one. The best AI operators I’ve seen don’t send everything to the most capable, most expensive model. They’ve built systems that triage: simple requests go to smaller, cheaper models; complex or high-stakes requests escalate. The result is a cost structure that looks nothing like their nominal API spend. (a lesson I learned the hard way recently with my own instance of OpenClaw).
The companies that will struggle are the ones whose entire pitch is “we use the best model and a good prompt.” That’s a product, not a business. The model gets cheaper, a competitor copies the prompt, and there’s nothing left.
The geography of inference
Something that rarely comes up in AI startup pitches, but probably should: where the compute actually runs is going to matter more as energy costs become relevant.
Data centers are not evenly distributed. They concentrate in regions with available land, stable power, and fast fiber connections. As AI inference scales, the limiting factor in many cases won’t be chip supply or model capability. It will be megawatts. Permitting a data center with enough power for serious inference workloads can take years. The companies that have secured those resources, or that are building hardware efficient enough to run at the edge where power is more diffuse, have a kind of advantage that isn’t reflected in any benchmark.
This is why “AI infrastructure” is a much bigger tent than it looks. It’s not just chips and cloud providers. It’s power purchase agreements and grid access and thermal engineering. Boring stuff, until it isn’t.
On-device inference, running models directly on phones and laptops rather than in a remote data center, gets a second wind in this environment. The latency advantages are obvious. The energy advantages are less discussed but real: the user’s device is dissipating heat into their room rather than into a cooling system drawing additional power. At scale, that’s not trivial.
What to ask when you’re doing diligence
When I’m evaluating an AI company that runs agentic workflows, meaning systems where the model is taking sequences of actions rather than answering a single question, I’ve started asking for four numbers. Most founders can’t answer all four. The ones who can are usually running a tighter operation.
Average input tokens per task. Average output tokens per task. Cache hit rate across their prompt and retrieval infrastructure. Task success rate without human intervention.
From those four numbers, you can reconstruct their real unit economics pretty quickly. Compute spend is roughly derivable from the token counts and their provider pricing. Energy floor you can sanity-check against the rough benchmark of 0.3 to 1.0 kilowatt-hours per million output tokens for frontier models, lower for smaller ones. And supervision cost, the one most founders aren’t tracking, you can estimate from success rate and whatever they’re paying the humans in the loop.
The supervision cost almost always dominates. I’ve seen companies where the AI compute is less than ten percent of their total cost of delivering a task, with the rest split between human review, infrastructure, and overhead. If the only thing you optimize is the token bill, you’re rearranging deck chairs.
The founders who understand this aren’t necessarily building cheaper products. They’re building better ones, where the feedback loops are tight enough to actually know what’s working. That’s the thing that compounds.
The fund strategy implications
If token prices keep compressing at anything like historical rates, the timeline to commodity is shorter than most people building on top of AI are planning for. Products that feel defensible today because they use a superior model or a clever prompt will feel less defensible in eighteen months. The frontier moves, and everyone moves with it.
That changes how I think about what to underwrite. Structural lock-in matters more than it used to: not just network effects in the abstract, but actual workflow integration where the product becomes harder to remove as it accumulates data about how a specific team or company operates. Proprietary data rights, with real governance, matter. Distribution wedges, the products that sit where users already spend time, matter. Capital eats labor, but you need to train the capital and just like losing a stellar employee hurts your company’s productivity, losing your AI will hurt even more.
What matters less is model arbitrage. The team that picked the right model at the right moment. That half-life is near zero.
The deeper point is about what kind of company survives a world where AI tokens are essentially free. The answer is probably the same kind of company that survives any commodity input: the one that builds something on top of it that isn’t commodity. The one that understands its cost structure well enough to optimize it. The one that earns its margin through something other than access.
Physics sets the floor. Everything above the floor is a choice.