The way AI infrastructure is judged is shifting quickly. Raw GPU power still matters, but the metric drawing more attention is cost per token, because that is closer to how generative AI services are actually measured in business use.
That change matters because the value of an AI system is no longer defined only by the hardware inside a data center. It is increasingly defined by how efficiently that system can turn compute into tokens, at scale, with stable performance and controlled operating costs.
Why cost per token is becoming the key metric
Cost per token reflects the full efficiency of an AI stack rather than the strength of one component alone. It captures hardware, software, networking, and system utilization in a single figure that links directly to service output.
For companies deploying generative AI, that makes the metric easier to use in real business planning. The central question is not simply which GPU is fastest, but how little it costs to produce large numbers of tokens consistently.
A new way to view data centers
This shift is changing how infrastructure is described and evaluated. NVIDIA has framed modern data centers as “AI token factories,” a view that pushes attention away from hardware prestige and toward production efficiency.
In that model, success is no longer measured only by FLOPS per dollar. The more relevant question is how many tokens a system can produce at the lowest possible cost while maintaining service quality.
Hopper and Blackwell show the gap
NVIDIA’s comparison between Hopper and Blackwell illustrates how wide the efficiency gap can be. Hopper is described as generating around 90 tokens per second per GPU at a cost of about USD4.20 per million tokens.
Blackwell performs on a very different level. It is described as producing around 6,000 tokens per second per GPU, with a cost of only USD0.12 per million tokens.
Even though Blackwell carries a higher rental price, the much larger output changes the economics. The result is an operational cost reduction of up to 35 times.
Software techniques also push costs down
Lower token costs are not driven by new hardware alone. NVIDIA also points to several optimization methods that help reduce spending while improving token generation efficiency.
Those methods include FP4 precision, speculative decoding, multi-token prediction, and KV-cache offloading. Used together, they help systems produce more output without increasing costs at the same pace.
For AI providers, that matters because it allows higher throughput without a proportional rise in operating expense. In practical terms, it gives service operators more room to scale.
Cloud partners are already moving
The shift is also visible among cloud partners. CoreWeave, Nebius, Nscale, and Together AI are among the providers said to be adopting Blackwell.
Their aim is straightforward: deliver AI services with the lowest possible token cost. That approach can improve margins, support more competitive pricing, and make total cost of ownership, or TCO, a more relevant way to assess infrastructure choices.
What this means for the AI business model
As token costs fall, AI companies gain more flexibility in how they package and price their services. Lower operating costs can create room to expand offerings while keeping prices more competitive for customers.
It also changes how infrastructure investments are evaluated. A GPU remains important, but the more meaningful measure is now how effectively the system converts computing power into tokens that users actually consume.
That is why the conversation around AI infrastructure is moving away from expensive hardware as the main headline. The real competition is increasingly about efficiency, and cost per token is becoming the clearest way to see it.
Source: www.medcom.id
