For decades, the CPU was treated as a generic resource: you increased the frequency, added cores, and processing problems simply resolved themselves. This convenience has created a dangerous intellectual addiction. Technology leaders began to ignore hardware as a strategic variable, delegating the choice to infrastructure teams who bought what was in the catalog. It was an acceptable simplification in a world where business workloads — reports, transactions, web pages — had similar computational profiles. This model worked as long as workloads were relatively homogeneous. With generative AI, large-scale inference, and embedded computing, it has broken for good. Hardware is once again a choice with real consequences.
Why the CPU lost its leading role
The CPU is designed for versatility. It executes sequential instructions with lots of control logic, sophisticated caches, and jump prediction mechanisms. For a transactional database or web server, this is exactly what you need. To train a neural network with billions of parameters is a monumental waste. Most of the silicon is occupied with logic that is never activated in that type of load.
The core problem is that machine learning operations are mathematically repetitive. Matrix multiplications, convolutions, activation operations: all of these benefit from massive parallelism, not flexibility. The GPU was the first to exploit this, with thousands of simple cores operating together. But the GPU is also a compromise, originally designed for graphics and adapted for AI. The result is capable hardware, but with energy and cost efficiency that still leaves room for something more specialized.
What TPU, NPU and ASIC do differently
The TPU, developed by Google, was built specifically for large-scale linear algebra operations. It doesn't try to be an AI-capable general-purpose processor: it does one thing very well, and that's running the tensor operations that underpin TensorFlow and, more recently, JAX models. The cost per operation drops dramatically when the hardware does not carry the weight of features that will never be used in that context.
The NPU, in turn, plays a different game. Instead of training models in the cloud, it runs inference directly on the device, whether it's a smartphone, an industrial sensor, or a self-driving car. Apple has integrated NPUs into A-series chips since 2017 and the results are visible: facial recognition, real-time translation and local text generation that do not depend on connecting to servers. Qualcomm and MediaTek have followed suit with their SoCs. What seemed like a distant future has become the standard in premium phones and is quickly descending into the mid-range market.
ASIC represents the extreme of specialization. You design a chip to solve a specific problem and only that problem, with maximum efficiency and very low marginal cost in volume production. Bitcoin miners were the first to demonstrate the scale of this reasoning. Today, companies like Groq and Cerebras build ASICs for LLM inference with latency and cost per token that general-purpose GPUs simply can't match in the same power envelope.
The FPGA as an intermediate point that few consider
Between the ASIC and the GPU there is a territory little explored by product teams: the FPGA. The programmable field of logic gates allows the hardware to be reconfigured after manufacturing, which eliminates the risk of relying on a fixed design before the product is mature. Microsoft uses FPGAs on Azure servers for network acceleration and model-specific inference. Intel bought Altera precisely to capture this market.
For products that need custom acceleration but don't yet have enough volume to justify their own ASIC, the FPGA offers a smart middle ground. The cost per unit is higher than a volume ASIC, but the development risk is drastically lower. Teams that ignore this option tend to fall into one of two mistakes: they get stuck with the CPU out of conservatism or they bet on ASIC too early and have to redo the design when the product changes. The FPGA buys time to learn without wasting the investment.
The chip decision as a product decision
Here is the point that most organizations have not yet internalized: the choice of hardware is no longer exclusively an infrastructure decision and has become a product variable with a direct impact on cost, latency, privacy and user experience. The silicon layer is important again, and ignoring it is an advantage that you are giving up to your competitor.
When you decide to run AI inference on your device instead of in the cloud, you're choosing data privacy, zero latency, and offline operation. This is not a technical preference. It's a product promise. And it depends on an adequate NPU on the target hardware. If your target audience uses devices with chips that don't have an efficient NPU, the product experience will be bad regardless of how good the model is.
Likewise, the decision to use Google Cloud's proprietary TPUs versus NVIDIA's GPUs versus ASICs like Groq's isn't just about price per compute hour. It's about response latency, about the ability to scale demand peaks, about the cost per query that your business model needs to support. Product teams that leave this decision entirely to infrastructure are, in practice, outsourcing a central competitive variable.
The correct strategic question is no longer "how many GPUs do we need?" and became "what chip geometry best serves this specific workload, in this deployment context, with this cost and latency envelope?" This question requires product, engineering and infrastructure to sit at the same table. Organizations that still treat hardware as a procurement detail will find this out the hard way when a competitor delivers the same functionality at a third of the operational cost because they made that choice deliberately.
Also read
- Inference chips and ASIC: when the specialized beats the generic
- The new generation of chips: GPU, NPU, ASIC and RISC-V for those who decide
- AI PCs and NPU: what changes when the device gains AI accelerator
- Neuromorphic and bio-inspired computing: when the chip starts to learn from the brain
- How to build a business case for AI in industrial operations
- AI Energy and Sovereignty: What Governments Need to Plan Now