Background: A Decade of Custom Silicon Reaches an Inflection Point
Google's Tensor Processing Unit (TPU) program has run for roughly ten years as a unified accelerator serving both model training and inference. <cite index="26-1">At Cloud Next 2026 on April 22, Google split TPU 8 into TPU 8t (training) and TPU 8i (inference) — the first time the TPU program has separated training and inference silicon in its ten-year history.</cite> The architectural bifurcation marks a deliberate bet that the economics of AI are now dominated by serving rather than by building models.
The Two Chips
<cite index="21-1">The TPU 8t was engineered for large-scale training and designed with Broadcom, while the TPU 8i was engineered for reasoning and serving and designed with MediaTek.</cite> The division of labor is intended to eliminate the performance compromises inherent in general-purpose Application-Specific Integrated Circuits (ASICs).
<cite index="19-3,19-4,19-5">The TPU 8i, codenamed Zebrafish, is the more technically distinctive of the two chips; rather than chase training flops, it is engineered around latency — the metric that determines whether an AI agent feels real-time or sluggish. Each TPU 8i delivers 10.1 FP4 petaflops of compute, fed by 384 MB of on-chip SRAM (triple the prior generation), 288 GB of HBM, and 8.6 TB/s of memory bandwidth.</cite>
<cite index="22-7,22-8">The large SRAM enables the active working set of a large model to remain entirely on-chip during inference, eliminating the latency of loading weights from HBM on every token generation. The chip uses a new Collective Acceleration Engine that offloads distributed inference communications — the all-reduce and all-gather operations that dominate synchronization overhead in large multi-chip inference clusters.</cite>
Performance-Per-Dollar Claims
<cite index="27-1">According to Google's own Cloud Blog technical deep dive, TPU 8i delivers up to 80% performance-per-dollar improvement over Ironwood TPU, particularly at low-latency targets for large Mixture-of-Experts (MoE) models.</cite> When stacked against earlier generations, <cite index="21-5">Google is mapping to roughly 6x more compute per unit of electricity versus five years ago, aiming at up to 2x performance-per-watt over Ironwood.</cite>
This efficiency trajectory compounds gains Google had already recorded internally. <cite index="4-1">On its 2025 earnings call, Google said it had cut Gemini's per-unit serving cost by roughly 78% that year, crediting model optimization plus efficiency and utilization gains from its own hardware.</cite> <cite index="20-13,20-14">Gemini is served exclusively on TPUs, while competing models such as Claude and GPT-5.5 are not, meaning the fundamentally different architecture that Gemini runs on is a key driver of Google's lower cost per token.</cite>
Ecosystem and Anchor Customers
<cite index="26-4">Anthropic anchors both generations with a 3.5 gigawatt capacity deal for 2027; Google simultaneously inked a Vera Rubin partnership with NVIDIA (Graphics Processing Unit), framing the play as a multi-silicon "AI Hypercomputer" rather than a head-on replacement strategy.</cite> <cite index="22-10">Both TPU 8t and 8i will be generally available on Google Cloud Platform later in 2026, accessible as standalone instances or as part of Google's AI Hypercomputer full-stack platform.</cite>
Gemini 3.5 Pro Delay
The infrastructure announcement arrives against a backdrop of model-level turbulence. <cite index="11-1">Google launched Gemini 3 in late 2025, and the 3.5 generation was supposed to continue a rapid release cadence, with the Flash version announced in May and the Pro version promised a month later.</cite> <cite index="12-4">Google's most powerful flagship AI model, Gemini 3.5 Pro, has fallen months behind schedule as engineers work to improve its coding capabilities, according to Bloomberg.</cite> <cite index="11-4">Google updated the data used to train Gemini in an attempt to improve its coding skills, and the results were disappointing.</cite>
<cite index="14-11">Providing comment on the matter to Bloomberg, Google said: "We're shipping quickly across a wide range of models while keeping them highly cost-effective for customers."</cite> <cite index="10-8">The report hit investor sentiment immediately, with Alphabet shares falling about 4% after the news rolled out.</cite> <cite index="11-5">Google has not announced a new launch date.</cite>
Structural Significance
<cite index="5-5,5-6">The TPU 8i chip is part of a broader infrastructure push that extends Google's decade-long custom silicon program into a new phase explicitly designed around the inference economics of Large Language Models (LLMs) rather than training alone — because inference is now the dominant cost driver across frontier AI.</cite> <cite index="20-3,20-4,20-5">According to industry analysts, inference could take the larger share of compute demand as soon as 2027; McKinsey estimates that in 2026, training and inference will each account for roughly 31.2 gigawatts of data center demand, but by 2027, inference becomes the larger share.</cite> Whether the silicon advantage translates into a durable competitive moat will depend in part on whether Google can close the model-capability gap that the Gemini 3.5 Pro delay has exposed.