Custom AI Silicon Exits Internal Confinement
Google recently shifted from releasing one TPU every two years to two in a single year, initiated selling the chips outside the company, and opened the PyTorch ecosystem with a new backend called TorchTPU. Anthropic committed to more than a million chips, Meta tests TPU capacity, and early benchmarks position Ironwood under Blackwell on cost per token. GPU shipments now cover a smaller share of the market. Here is what changes for silicon teams and IT buyers.
Contributed By Jon Peddie – Jon Peddie Research
Google compressed its TPU development cycle this year. Amin Vahdat, Google’s chief technologist and SVP for AI infrastructure, said that Google moved from shipping a TPU roughly every two years to two generations within a single year, expecting that pace to increase. Alphabet backs that roadmap with capital: 2026 capital spending guidance sits at $195 to $205 billion, supporting next-generation AI data centers. That cadence demands aggressive chip design. Each new processor forces Google and its partners to coordinate architecture, verification, packaging, memory, boards, networking, cooling, manufacturing, and software on shorter schedules.
Google’s newest chips illustrate specialization directly: TPU 8t targets training, and TPU 8i targets inference, marking the first time Google split training and inference into distinct architectures instead of shipping one design tuned for both. Training and inference require different compute density, memory, communication, power, and utilization. A hyperscaler operating at Google’s scale optimizes silicon around those specific characteristics. TPU 8i replaces the torus network topology used in earlier TPU generations with a new design called Boardfly, a flatter, high-radix fabric that cuts network diameter by more than half compared with a comparably sized 3D torus. TPU 8i also doubles ICI bandwidth to 19.2 terabits per second and triples on-chip SRAM to 384MB, sized specifically to hold reasoning and agentic model KV caches on-chip.
Figure 1. How Google’s TPU stack reaches external PyTorch inference workloads.
Ironwood, Google’s current TPUv7 generation, breaks from the fused-core convention that defined TPU v4 and v5p. Instead of sharing one unified memory space across two physical cores, Ironwood carries two separate compute dies connected by a high-bandwidth die-to-die link, each exposed as an independent logical device. Each chip carries 2 TensorCores and 4 third-generation SparseCores, the latter accelerating embedding lookups and other sparse operations. Memory capacity runs roughly 6 times Trillium’s HBM, and Ironwood marks the first TPU generation with native FP8 hardware support. The matrix multiply unit doubled from 128×128 to 256×256 starting with TPU v6e, delivering 4 times more FLOPs per cycle. That larger array pays off when matrix dimensions actually fill it: Llama 3 8B’s 128-wide attention head dimension caps utilization at 50 percent on Ironwood’s 256-wide array.
Google integrates networking as an architectural component. Vahdat says Google builds the equivalent of an Ethernet switch into each TPU directly, extending that same approach from individual processors up through entire data centers. TPUs connect peer-to-peer over a custom Inter-Chip Interconnect that bypasses the host CPU entirely, arranged in a 3D torus where each chip wires to 6 neighbors, scaling all the way to a 9,216-chip superpod. Google links multiple pod-scale cubes together using optical circuit switches, letting it physically reroute around a failed link or dead chip in seconds using mirrors.
Independent benchmarking backs the architecture claims with numbers. At an interactivity of 100 tokens per second per user, Ironwood costs about $0.181 per million total tokens, versus $0.222 for Nvidia’s B200 and $0.276 for B300. At lower interactivity, 20 tokens per second per user, Ironwood leads on raw throughput too, hitting 9,364 total tokens per second per chip against 8,903 on B200 and 8,925 on B300. Nvidia retains an advantage specifically on FP4 workloads, since Ironwood lacks native FP4 hardware entirely; that gap closes once TPU 8i ships with native FP4 support built in. Disaggregated serving remains Nvidia’s other current edge.
Table 1. Ironwood beats Blackwell on cost, loses on first-token latency.
Real customer adoption backs these numbers beyond internal Google workloads. Anthropic committed to more than 1 million TPUs, roughly 400,000 through direct purchase and 600,000 rented through Google Cloud, maintaining a multi-sourced strategy across AWS Trainium and Nvidia GPUs. Meta tests TPU systems for both training and inference too. Citadel Securities flagged performance advantages in specific latency-sensitive workloads, and G42 explores TPUs for national-scale AI infrastructure. Software externalization runs through a new stack called TorchTPU. TorchTPU exposes TPUs as native PyTorch devices directly, letting vLLM and SGLang reuse upstream model code.
Nvidia responds to this competitive pressure directly. Its multibillion-dollar licensing and talent deal with Groq brings low-latency inference expertise into Nvidia’s own platform, and Nvidia now deploys hybrid designs pairing GPUs with Groq-derived LPU technology, positioning GPU-based systems for large-scale training and prefill, and ASIC-style designs for low-latency decode. JPR estimates worldwide data center compute and IT hardware spending at roughly $800 billion in 2026, with GPUs and AI accelerators together representing about $350 billion of that total, a pool Nvidia, Google, AWS, Microsoft, Meta, and AMD now divide.
Table 2. JPR’s 2026 hardware pool, accelerators against everything else.
Google’s TPU strategy shows how AI infrastructure moved from a processor contest to a full systems-engineering contest, spanning compute, memory, networking, packaging, manufacturing, and software. Google pursues that integration at scale, and its real challenge runs through execution: getting each new TPU generation through manufacturing and into productive systems before another architecture, workload, or supply constraint shifts the target underneath it.
What do we think?
Google turned custom silicon into an infrastructure strategy. Two TPU generations in a year and $195–$205 billion of capital spending put pressure on design, manufacturing, and deployment at once. TorchTPU matters because framework parity decides who can actually buy the capacity. Nvidia keeps the installed base. New capacity decisions now include real alternatives with different cost curves.
The inflection point sits in commercialization. Proprietary accelerators built for internal workloads now compete for external inference revenue, with custom-silicon partners supplying the design capacity. Compute turns from a single-supplier dependency into a portfolio decision priced on cost per token, performance per watt, and supply resilience. The next phase measures integration across silicon, networks, compilers, and software stacks. Expect procurement to shift toward stack evaluations.
If you’re involved with AI Processors, or just interested, check out JPR’s web page, which publishes AIP stories almost every day.
Jon Peddie
President
Dr. Jon Peddie is a recognized pioneer in the graphics industry, president of Jon Peddie Research, and named one of the world’s most influential analysts. Jon is an ACM Distinguished Speaker and is an IEEE Distinguished Visitor, and named an IEEE Computer Society Distinguished Contributor and Charter member. He lectures at numerous conferences and universities on topics about graphics technology and the emerging trends in digital media technology. Former president of Siggraph Pioneers, he serves on the advisory boards of several conferences, organizations, and companies, and contributes articles to numerous publications. In 2015, he was given the Lifetime Achievement Award from the CAD Society. Jon has published hundreds of papers, co-authored a half-dozen technical books, and written 10 technical books (including “Augmented Reality: Where We All Will Live,” “Ray Tracing: A Tool for All,” and his latest, the three-book series “The History of the GPU” as well as two fictional titles.
Guest Column: The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the official policy or position of Autonomy Magazine.
Leave a Reply