AI is collapsing the boundaries inside modern data centres

What is changing inside the data centre?

AI hardware coverage often focuses on GPUs. Data Center Knowledge's most-read hardware stories from June 2026 show a wider picture, with networking, memory, CPUs, orchestration software and chip design getting attention alongside GPUs ^[3].

The same publication describes the trend as increasingly systems-focused, and says the race is no longer only about GPUs ^[3].

Why has memory become the limit?

Epoch AI reports that AI inference is frequently bottlenecked on memory bandwidth, not compute ^[1]. It also reports that the total memory bandwidth of AI chips shipped since 2022 has reached 70 million terabytes per second, growing 4.1x per year ^[1].

Epoch calls aggregate HBM bandwidth a rough proxy for the world's total capacity to serve AI models ^[1]. Because that figure is an aggregate across chips shipped, it answers a different question from the specification of any single chip.

Can memory sit outside the GPU?

A 2025 arXiv paper on 1-million-token inference cites earlier work by Tang et al. showing that offloading the KV-cache to CXL memory can reduce GPU memory usage by up to 87% while meeting latency requirements ^[2].

The paper argues that CXL provides a practical way to extend memory capacity without sacrificing performance ^[2]. Its own contribution is a processing-near-memory design for this kind of CXL-attached memory ^[2].

Why does networking matter as much as the chip?

Data Center Knowledge says networking, memory, orchestration software and end-to-end efficiency are emerging as the decisive technologies shaping the next generation of AI data centres ^[3]. That is the publication's judgment, not a measurement, and the sources here do not quantify network cost.

What does this article leave out?

This article does not cover what AI services cost to buy or run, including token prices. Those figures need their own sources. The sources used here report research and hardware news, not measured deployments at scale.

What should teams watch next?

Check whether each vendor's memory figure covers a single chip or the whole system. Epoch's total is an aggregate across chips shipped ^[1], so a per-chip specification and an aggregate figure answer different questions.