Computex 2026: Seeing the Physical Layer of the AI Factory

The data center is being rebuilt, and NVIDIA gave the rebuild a name: the AI factory. I think that name is right, and it is not marketing. A normal data center stores and serves information. An AI factory does something different. Power goes in, finished intelligence comes out: an answer, a passed test, a merged change. The unit of production is the token, and the new bottleneck is not how many GPUs you own. It is how much useful work you get per watt of power and per dollar of hardware.

Here is my read on why Computex 2026 mattered. The first wave of this buildout was about raw compute: more GPUs, bigger racks, more power. This wave is about memory and movement. The moment you run real AI agents, the same context gets used over and over, and the factory's real job becomes not redoing work it already did. That is a memory problem, not a compute problem. It is why the quiet story at Computex was KV cache offload becoming real hardware.

KV cache is the part of the model's memory you can reuse, and it is the heart of the whole thing, so this post is half explanation and half opinion about where it goes. NVIDIA already named the software layer for it, Dynamo, and the whole factory is wired together by interconnect: NVLink inside the rack, PCIe and CXL between the chips and the memory. That wiring is the nervous system. When it is slow, the GPUs sit and wait, and the factory burns power producing nothing.

The metric that ties it together is what we call AI Output Yield: useful work out per dollar and per watt in. Cost per finished task, not tokens. The rest of this post is the hardware that showed up at Computex, how the memory actually moves, the companies building each piece, and one coding agent walked all the way through.

Penguin Solutions: the warm memory tier

Penguin's MemoryAI server is the first production CXL-based KV cache server. A 4U box: dual EPYC CPUs, 3 TB of DDR5, plus up to eight 1 TB CXL cards, for up to 11 TB of memory sitting next to the GPUs. It holds the repeated KV that does not need to be in HBM this instant.

The point is capacity next to compute. A single B200 holds about 180 GB of HBM, enough for roughly one long agent's KV after weights and decode. Penguin's box holds about 110 of those histories warm at once. Penguin says the CXL tier is 10x faster than NVMe and works with NVIDIA Dynamo. Treat the numbers as vendor claims until a workload proves them, but the direction is real: the warm tier is now a product, not a paper.

The warm tier as a product: Penguin's MemoryAI server

A 4U box with up to 11 TB of DDR5 and CXL memory next to the GPUs, holding repeated KV that does not need HBM this instant.

Penguin's MemoryAI KV cache server, the first production CXL-based KV cache box: dual EPYC CPUs, up to eight 1 TB CXL cards, up to 11 TB of memory next to the GPUs. Image: Penguin Solutions.

Astera Labs: the fabric that moves it

Penguin holds the memory. Astera makes you able to reach it. Leo is the CXL memory controller. Scorpio is the PCIe 6 fabric switch, up to 320 lanes. COSMOS is the telemetry that tells you the path is healthy. Without this layer the warm tier is just memory you cannot reach fast enough to beat rebuilding the KV.

This is not a side detail. On its last earnings call Astera named KV cache offload as a growing use case, with a design win on a customized Leo controller, and said PCIe 6 was already more than a third of revenue. The fabric is where the factory either behaves like one system or like a pile of expensive parts.

The fabric and the controller that move the warm tier

Scorpio is the PCIe 6 switch that wires GPUs to the memory tier; Leo is the CXL controller that makes an attached box behave like RAM.

Left: the Scorpio Smart Fabric Switch X-Series, the PCIe 6 layer wiring GPUs to the memory tier. Right: the Leo CXL Smart Memory Controller and the Aurora A1000 add-in card it powers, shown at GTC. Images: Astera Labs (Scorpio); ServeTheHome (Leo).

How CXL rides PCIe

This is the part most recaps skip, and it is what makes offload possible at all.

PCIe is the road. It is the lanes that connect CPUs, GPUs, NICs, and SSDs, and it moves data as I/O packets, the same way it moves a file off a disk. PCIe 6 doubles the per-lane speed again, so more KV traffic can take it.

CXL rides the same lanes and changes what they mean. The piece that matters is CXL.mem: it lets the GPU read and write the attached memory with plain loads and stores, like RAM, not block reads like a disk. So an 11 TB box on a PCIe slot looks like memory, not storage. KV cache is memory, so that is exactly what it needs to come back fast enough to beat recomputing it.

How the pieces physically connect

Walk one KV block from the GPU out to the warm tier and back. That path is the whole product, and every company above owns a step of it.

1. Prefill writes the KV into HBM on the GPU. This is L0, soldered to the die, the fastest memory and the most expensive.
2. The block cools. The KV block manager decides it does not need to sit in HBM right now and copies it out over the GPU's PCIe lanes.
3. A retimer cleans the signal. PCIe 6 runs at 64 GT/s per lane, and that signal degrades across the board traces and cables, so a retimer (Parade, Phison) regenerates it. Without it the link drops speed or fails.
4. A fabric switch fans it out. A CPU only has so many PCIe lanes, so an Astera Scorpio switch (up to 320 lanes) lets many GPUs, NICs, and memory cards share one fabric.
5. It reaches the CXL card. A CXL controller (Leo, Wolley) drives DDR5 RDIMMs (Innodisk). At boot, CXL.io enumerates the card like any PCIe device. At runtime, CXL.mem carries the actual loads and stores. The card is a CXL Type 3 device, pure memory expansion, byte-addressable and coherent with the host, which is why it acts like RAM.
6. The controller writes the block into DDR5. That is L2, the warm tier.
7. The next turn needs it. The GPU issues a memory load, CXL.mem fetches the block back over the same path, the retimer keeps it clean, COSMOS measures the latency, and the GPU reuses it instead of recomputing.

Why it is warm and not hot is just physics. A B200's HBM moves about 8 TB/s. An Innodisk CXL card adds about 32 GB/s over PCIe Gen5 x8, orders of magnitude less, and a memory access over CXL takes a few hundred nanoseconds against HBM's tens. So CXL is the wrong place for the token being written this instant, and the right place for the repeated context the agent will need again in a few turns. That gap is the entire reason the tiers exist.

The memory hierarchy: hot, warm, cold

KV does not all belong in one place. The job is to keep each piece of the agent's memory in the cheapest tier that is still fast enough. There is no industry-standard numbering for this, so the table below is our operator map, L0 at the top, L4 at the bottom. Faster and scarcer up top, larger and cheaper as you go down. To make it real, the second column is exactly what each tier holds for our coding agent.

The B200 gives you about 180 GB of L0 HBM per GPU; a GB200 NVL72 rack shares about 13.4 TB of it. Either way the rule is the same: hot KV stays in HBM, warm KV moves to DRAM or CXL if transfer beats recompute, cold KV goes to SSD or remote only if the latency is tolerable. CXL is the new line, L2, because it is the first tier bigger than HBM that still acts like memory instead of storage. That is the tier that did not exist in a normal data center, and it is what makes the AI factory a different machine.

How KV cache offload actually works

Offload is one decision repeated on every block: keep it hot, move it down a tier, or drop it and recompute later. The loop:

prefill -> KV block manager -> place hot or warm -> register location -> route next request -> fetch vs recompute -> receipt

In plain terms: the GPU builds KV during prefill. A KV block manager decides which blocks stay in HBM and which move to L1 or L2. It records where each block went, so a later request can find it. The scheduler routes the next request to a worker or tier that already holds the block. If the block can come back faster than rebuilding it, fetch it. If not, recompute. Then log the receipt: hit rate, bytes moved, transfer time, TTFT, and whether reuse actually beat recompute.

The hardware decides where KV can live. The software decides what goes where, and that software is LMCache.

LMCache is the open-source layer that makes KV reusable and movable. It stores and moves KV across GPU, CPU, disk, and remote backends, and exposes it so you can see and route it instead of leaving it buried inside one serving engine. It plugs into vLLM and NVIDIA Dynamo. Its CacheBlend work on an OpenClaw-style agent is where the 98% hit rate and 42% lower latency came from. Think of it this way: LMCache is the brain that decides a block should move to the L2 Penguin tier, and CXL plus the fabric is the road it travels on.

A couple of pieces above LMCache make the reuse possible, and I went deep on all of it in an earlier briefing, KV Cache Is Becoming the Memory Hierarchy of Inference. SGLang turns repeated prompt structure into runtime structure: it sees that two requests share a system prompt and reuses the KV instead of rebuilding it. Dynamo splits prefill and decode onto different workers, which makes the transfer path, not just the math, the thing you tune. And for visual agents the vision encoder's output is the new KV: a screenshot encoded once should not be re-encoded every turn either. The earlier post walks the full lifecycle of a single bad turn, Mooncake, CacheBlend, the encoder reuse, and what we are trying to prove with InferGuard.

The companies that build this

Penguin and Astera get the headlines, but moving a KV block out of HBM and back is a physical journey, and the interconnect that carries it, PCIe and CXL, is itself a stack of separate parts, each from a different company. Follow the signal from the GPU out to the warm tier and back, one component at a time.

1. It starts on copper, and at Gen6 copper barely cooperates. KV leaves the GPU on PCIe lanes. PCIe 6 runs at 64 GT/s and switches to PAM4, four voltage levels instead of two, two bits per symbol. That doubles the data rate without doubling the clock, but it shrinks the eye openings to about a third the height, so the signal is far more fragile and PCIe 6 has to add forward error correction just to stay reliable. The loss budget got tighter too, 32 dB end to end at Gen6 versus 36 dB at Gen5. Every inch of board trace, every via, every connector burns dB, and the highest-frequency content dies first. Passive copper carries Gen5 maybe three meters; at Gen6 it is a few inches on a board before the link will not train. Everything after this step exists to buy that reach back.

2. The cheap fix is a redriver. When the signal is degraded but not dead, you reshape it with a redriver, a pure analog part. It applies continuous-time linear equalization, boosting the high frequencies the channel ate, and re-drives the waveform. It does not recover the clock and does not retransmit clean bits, so it is lower power, lower latency, and cheaper than the alternative, but it can only stretch the loss budget, not reset it. Phison's PS7161 and PS7162 are Gen6 linear redrivers, one of which Phison demoed inside a Molex active copper cable. Diodes' PI3EQX64904 was billed as the first redriver to hit PCIe 6.0's 64 GT/s. Parade's PS8592 is an 8-channel Gen6 redriver with up to 18 dB of boost, a vendor number. You reach for a redriver when a retimer is overkill.

3. The real fix is a retimer. When the eye is closed and a redriver cannot save it, you use a retimer, a protocol-aware DSP. It recovers the embedded clock, re-equalizes, regenerates clean digital bits, and retransmits a brand-new full-amplitude signal, so the next segment starts with a fresh budget. That roughly triples the reach, at the cost of more power and a little latency, and the retimer joins PCIe link training as a real participant. This is the most contested chip in the interconnect. Astera Labs is the merchant leader: its Aries 6 compensates up to 36 dB of loss at around 11 W for a 16-lane PCIe 6.x / CXL 3.x part. Microchip launched its XpressConnect PCIe 6.0 / CXL 3.1 retimers on June 2 at the show, claiming under 12 ns pin-to-pin. Montage's M88RT61632 claims a 43 dB link budget, Credo's Toucan claims 40 dB at sub-7 ns and just passed PCI-SIG compliance, Parade's PS8936 covers the Gen5 / CXL 2.0 generation, and Phison's PS7261 adds live PAM4 eye and LTSSM telemetry. The dB and latency figures are all vendor specs.

4. To leave the board, the retimer moves into the cable. A board trace gets you across a server; reaching the next rack needs a cable, and at 64 GT/s passive copper cannot go far. The answer is an active electrical cable, an AEC, which buries retimer silicon in the connector shells at each end so thin twinax copper runs several meters at full speed, with optics-class reach at a fraction of the power and cost. Credo created this category and dominates it with its HiWire line, including a ZeroFlap feature that keeps links from dropping during long training runs. Astera Labs sells Aries Smart Cable Modules that push 64 GT/s up to about 7 meters, and Montage shipped a PCIe 6.x / CXL 3.x AEC built on its own retimer. Molex, Amphenol, and TE Connectivity assemble the cables. This is how a KV pool in one rack serves GPUs in another.

5. One GPU has to reach many memory devices, so next is the switch. A GPU has a fixed lane budget. To fan out to a shelf of CXL memory, or to let several GPUs share one pool, you put a PCIe / CXL switch in the middle, a non-blocking crossbar that routes packets at line rate and adds CXL fabric features like pooling and memory-semantic access. Broadcom is the incumbent: its Atlas line and the 144-lane PEX90144 PCIe 6 / CXL 3.1 switch sit in most sockets. Astera Labs' Scorpio X-Series pushes 320 PCIe 6 lanes through one switch to wire up to 80 accelerators, with a Hypercast engine for collective-op replication and in-network compute that offloads AllReduce. The independent specialist XConn built the first hybrid CXL-plus-PCIe switch, Apollo; Marvell bought XConn for $540M in January and now ships both that line and its own 260-lane PCIe 6 switch. The switch is what turns a point-to-point link into a fabric.

6. All of it has to physically mate, which is the connector layer. The signal still has to cross board-to-board, board-to-cable, and into the memory module's slot while holding 64 GT/s PAM4 integrity. That is MCIO for cabled PCIe, CEM card-edge slots for add-in cards, and EDSFF, the E3.S and E1.S form factors, for the CXL modules themselves. Amphenol with its Cool Express Link, Molex with NextStream rated to 64 Gbps PAM4, and TE Connectivity build these. Unglamorous, but a bad connector throws away the dB the retimer just bought.

7. The attached box has to behave like RAM, not a disk, so now the CXL memory controller. A box of DDR5 on a PCIe slot is just storage; the GPU can only move data to it in blocks. The piece that changes that is a CXL Type 3 memory controller: it speaks CXL.mem, so the host reads and writes the attached DRAM with plain loads and stores at cache-line granularity, the same way it touches local RAM. Astera Labs' Leo is the one already shipping in production KV boxes, CXL 2.0, up to 2 TB, doing expansion plus pooling and sharing, with COSMOS for telemetry. Montage's MXC has been the merchant default since 2022 and is the only CXL 3.1 controller in silicon so far, running DDR5 to 8000 MT/s. Microchip's SMC 2000 sits at CXL 2.0 and DDR5-4800. Marvell's Structera goes further: the X expander claims over 6 TB per controller, and the A variant puts 16 Arm cores right in the memory path so you can run light work on the KV where it sits instead of dragging it back to the GPU. Wolley's twist is NVMe-over-CXL, and InnoGrit's Cascade is a CXL 3.1 controller that fronts XL-FLASH as storage-class memory for long context. This controller is the whole trick: it is what lets KV live off-HBM and still come back as a memory load.

8. The controller needs real capacity behind it, so then the memory module. HBM is too expensive to hold idle KV, so you want cheap DDR5 in bulk. A CXL memory module is that DDR5, on a card or an E3.S sled behind the controller, running about 36 GB/s per channel, roughly one DDR5 channel. You are buying capacity, not HBM speed. Samsung's CMM-D ships at 128 and 256 GB, and its CMM-B box pools up to 24 TB across a rack. SK hynix ships CMM-DDR5 with Niagara for pooling and HMSDK software to tier hot pages back to DRAM. Micron's CZ120 is in volume production and explicitly names transformer KV cache as a use case. The card stacked eight-high inside Penguin's box is SMART Modular's CXA-8F2W, eight DDR5 slots behind two controllers, and Innodisk builds a smaller 256 GB AIC for tighter boxes. Everything shipping is still CXL 2.0 while the 3.x parts sample.

9. Someone packages it into a server, and there are two builds. The CXL-DRAM build puts the memory on the bus: Penguin Solutions' MemoryAI is the first production box, Astera Leo controllers and SMART Modular cards, up to 11 TB next to the GPUs, compatible with NVIDIA Dynamo. The other build puts the KV on the network: NVIDIA's CMX reference, where a BlueField-4 DPU moves KV to NVMe flash over Spectrum-X Ethernet. ASUS (the UF920-E3-RS24), AIC, and Supermicro build that box, and Graid turns its SupremeRAID GPU-RAID into a KV storage pool aligned to the same reference. One build keeps the KV on the memory bus, the other on the fabric, same goal.

10. Finally, software decides which block goes where and moves it. The hardware says where KV can live; something still has to choose what stays hot, what spills, and route the next request to wherever its KV already sits. MemVerge's Memory Machine tiers and pools CXL memory, and with XConn's switch it ran a CXL pool offloading KV and prefill/decode for Dynamo. VAST Data is the flash store behind NVIDIA CMX, reusing context through Dynamo. And the open layer ties it together: LMCache moves KV across GPU, CPU, disk, and remote backends for vLLM, SGLang, and Dynamo, and Dynamo's own KV Block Manager is the connector interface every one of these hardware tiers plugs into. Hardware decides where KV can go; this layer decides where it does.

Who builds the interconnect, in depth. Four companies define the silicon. Astera Labs (Nasdaq: ALAB) created the merchant retimer market: founded in 2017, it went public in March 2024 near a $5.5B valuation and now carries a market cap above $50B on roughly $900M of 2025 revenue, and it is the only vendor that spans almost the whole path, Aries retimers, Aries Smart Cable Modules, the Scorpio switch, the Leo CXL controller, and the COSMOS software that manages them. Broadcom (Nasdaq: AVGO), the $64B-a-year giant, owns the incumbent switch franchise it inherited when Avago bought PLX in 2014; its 144-lane Atlas 3 sits in most server designs. Marvell (Nasdaq: MRVL), an $8B data-infrastructure chipmaker, is assembling an end-to-end rival stack, Alaska P retimers, Structera CXL controllers, and a 260-lane switch, and it bought the hybrid-switch startup XConn for $540M in early 2026 to get there. Microchip (Nasdaq: MCHP) became a full-stack connectivity supplier through its 2018 Microsemi acquisition, and is the only other vendor besides Astera shipping a retimer, a switch, and a CXL controller all three.

Then the specialists. Montage Technology (Shanghai STAR: 688008) is one of only three makers of DDR5 register clock drivers and the company that shipped the first CXL memory controller in 2022; its MXC is still the only CXL 3.1 controller in silicon. Parade Technologies (TPEx: 4966), a roughly $530M interface-chip company best known for the DisplayPort and HDMI controllers in laptops, applies that signaling expertise to the PS8936 retimer and PS8592 redriver. Phison (TPEx: 8299), the largest independent NAND controller maker, extended into the PS7261 retimer and PS7161 redriver. Diodes (Nasdaq: DIOD), a $1.5B broad-line analog maker, built what it calls the first PCIe 6.0 redriver. Credo (Nasdaq: CRDO) created the active electrical cable category, dominates it with HiWire (Amazon is a major customer), and just put its Toucan retimer through PCI-SIG compliance. Two startups round out the controller layer: Wolley, pioneering NVMe-over-CXL, and InnoGrit, founded by an ex-Marvell CTO, whose Cascade controller fronts XL-FLASH over CXL 3.1.

The DRAM comes from the big three. Samsung, the world's largest chipmaker and the No. 1 DRAM supplier, ships the CMM-D module and the 24 TB CMM-B box. SK hynix, the HBM leader, ships CMM-DDR5 plus its Niagara pooling and HMSDK tiering software. Micron, the only US-based major memory maker, ships the CZ120 and explicitly names KV cache as a use case. SMART Modular, now part of Penguin Solutions, builds the CXA-8F2W card that stacks eight-high inside Penguin's box, and Innodisk, an industrial-memory specialist, builds the smaller 256 GB card for edge boxes.

Someone turns the parts into a server. Penguin Solutions (Nasdaq: PENG), the AI-infrastructure company formerly known as SMART Global Holdings, ships the first production CXL KV cache box. On the NVMe-over-Ethernet road, ASUS, AIC, and Supermicro each build NVIDIA CMX storage servers around BlueField-4 and the Vera CPU, and Graid, the GPU-RAID startup, turned its SupremeRAID into a KV cache storage pool. MemVerge, a Big Memory software startup, pools CXL memory with its Memory Machine and feeds it to Dynamo as a KV store; VAST Data, valued at $30B in its April 2026 round, runs the flash store behind NVIDIA CMX. The physical mating comes from the connector giants: Amphenol (MCIO and Cool Express Link), Molex (a Koch Industries company, the NextStream system rated to 64 Gbps PAM4), and TE Connectivity (MCIO, SlimSAS, and M-XIO assemblies).

The same path as a parts list, in order. The dB, latency, and bandwidth numbers are vendor specs, not independent benchmarks.

The pattern is the point. The whole path already exists as shipping parts, the lanes, the redriver, the retimer, the cable, the switch, the connector, the controller, the memory card, the box, and the software, each from a company that does only its step well. Every shipping CXL part is still CXL 2.0 while the 3.x silicon samples, and the vendor numbers need a real workload before they go in a board deck. But the direction is unambiguous: at Computex 2026 the warm tier stopped being a research idea and became a path you can buy end to end.

One example: a coding agent, step by step

Here is the whole thing on one workload. You are refactoring a Go service, about 40 files: add API-key auth, keep the tests green. You drive a Hermes-style agent (same shape as OpenClaw or Claude Code) running Kimi K2.6, a 256K-context model, on a GB200. Its tools: read_file, grep, edit_file, run_tests. Watch the prompt grow. If you want to run this loop yourself on a laptop first, we wrote a hands-on version: set up a local OpenClaw or Hermes KV-cache loop on a Mac with MLX.

Compaction is the moment reuse lives or dies. A cache read is about 92% cheaper than a cold read, roughly $0.019 versus $0.23 on a 60K-token prefix. Once compaction invalidates the cache, every turn after it pays the cold price until the prefix warms back up.

So the felt experience splits. Without offload, the head KV often got evicted under HBM pressure before compaction even hit, so turns were already slowing. Then compaction lands, the agent pauses, forgets, and turn 47 feels like starting over. With offload, the head is a memory load away, so turn 50 stays as fast as turn 1. One agent run measured a 98% hit rate and 42% lower latency.

The CFO math, in real dollars

This is where the warm tier earns its place, so here are real market numbers, not invented ones. Two things move the decision: the price of memory, and the price of the GPUs you would otherwise buy to hold it. Memory and GPU prices move week to week, so every figure here is dated, sourced, and split into market, derived, and modeled in the provenance table at the end of this section. All of it was indexed on 06/08/2026. Do not quote these numbers without re-checking the date.

Memory price. HBM is the most expensive memory in the building. HBM3E runs about $13 to $20 per GB; server DDR5 runs about $3 to $5 per GB, so HBM is roughly four to five times the price per gigabyte, and that gap is only expected to narrow through 2026. HBM is also capacity-locked: you cannot buy it on its own. It comes soldered to a $30,000 to $50,000 GPU, about 180 GB per B200.

The capex lever. Say you want to keep 110 agent histories warm, the 11 TB from earlier. Here is what that costs two ways.

That is the headline for a CFO. Holding that context in HBM means spending two to three million dollars on GPUs just to act as memory. A CXL box holds it for a fraction and frees the GPUs to compute. Treat the box figure as indicative, not a quote, but the order of magnitude is the point.

The recurring lever. On one workload, 120,000 repeated prefix tokens, 1 million requests a month, 70% hitting warm cache, at a modeled $0.30 per million repeated tokens, is about $25,200 a month of prefill you stop paying. On compute, a B200 rents for about $3 to $10 per GPU-hour. If offload frees enough HBM to double concurrency, you serve the same users on half the GPUs: on a 16-GPU B200 deployment at $5 an hour, that is about $29,000 a month, or you defer the next GB200 NVL72 rack, which rents for roughly $750 to $1,900 an hour and costs around $3M to buy.

Same power bill, same GPUs. You stopped paying twice for the same context, and you stopped buying GPUs to hold memory.

Three kinds of number sit in that table, and they are not equal. Market figures are third-party prices with a date, and they move. Derived figures are our arithmetic on top of those prices. Modeled figures are a workload assumption, useful for shape, not a promise. Re-index before you put any of these in a board deck.

One caveat, and it is real. The CXL tier costs money too, and it only wins if fetching warm KV back beats recomputing it, at p95 and p99, on a workload that repeats context. A chat workload that does not reuse much gets none of this. So you measure first.

The compute underneath: Blackwell now, Vera Rubin next

The warm tier sits next to GPUs, and which GPU you have decides how much hot HBM you get before you have to offload. Two generations matter right now: Blackwell, shipping today, and Vera Rubin, which NVIDIA just put into production. Every number in this section is sourced and dated in the table at the end of it, indexed 06/08/2026.

Blackwell: B200, B300, GB200, GB300

The clean way to read the lineup: B is the chip, GB is the rack, and the jump from 200 to 300 is mostly more memory. That matters here because more HBM means more KV stays hot before it has to spill to the warm tier.

For this post the only difference that matters is memory. B300 and GB300 give you about 1.5x the HBM of the 200-series, so more agent state stays hot before it spills. That does not remove the need to offload, it just raises the line where offload starts. GB200 is the current workhorse. GB300 is for larger context and bigger batches where the extra HBM pays for itself. Both still hit the wall on long agents, which is the whole reason the CXL warm tier exists.

Vera Rubin: the next platform, and what came out at Computex

Vera Rubin is the next generation, and it is the headline hardware news of this cycle. NVIDIA announced it at CES 2026, and Jensen Huang confirmed full production at GTC Taipei on June 1, 2026, naming Samsung, SK hynix, and Micron as the HBM4 suppliers.

The chips. The Rubin GPU is 336 billion transistors across two dies, with up to 50 PFLOPS of NVFP4 inference, which NVIDIA frames as 5x Blackwell. It moves to HBM4: up to 288 GB per GPU and up to 22 TB/s of bandwidth. The Vera CPU is 88 custom Arm cores with up to 1.5 TB of LPDDR5X. The rack, Vera Rubin NVL72, is 72 Rubin GPUs and 36 Vera CPUs over NVLink 6, with 20.7 TB of HBM4, 54 TB of LPDDR5X, and 1.6 PB/s of HBM bandwidth.

HBM4 is the real memory change. It doubles the interface from 1,024 to 2,048 bits and the channels from 16 to 32, so each stack moves at least 2 TB/s. More bandwidth feeds the hot tier harder, but capacity per GPU is still finite at 288 GB, so the warm-tier story does not change, it just moves up a generation.

The claim a CFO will ask about: NVIDIA says Vera Rubin delivers up to 5x inference performance and 10x lower cost per token than Blackwell, especially for MoE inference. That is a vendor claim dated CES 2026, not an independent benchmark, so treat it as direction, not a receipt. Price is around $5M to $7M per VR NVL72, with reports up to $8.8M, roughly $1M of which is NAND storage. At GTC Taipei on June 1 NVIDIA reiterated full production and said the first complete systems ship in the fall, with broad availability in 2H 2026, and a supply chain it calls twice the size of Grace Blackwell.

Vera Rubin NVL72, the next platform

72 Rubin GPUs and 36 Vera CPUs over NVLink 6, with 20.7 TB of HBM4. The hot tier gets bigger and faster; the warm-tier story just moves up a generation.

NVIDIA's Vera Rubin NVL72, confirmed in full production at GTC Taipei on June 1, 2026. Image: NVIDIA.

The Vera CPU: the warm tier moves onto a coherent link

The part that matters most for KV cache is not the Rubin GPU, it is the CPU sitting next to it. Vera is NVIDIA's first fully custom data-center CPU core, 88 Olympus cores and 176 threads, Arm v9.2 compatible. It is the successor to Grace in the line that matters here, and it carries up to 1.5 TB of LPDDR5X per socket at up to 1.2 TB/s. Each Vera pairs with two Rubin GPUs, so a Vera Rubin NVL72 rack holds 54 TB of LPDDR5X sitting next to the HBM4.

What it takes to make that host memory useful for KV is the link. Vera connects to Rubin over NVLink-C2C at 1.8 TB/s of coherent bandwidth, which NVIDIA puts at 7x PCIe 6 and exactly double the 900 GB/s Grace-to-Blackwell link in GB200. Coherent is the word that does the work. The GPU and the CPU share one address space, so the LPDDR5X is not a disk you copy to, it is memory the GPU can load and store. NVIDIA says it plainly: treat LPDDR5X and HBM4 as a single coherent pool, and the named use case is KV cache offload.

So here is how it ties to everything above. HBM4 is the hot tier, still finite at 288 GB per GPU. The first place cooled KV goes is the Vera LPDDR5X, over the 1.8 TB/s coherent link, fast enough to beat recompute and far faster than the PCIe path the board-level boxes use. Only past that do you reach the Penguin CXL box or NVIDIA's own CMX flash tier. In software this is the same warm tier LMCache fills with its LOCAL_CPU backend and Dynamo fills with KVBM CPU offload (DYN_KVBM_CPU_CACHE_GB), except now the wire underneath it is a coherent 1.8 TB/s C2C link, not PCIe. The warm tier did not just get bigger with Vera Rubin. It moved onto a memory link.

Same caution as the rest of this section. Every Vera number here is from NVIDIA's CES 2026 launch and is vendor claimed, not independently benchmarked. The 1.5 TB per CPU is roughly 3x Grace's 480 GB and the 1.8 TB/s link is exactly 2x, so the direction is real, but treat the maxima as direction, not a receipt, until the silicon is tested.

For an engineer, that warm tier is a couple of flags.

RTX Spark and DGX Spark: the coherent-memory idea moves onto the desk

The same memory story is now showing up outside the data center.

NVIDIA's RTX Spark is not just another AI PC label. It is a new class of machine between a workstation and a small AI server: a Windows PC built around a Grace Blackwell-style superchip, a Blackwell RTX GPU, a 20-core Arm CPU, CUDA, and up to 128 GB of unified memory. Microsoft, Dell, HP, Lenovo, ASUS, MSI, and others are building around it. The market may or may not be ready for it yet, but the architecture matters either way.

The important part is not the branding. It is the memory model.

On a normal discrete GPU workstation, model weights live in VRAM. The KV cache fights those weights for the same fixed pool. When the working set spills, it crosses PCIe into host memory. That copy boundary is exactly where latency and operational weirdness start.

On DGX Spark and RTX Spark, the CPU and GPU share one coherent memory pool. The weights, KV cache, OS, runtime, and application all draw from the same 128 GB LPDDR5X pool. That does not make it faster than HBM or high-end GDDR. It makes it simpler and larger from the model's point of view. The machine can fit local inference workloads that a 96 GB discrete card may not fit cleanly, especially quantized MoE models, long-context coding agents, and small-batch developer loops.

That is the real connection to this post.

Vera Rubin does the same thing at rack scale. Rubin HBM4 is the hot tier. Vera LPDDR5X is the first warm tier, connected over a coherent NVLink-C2C path. DGX Spark and RTX Spark are the desk-side version of the same idea: stop treating host memory as a slow copy destination and start treating it as part of the inference memory hierarchy.

The catch is bandwidth.

DGX Spark's unified memory is about 273 GB/s. That is useful capacity, not HBM-class speed. A data-center B200 moves HBM at terabytes per second. An RTX PRO 6000 Blackwell has 96 GB of GDDR7 at roughly 1.6 TB/s, faster memory but smaller and not unified. So Spark is not the right mental model for high-concurrency serving or maximum decode throughput. It is the right model for local agents, model prototyping, small-batch inference, edge workflows, demos, and education.

That distinction matters for Touchdown Labs.

Spark gives teams a physical way to feel the memory hierarchy before they touch a rack. You can run a local coding agent, watch the context grow, watch KV cache utilization move, tune vLLM flags like --gpu-memory-utilization, --max-model-len, and --max-num-seqs, and see why long-context inference is not just "more tokens." It is a memory placement problem.

In the data center, the question is where this KV block should live: HBM, CPU memory, CXL memory, NVMe, or remote storage. On Spark, the same question becomes smaller and more teachable: how much of this 128 GB unified pool should hold weights, how much should hold KV, how much headroom does the OS and runtime need, and when does longer context kill interactivity.

That is why Spark belongs in this article. It is not a replacement for Penguin, Astera, BlueField, CMX, or Vera Rubin. It is the local developer version of the same architecture shift.

From desk to rack, the answer to inference pressure is becoming the same. Keep hot KV close. Move warm KV without crossing unnecessary boundaries. Measure whether reuse actually beats recompute. Optimize for cost per finished task, not just tokens per second.

Spark is the desk-side version of the AI factory memory hierarchy: not fastest memory, but fewer boundaries between weights, KV, runtime, and the developer loop.

BlueField-4: the context-memory mover

This is the piece that ties Vera Rubin back to everything above. BlueField-4 is NVIDIA's new DPU: 800 Gb/s of throughput, double BlueField-3, with 64 Arm cores, 128 GB of LPDDR5, a 512 GB onboard SSD, and a PCIe Gen6 host link. NVIDIA calls it "the operating system of the AI factory," and its actual job is moving data between storage and compute at 800 Gb/s. That is exactly the KV and context-memory path this whole post is about. It ships as a card or built into the Vera Rubin NVL144 rack.

So here is the new capability in plain terms. Vera Rubin raises the compute and the HBM bandwidth, and BlueField-4 builds the context-memory movement into the rack itself. The KV offload we described with Penguin and Astera at the board level is being pulled up into NVIDIA's own platform at the rack level. Same idea, bigger and more integrated: keep the repeated context close, move it fast, do not rebuild it.

BlueField-4: the context-memory mover

800 Gb/s of throughput, 64 Arm cores, a 512 GB onboard SSD, and a PCIe Gen6 host link. Its job is moving data between storage and compute, which is the KV path.

The BlueField-4 DPU, which NVIDIA calls "the operating system of the AI factory." Image: NVIDIA.

CMX: NVIDIA's own KV cache tier, built into the rack

The storage side of Vera Rubin is where this gets concrete for KV. CMX (Context Memory Storage) is a BlueField-4-powered platform built specifically to hold and share KV cache for long-context and agentic inference. NVIDIA calls it a new G3.5 tier, sitting between local node storage and shared network storage. BlueField-4 manages the NVMe flash, runs the storage services, and does line-rate encryption and integrity on the KV cache at the full 800 Gb/s without spending Grace CPU cycles. A framework called DOCA Memos exposes key-value APIs and turns Ethernet-attached flash into a pod-level cache tier, treating context cache as a first-class resource. NVIDIA cites up to 200 GB/s of storage bandwidth and up to 5x higher throughput and 5x better power efficiency versus general-purpose storage on this path. Spectrum-X Ethernet is the RDMA fabric linking the CMX flash enclosures to the GPU nodes.

Read that against the tiers from earlier. CMX is NVIDIA building the L2/L3 context tier into the rack, the same job the Penguin CXL box does at the board level, but pod-wide, over Ethernet, with the DPU handling the movement, encryption, and sharing. Different road, same destination: keep the repeated KV close, move it fast, do not rebuild it.

CMX: NVIDIA's KV cache tier, built into the rack

A BlueField-4-powered platform that holds and shares KV cache pod-wide over Spectrum-X Ethernet, the L2/L3 context tier built into the rack.

The CMX Context Memory Storage Platform: a BlueField-4 storage tray sliding into the rack, holding KV cache as a first-class, pod-level resource. Image: NVIDIA.

The same KV block, three builds: walk it yourself

Every section above is one piece of a single path. Here it is as one moving part, so a CEO, a CFO, and an engineer can each read the same picture. Pick a build, pick a lens, and step a single KV block from HBM out to the warm tier and back. The three builds are the real choices, and they are not the same hardware generation: plain host memory and disk, the Penguin and Astera CXL box at the board level on Blackwell (B200, GB200) today, and NVIDIA's BlueField-4 and CMX at the rack level on Vera Rubin next. Penguin is the Blackwell-era answer you can buy now; BlueField-4 and CMX ship with Vera Rubin only. Across all three, and on either GPU generation, the software is the same stack: vLLM or SGLang serve the model, LMCache moves and reuses the KV, and Dynamo routes it through its KV Block Manager. The hardware below is the only thing that changes. The widget highlights which layer is doing the work at each step.

Without either, build 0. The KV that does not fit in HBM goes to host DRAM, then spills to local NVMe, and the host CPU does the copying over PCIe. That works for a handful of agents on one box. Past that, DRAM fills, the spill runs about 10x slower, and a cache hit that arrives from disk too late is just a slower miss. This is LMCache running its LOCAL_CPU and LOCAL_DISK backends, nothing exotic.

Add Penguin, build 1 (Blackwell, today). This is the answer you can buy now for B200 and GB200 systems. The 11 TB CXL box, reached through Astera's Scorpio switch and Leo controller, turns the warm tier into byte-addressable memory that behaves like RAM, on the PCIe and CXL bus right next to the Blackwell GPUs. About 110 agent histories stay warm at once, roughly 10x faster than NVMe. CacheBlend then reuses 85% of each 256-token chunk and recomputes 15%, and the 85% comes back from a big fast pool instead of slow disk. The catch: it is still bus-local. Sharing KV across many nodes is not its strength.

Add BlueField, build 2 (Vera Rubin only, next gen). This one is not a Blackwell upgrade. BlueField-4 and CMX ship with NVIDIA's Vera Rubin rack, so it is the next-generation, rack-level version of the same idea, not something you bolt onto a B200 box. The DPU moves the KV by RDMA at 800 Gb/s without spending host CPU, NVMe flash over Spectrum-X holds it pod-wide, and Dynamo routes the next request to wherever its KV already sits. You trade CXL's near-RAM latency for fleet-wide reach and CPU bypass. One build keeps the KV on the memory bus, the other on the fabric, same goal.

The hardware decides where KV can live. The software decides where it does. In all three builds the open layer is the same: LMCache moves KV across GPU, CPU, disk, and remote, and Dynamo's KV Block Manager is the connector every one of these tiers plugs into.

Back to the factory

Start where we began. NVIDIA reframed the data center as an AI factory, a machine that turns power into finished intelligence. Everything in this post is that thesis playing out one level down. The factory runs on repeated inference. The repeated part is KV cache. And the whole thing is held together by interconnect, NVLink inside the rack, PCIe and CXL out to the memory, the nervous system that decides whether the GPUs work or wait.

That is why Computex 2026 mattered. The physical layer of inference stopped being a diagram and became products you can buy. At the board level, today, on Blackwell: Penguin built the warm tier to hold the repeated context, Astera built the fabric to move it, and CXL is what lets an 11 TB box act like memory instead of storage. At the rack level, next, on Vera Rubin: the Vera CPU puts the warm tier on a coherent 1.8 TB/s link, and BlueField-4 and CMX move the context tier onto the fabric. And it scaled all the way down to a desk, where DGX Spark and RTX Spark run the same coherent-memory idea on one 128 GB pool. vLLM, SGLang, LMCache, and Dynamo are the software deciding what moves where.

That is the one pattern under all of it. From a box on your desk to a Vera Rubin rack, the answer to KV pressure is the same. Stop copying across boundaries, put the CPU and GPU on one coherent memory pool, keep the hot KV close, and move the warm KV without rebuilding what the model already knew. Hot, warm, cold, L0 to L4, is just the factory keeping each piece of memory in the cheapest place that is still fast enough.

The open job is proof. Vendor numbers are a start, not an answer. The receipt that matters is whether the warm tier actually lowers cost per finished task on your workload, on your GB200 or your Spark, with your agent. That is the part we work on at Touchdown Labs with InferGuard, our open-source profiler for exactly this lifecycle, and it is the through-line of our earlier deep dive, KV Cache Is Becoming the Memory Hierarchy of Inference.

Tokens per watt matters. AI Output Yield matters more.