Just realized that: after all, separated PCMem sticks connecting to GPU(&CPU) through motherboard is much better configuration than GPU dedicated PCMem. For example, 16 separated PCMem sticks(64GB per stick) can be installed on one motherboard to connect to one set of GPU/CPU or CGPU, which takes much less space, cost and power with higher performance.
So I bet that even for Nvidia it’s better to adopt totally new configuration for PCMem other than inheriting old platform, to make separated PCMem sticks and GPU.
Gemini agrees pretty much and makes a detailed analysis for this as below as I requested.
🏛️ Architectural Analysis: 16× Motherboard PCMem Sticks vs. 16× Discrete GPUs with Dedicated PCMem
When comparing a server populated with 16× 64GB PCMem sticks directly in standard motherboard memory slots against a server carrying 16 discrete GPU accelerator cards with dedicated PCMem on each card, the native motherboard architecture is vastly superior in silicon efficiency, power consumption, interconnect latency, physical form factor, and total system cost.
Below is the complete engineering breakdown, incorporating 100% full-power benchmarks at 200 MHz (0.56V on TSMC 2nm GAA + 1β/1γ planar LPDDR6):
codeCode
========================================================================================================================
METRIC 16× DISCRETE GPUS (WITH DEDICATED PCMEM) 1× MOTHERBOARD + 16× PCMEM STICKS (NATIVE)
========================================================================================================================
Total In-Situ Memory Array 1,024 GB (1 Terabyte across 16 GPU cards) 1,024 GB (1 Terabyte across 16 DIMM slots)
Total In-Situ MPUs 102,400,000 MPUs (102.4 Million Cores) 102,400,000 MPUs (102.4 Million Cores)
------------------------------------------------------------------------------------------------------------------------
Companion GPU / Host Silicon 16 Separate Discrete GPU Silicon Dies 1 Single Host CPU / Companion GPU (or CGPU)
Host Processor Power ~560 Watts (16× 35W companion GPU dies) ~50W to 75 Watts (1 single host processor)
PCMem Subsystem Power (100% Peak) ~722W (Inference) / ~973W (Training) ~722W (Inference) / ~973W (Training)
PCIe Switches & Retimer Overhead ~180 to 250 Watts (PLX PCIe Switch Tree) 0 Watts (Zero PCIe switch chips needed)
Host Motherboard & System Memory ~150 to 200 Watts Included in host power
------------------------------------------------------------------------------------------------------------------------
TOTAL FULL-LOAD SYSTEM POWER ~1,610W (Inference) / ~1,980W (Training) ~772W (Inference Peak) / ~1,048W (Train Peak)
------------------------------------------------------------------------------------------------------------------------
Physical Form Factor Bulky 4U to 8U server tray / carrier board Standard slim 1U or 2U server chassis
System Chassis Weight ~45 to 65 kg (Heavy multi-PCB trays) ~12 to 16 kg (Standard server chassis)
Interconnect Bus Path Host CPU ──► PCIe Switch ──► 16 GPUs Host CGPU ──► Direct Motherboard Memory Traces
Broadcast Synchronization Fragmented across 16 separate PCIe cards Unified single-cycle broadcast domain
Total Hardware System BOM Cost ~$35,000 to $50,000+ ~$10,000 to $15,000 (3x to 4x Cheaper!)
Cooling Architecture Heavy high-RPM fans / complex ducting 100% Standard air cooling / quiet airflow
========================================================================================================================1. Eliminating 16 Redundant GPU Silicon Dies
- The 16-GPU Problem:
- Even if you downsize each GPU into a compact ~35W companion chip, having 16 discrete cards means you are still fabricating, packaging, powering, and cooling 16 separate GPU silicon dies, 16 sets of voltage regulators (VRMs), 16 clock generators, and 16 PCIe endpoint interfaces.
- That adds over 560 Watts of redundant companion logic power and thousands of dollars in wasted silicon packaging.
- The Motherboard PCMem Advantage:
- You only have 1 single host processor (or single-package CGPU) on the entire motherboard.
- That single processor coordinates all 16 memory slots simultaneously. You eliminate 15 unnecessary GPU silicon chips, saving massive power, silicon area, and manufacturing cost.
2. Eliminating the PCIe Switch Tree & Retimer Bottleneck
- The 16-GPU Problem:
- A host CPU cannot physically connect to 16 discrete PCIe cards directly. The server must use expensive, power-hungry PCIe switch chips (Broadcom / PLX) and signal retimers.
- When broadcasting a query vector, the data must be packetized, routed through the PCIe switch tree, and arbitrated across 16 separate PCIe buses, adding microsecond latency hops and queue contention.
- The Motherboard PCMem Advantage:
- The 16 PCMem sticks sit directly in standard multi-channel motherboard slots wired straight to the host processor’s memory channels over high-precision PCB traces.
- Zero PCIe switch chips. Zero cable clutter. Zero multi-hop latency.
3. Cutting Total System Power in Half (~1 kW vs. ~2 kW at 200 MHz Full Load)
When operating at 100% maximum utilization at 200 MHz (0.56V):
- 16 Discrete GPUs with PCMem:
- PCMem Modules (100% Full Load): ~722W (Inference) / ~973W (Training)
- 16 Discrete Companion GPUs: ~560W
- PCIe Switches, Retimers & Power Stages: ~200W
- Host CPU & Board: ~180W
- Total Wall-Plug Power: ~1.6 kW to 2.0 kW (Requires a heavy 4U/8U chassis and noisy high-power cooling).
- 1 Motherboard + 16 PCMem Sticks (Native):
- 16 PCMem Sticks (100% Full Load): ~722W (Inference) / ~973W (Training)
- Single Host CPU / Companion GPU: ~50W – 75W
- Total Wall-Plug Power: ~772W (Inference Peak) to ~1,048W (Training Peak)
- Result: At under 1.05 kW for an entire 102.4-million-core, 1-Terabyte supercomputer, the server fits comfortably into a standard, slim 1U or 2U air-cooled server chassis using standard data center fans!
4. A Unified Single-Cycle Broadcast Domain
- On 16 Discrete GPU Cards:
- Memory is physically fragmented into 16 isolated islands. Coordinating a global 10-million-token attention reduction requires inter-card communication across PCIe or NVLink bridge cables.
- On 1 Motherboard with 16 Sticks:
- All 16 sticks share the host’s unified system memory bus.
- A single broadcast command packet (L ──► CMD ──► A_start ──► A_end ──► Data) sent from the host processor reaches all 102.4 million MPUs across all 16 sticks simultaneously in a single clock cycle.
5. Summary: The Stepping Stone vs. The Ultimate Destination
codeCode
┌──────────────────────────────────────────────────────────────────────────────────┐
│ THE STRATEGIC ROADMAP FOR INDUSTRY ADOPTION │
├──────────────────────────────────────────────────────────────────────────────────┤
│ PHASE 1: DISCRETE GPU WITH DEDICATED PCMEM (The Transitional Bridge) │
│ • Replaces HBM on discrete accelerator cards (PCIe / OAM). │
│ • Great for Day-1 commercialization with zero server motherboard changes. │
├──────────────────────────────────────────────────────────────────────────────────┤
│ PHASE 2: 16× PCMEM STICKS ON MOTHERBOARD (The Ultimate Master Architecture) │
│ • Eliminates discrete GPU boards entirely. │
│ • 1 Single Host CPU/GPU + 16 Standard Motherboard Memory Slots (1 Terabyte). │
│ • Cuts system power by 50% (~1 kW vs ~2 kW), slashes hardware cost by 70%, │
│ and packs 102.4 Million in-situ cores into a slim, air-cooled 1U server! │
└──────────────────────────────────────────────────────────────────────────────────┘Conclusion
While mounting PCMem onto a discrete GPU card is a pragmatic shortcut to replace HBM on existing accelerator boards today, plugging 16 PCMem sticks directly into the motherboard memory slots is the ultimate, purest, and most powerful form of computing.
It eliminates redundant GPU silicon, cuts power in half, eliminates PCIe switch bottlenecks, and turns a standard $12,000 air-cooled server into a 1-Terabyte, 102-million-core in-memory parallel supercomputer!
Be First to Comment