I just got an idea in talk with Gemini to publish it here: this idea brings LAN’s partial scope broadcasting (subnet masked broadcasting) into PCM (parallel computing memory). The concepts and following names were brought up exactly by me, but Gemini did a lot of help by providing validation and technical details.
MPC is memory parallel computing, which is computing without need for sequential computing of different memory units.
MPU is memory parallel processing unit, and each MPU connects to a dedicated memory unit through a dedicated link between the MPU and the memory unit. A memory unit is a fixed number of bits of DRAM.
For example, a MPU is a micro cpu with 2000 to 4000 transistors and corresponds to 2048 to 4096 bits memory of a planar DRAM. A MPU may include the basic arithmatics and the calculations needed for parallel computing of transformer.
PCM is parallel computing memory, and a PCM chip may include as many as millions of MPUs and memory units (like 4GByte per chip). Multiple PCM chips (like 16 PCM chips of DDR5) may be installed on one PCB to make a PCM stick (like 64GByte per stick), and a PCM stick may be inserted in a standard DDR slot like DDR5 slot.
Each MPU on one PCM chip can not only do parallel computing as separated individual unit but also do parallel computing with each other in like doing sum computing through specific logic circuits and registers of MPU.
Each chip/PCB of PCM includes an chip/PCB bus which includes data bus and address bus, a PCB bus connects to each chip bus on the PCB, and a chip bus connects to and is shared by each MPU on the chip, and the PCB bus connects to CPU/GPU through a system bus.
There is a memory controller on a chip bus or on a PCB bus or on a system bus or on two or three of them.
A memory controller can forward or generate partial scope broadcasting to a chip bus (like with start and ending address for MPU/memory units on the chip), and all MPUs connecting to the chip bus will inspect the broadcasting packet, and if a MPU’s address is in the address scope of the partial scope broadcasting (like MPU’a address is in the scope from start address to the end address of the packet), this MPU will accept the packet to do computing or send to its memory unit. The memory controller can also forward or generate one address packet and uni-broadcast packet.
By the way, PCM/MPC better work with DRAM not HBM for lower cost and better stability/cooling.
Gemini thinks this is a good idea and patentable, so I ask Gemini to write it in patent format to paste it here, haha.
PATENT SPECIFICATION & CLAIMS
TITLE OF THE INVENTION:
PARALLEL COMPUTING MEMORY (PCM) APPARATUS AND PARTIAL SCOPE BROADCAST ADDRESSING METHOD FOR IN-SITU MEMORY PARALLEL PROCESSING UNITS (MPUS)
INVENTOR: oknomad
DISCLOSURE DATE: August 15, 2026
- FIELD OF THE INVENTION
The present invention relates to semiconductor memory architectures, memory-centric parallel computing systems, and hardware acceleration for artificial intelligence, and more particularly to a Parallel Computing Memory (PCM) architecture comprising a hierarchical bus structure (system bus, PCB bus, and chip bus), flexible memory controller placement across bus tiers, distributed Memory Parallel Processing Units (MPUs) acting as in-line micro-CPUs to dedicated planar DRAM memory units, and a Partial Scope Broadcast Addressing Scheme over the bus hierarchy supporting range-based (start-to-end address) and masked broadcast addressing for selective single-cycle concurrent activation of arbitrary MPU subsets. - BACKGROUND OF THE INVENTION
Traditional computing architectures rely on the Von Neumann model, wherein processing units (CPUs, GPUs) are separated from passive storage units (DRAM) by a shared data bus. In modern artificial intelligence workloads (such as Transformer Large Language Models, Generative World Models, and Physical AI vision-language-action systems), data must be sequentially transferred across this narrow bus, creating a severe memory data bus bottleneck where latency scales linearly with data size (O(N)).
Furthermore, conventional memory standards (DDR4, DDR5, LPDDR5, HBM) use strict 1-to-1 Unicast Addressing, where every memory command addresses only a single memory location. In parallel memory systems with millions of distributed processing elements, transmitting millions of individual unicast commands creates severe address bus congestion.
There is an urgent need for an architecture that eliminates the memory data bus bottleneck by combining distributed in-line micro-CPUs (MPUs) with a multi-tier bus architecture (PCB bus and chip bus) capable of receiving and forwarding partial scope broadcast addressing (including start-to-end address ranges and bitmasks) to drive millions of in-situ processing units concurrently over standard memory form factors such as DDR5 slots.
- SUMMARY OF THE INVENTION
The present invention provides a Parallel Computing Memory (PCM) apparatus implementing Memory Parallel Computing (MPC):
3.1 Hierarchical Bus Architecture: A PCM apparatus comprises a system bus connected to a host processor (CPU/GPU), a printed circuit board (PCB) bus comprising a PCB address bus and a PCB data bus, and a plurality of chip buses disposed on respective PCM chips. The PCB bus connects to each chip bus on the PCB, each chip bus connects to and is shared by a plurality of Memory Parallel Processing Units (MPUs) on the chip, and each MPU connects to a dedicated DRAM memory unit via a dedicated local link.
3.2 Flexible Memory Controller Configuration: One or more memory controllers are disposed on a chip bus, on a PCB bus, on a system bus, or distributed across two or three of said buses, configured to generate or forward single-address packets, partial scope broadcast packets, and universal broadcast (uni-broadcast) packets across the bus hierarchy.
3.3 In-Line Micro-CPU (MPU) Architecture: Each MPU is a compact micro-CPU comprising approximately 2,000 to 4,000 transistors, coupled to its respective chip bus and interfacing with a dedicated planar DRAM memory unit (e.g., 2,048 to 4,096 bits of 1T1C DRAM) via a dedicated, micrometer-short local link. The MPU operates in an In-Situ Compute Mode (executing math locally) and a Passthrough Mode (passing host read/write data with negligible latency).
3.4 Inter-MPU Cooperative Computing: MPUs on a PCM chip are configured to compute both as independent parallel units (for local vector math) and cooperatively with each other through specific logic circuits and registers across an interconnected reduction network (for collective operations such as global summation and maximum reduction).
3.5 Partial Scope Address Broadcasting: Each MPU contains an address comparator coupled to its chip bus. When a memory controller forwards or generates a partial scope broadcast packet defining an address scope (such as via a start address and an ending address, or a subnet bitmask), all MPUs on the chip bus inspect the packet. If an MPU’s address falls within the start-to-end address scope of the partial scope broadcast, the MPU accepts the packet to perform in-situ computation or send data to its dedicated memory unit. The system further supports single-address unicast packets and universal broadcast (uni-broadcast) packets targeting all MPUs.
3.6 Module Scale and Standard Slot Compatibility: A PCM chip comprises millions of MPUs and memory units (e.g., providing approximately 4 GBytes of storage capacity per chip). A plurality of PCM chips (e.g., 16 chips) are mounted on a common PCB to form a PCM stick (e.g., providing 64 GBytes of capacity per stick) configured to be inserted into standard memory slots such as DDR5 or CXL slots.
- BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram illustrating the hierarchical PCM architecture showing the system bus, the PCB bus, the chip buses, the memory controller placement options, and the plurality of in-line MPUs coupled to dedicated DRAM memory units.
FIG. 2 is a schematic diagram illustrating the In-Line MPU Gateway, showing the dedicated local link to the 2,048–4,096 bit DRAM memory unit and the address comparator.
FIG. 3 is a circuit-level diagram of the MPU micro-architecture, depicting the micro-CPU control logic, SRAM working registers, bit-serial ALU, and inter-MPU cooperative reduction paths.
FIG. 4 is a diagram illustrating Single-Address Mode, Partial Scope Broadcast Mode (Start-to-End Address Range and Subnet Mask), and Universal Broadcast Mode across the PCB and chip bus hierarchy. - DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
5.1 Hierarchical Bus Architecture and Memory Controller Placement
Referring to FIG. 1, the system comprises a multi-tier bus hierarchy:
- A system bus interfaces with the host CPU/GPU.
- A PCB bus (on a DIMM or module substrate) connects the system bus to multiple PCM chips (e.g., 16 chips on a 64 GByte PCM module).
- A chip bus on each PCM chip (e.g., a 4 GByte chip) connects to and is shared by all MPUs on that chip.
- One or more memory controllers are situated on the chip bus, the PCB bus, the system bus, or across any combination of them. The memory controller is configured to generate or forward single-address packets, partial scope broadcast packets, or universal broadcast packets down through the bus hierarchy to the chip buses.
5.2 In-Line MPU Gateway and Dedicated Link
Referring to FIG. 2, each MPU is positioned as an in-line micro-CPU between its local chip bus and its dedicated DRAM memory unit (e.g., 2,048 to 4,096 bits of planar 1T1C DRAM). The dedicated link is an ultra-wide, micrometer-short local interconnect routed vertically (via direct hybrid bonding) or horizontally adjacent to the DRAM sense amplifiers.
In Passthrough Mode, the MPU enables bypass gates or pipeline latches to pass host CPU/GPU read/write traffic at full line rate. In In-Situ Compute Mode, the MPU isolates the DRAM unit and executes calculations locally on the bitlines without transmitting raw operand data over the chip bus, PCB bus, or system bus.
5.3 Address Decoding and Scope Protocol
Referring to FIG. 4, an address packet transmitted across the bus hierarchy is selectively configured as:
- A Single-Address Packet: Targeted to a single physical MPU address for individual access.
- A Partial Scope Broadcast Packet: Comprising a defined address scope, such as a designated Start Address and Ending Address, or a Subnet Bitmask with a Group/Layer identifier.
- A Universal Broadcast Packet: Configured to address all MPUs on the chip or PCB.
Each MPU incorporates an address comparator:
(1) Single-Address Activation: If the packet is a single-address packet, the comparator matches the MPU’s unique physical ID to read, write, or compute on a single unit.
(2) Partial Scope Broadcast Activation: If the packet is a partial scope broadcast packet, the comparator evaluates whether the MPU’s address falls within the scope bounded by the start address and ending address (or matching masked subnet bits). If matched, the MPU accepts the packet to execute computation or send data to its dedicated memory unit. This allows the memory controller to activate specific subsets of MPUs (e.g., all MPUs allocated to Expert #3 in an MoE model) in a single clock cycle.
(3) Universal Broadcast Activation: If the packet is a universal broadcast packet, all MPUs across the chip or module accept the packet simultaneously.
5.4 MPU Micro-Architecture and Inter-MPU Cooperative Computing
Referring to FIG. 3, each MPU comprises:
- An SRAM working register file comprising working registers (e.g., 2 to 3 registers of 64 bits each) and local status/format registers. Intermediate results may also be written directly to designated DRAM scratchpad rows.
- A bit-serial arithmetic datapath comprising arithmetic and logic circuits configured to execute shift-and-add multiplication, base-2 Softmax (2^(x – max)), right-shift division, CORDIC trigonometric operations (sin/cos for RoPE), and basic arithmetic operations.
- Inter-MPU Communication: Adjacent MPUs are interconnected via specific reduction logic circuits and registers across an on-chip reduction network (such as a binary H-tree adder network), allowing MPUs to cooperatively compute collective operations (such as global sum and global maximum) in logarithmic time (O(log M)).
- Multi-Format Support: Dynamically reconfigurable to execute calculations across multiple formats including FP4, INT4, FP8, INT8, FP16, BF16, FP32, BF32, and FP64.
- PATENT CLAIMS
WHAT IS CLAIMED IS:
- A Parallel Computing Memory (PCM) apparatus implementing Memory Parallel Computing (MPC), comprising:
a printed circuit board (PCB) bus comprising a PCB address bus and a PCB data bus, wherein said PCB bus is configured to connect to an external host processor through a system bus;
a plurality of PCM chips disposed on said PCB, wherein each PCM chip comprises a chip bus comprising a chip address bus and a chip data bus, and wherein said PCB bus connects to each chip bus on the PCB;
at least one memory controller disposed on a chip bus, on said PCB bus, on said system bus, or on two or three of said buses;
a plurality of Memory Parallel Processing Units (MPUs) disposed on each PCM chip, wherein the chip bus of each PCM chip connects to and is shared by each MPU on the chip; and
a plurality of dedicated DRAM memory units, wherein each MPU is coupled to a corresponding dedicated DRAM memory unit through a dedicated local interconnect;
wherein each MPU comprises an address comparator configured to inspect address packets transmitted over its respective chip bus and selectively activate said MPU to perform in-situ computation or data transfer with its corresponding dedicated DRAM memory unit. - The apparatus of claim 1, wherein:
said at least one memory controller is configured to forward or generate a partial scope broadcast packet to said chip bus, wherein said partial scope broadcast packet defines an address scope bounded by a start address and an ending address, or defined by a subnet bitmask; and
the address comparator of each MPU is configured to inspect said partial scope broadcast packet and accept the packet if the address of the MPU falls within the address scope of the partial scope broadcast packet, thereby enabling concurrent activation of an arbitrary subset of said plurality of MPUs in a single clock cycle without bus collision. - The apparatus of claim 2, wherein:
said at least one memory controller is further configured to selectively forward or generate a single-address packet targeting an individual MPU, and a universal broadcast packet targeting all MPUs on the chip bus. - The apparatus of claim 2, wherein:
said concurrent activation of a subset of MPUs selectively activates only MPUs associated with a designated expert network within a Mixture-of-Experts (MoE) neural network architecture, while unselected MPUs remain in an un-activated low-power state. - The apparatus of claim 1, wherein:
each MPU is an integrated micro-CPU comprising approximately 2,000 to 4,000 transistors; and
each dedicated DRAM memory unit comprises 2,048 to 4,096 bits of planar DRAM. - The apparatus of claim 1, wherein:
each PCM chip has a capacity of approximately 4 GBytes; and
said PCB comprises approximately 16 of said PCM chips to form a PCM stick having a capacity of approximately 64 GBytes insertable into a standard DDR5 memory slot or CXL interface. - The apparatus of claim 1, wherein:
said at least one memory controller comprises a first controller on the system bus and a second controller on the PCB bus or on each chip bus, configured to cooperatively forward broadcast commands across the bus hierarchy. - The apparatus of claim 1, wherein:
each MPU is configured to perform parallel computation independently on its dedicated DRAM memory unit, and is further coupled to adjacent MPUs through dedicated logic circuits and registers across an inter-MPU communication network to perform cooperative parallel computation across multiple MPUs, including collective summation and maximum reduction. - The apparatus of claim 1, wherein each MPU is configured to operate in:
an In-Situ Compute Mode, wherein the MPU reads data from its dedicated DRAM memory unit over the dedicated local interconnect, executes mathematical operations in-place, and writes results back to said dedicated DRAM memory unit without transmitting raw operand data over the chip bus, PCB bus, or system bus; and
a Passthrough Mode, wherein the MPU acts as a transparent low-latency buffer passing data directly between the chip bus and its dedicated DRAM memory unit during external host read and write operations. - The apparatus of claim 1, wherein each MPU comprises:
a plurality of static random-access memory (SRAM) working registers; and
a bit-serial arithmetic logic unit (ALU) configured to execute shift-and-add multiplication, base-2 exponential calculation (2^x), division, and Coordinate Rotation Digital Computer (CORDIC) trigonometric calculations for Rotary Position Embeddings (RoPE). - The apparatus of claim 10, wherein:
said MPU executes prefill linear projections (Q, K, V = X @ Wqkv) and self-attention operations (M = softmax(Q @ K.T) @ V) in-situ, wherein computation latency is independent of context window size (O(1) constant-time latency scaling). - A method for parallel memory computing in a Parallel Computing Memory (PCM) apparatus, the method comprising:
transmitting or forwarding an address packet defining a partial scope broadcast from a memory controller across a bus hierarchy comprising a system bus, a PCB bus, and a chip bus shared by a plurality of distributed Memory Parallel Processing Units (MPUs), wherein said partial scope broadcast defines an address scope bounded by a start address and an ending address, or defined by a subnet bitmask;
inspecting the address packet at said plurality of MPUs using local address comparators;
concurrently accepting the packet at a selected subset of MPUs whose address falls within the address scope of the partial scope broadcast;
loading operand data from dedicated DRAM memory units into the accepted MPUs over dedicated point-to-point local interconnects; and
executing arithmetic operations in parallel across all accepted MPUs in-situ without streaming raw operand data over the chip bus or PCB bus. - The method of claim 12, further comprising:
performing cooperative parallel reduction across multiple accepted MPUs through specific logic circuits and registers across an inter-MPU reduction network to compute a global sum or global maximum value across intermediate results of said arithmetic operations.
Be First to Comment