The Unseen Reshaping of AI Hardware: Memory Shortages and CPO Delays Take Center Stage

In a recent in-depth discussion, Dylan Patel, founder of semiconductor and AI infrastructure research firm SemiAnalysis, outlined the less-discussed hardware realities underpinning the current AI compute race. His analysis highlights complex supply-demand adjustments within the foundational hardware stack, with two trends standing out: memory chips may face supply constraints for years to come, while the rollout timeline for the next-generation interconnect technology, CPO, is significantly later than many anticipated.

Memory: A Structural Shortage with Room to Run

Dylan Patel emphasized that the issue in memory is not a typical cyclical swing but a structural supply-demand imbalance. The explosive growth in AI model parameters and datasets has skyrocketed demand for high-bandwidth memory (HBM) and high-speed storage, driven by both training and inference workloads. Ramping up production capacity takes time. He suggests this tight environment could persist for several years, and current prices may not yet fully reflect the impending strain, indicating significant potential for further price increases.

This assessment carries weight for investors and AI companies alike, implying:

  • Memory costs will become a more permanent and rigid component of AI compute cost models.
  • Companies with secure memory supply chains or advanced packaging tech could gain a strategic edge.
  • Innovation focusing on memory efficiency and alternatives will gain urgency.

CPU vs. GPU: A Reality Check on Value

Another pointed observation concerned the role of CPUs. While workloads like AI Agents and reinforcement learning are boosting CPU demand, Patel argues the market may be overestimating the quality of this growth. Some of it stems from "catch-up" configurations in servers rather than purely new demand. In the total bill of materials for AI servers, GPUs continue to dominate absolutely, with CPUs playing a supporting role. This calls for a more nuanced analysis of value distribution within AI investments.

The CPO Delay: An Unexpected Reprieve for Copper

Perhaps the most surprising forecast concerns Co-Packaged Optics (CPO). This promising technology, designed to tackle the power and bandwidth bottlenecks of high-speed data center interconnects, is now expected to see mass commercial adoption only by late 2028 or 2029—later than many industry watchers had projected.

This delay creates ripple effects:

  • It grants existing copper interconnect solutions a longer-than-expected technology lifecycle and market window.
  • Suppliers of copper connectors may enjoy an extended "bonus period."
  • It allows more time for CPO technology to mature, costs to decrease, and the ecosystem to develop.

Overall, Dylan Patel's analysis paints a picture of a more complex and gradual AI hardware evolution. Beneath the rapid pace of software innovation, the physical infrastructure that supports it moves at its own rhythm, presenting both immediate constraints like memory shortages and technological realities like CPO delays. For anyone navigating the AI wave, understanding these hardware-level variables is becoming crucial for sound decision-making.