Memory bandwidth growth is far outpaced by the expansion of model scale, and the "memory wall" is becoming the core bottleneck in AI infrastructure. According to Bank of America Securities' latest report, industry giants such as Micron, Samsung, SK Hynix, and Broadcom reached a strong consensus at the 2026 AI Infrastructure Summit: the industry is shifting from computing power stacking to efficiency-driven approaches, with the evaluation framework transitioning from FLOPs to tokens/W. Hierarchical memory decoupling and open interconnect architectures are set to become the central narrative for the next wave of semiconductor investment.
The memory wall is emerging as a central bottleneck to the expansion of AI infrastructure, and the semiconductor industry is turning to architectural re‑engineering as its breakthrough strategy.
According to the Zhaofeng Trading Desk, Bank of America Securities' research team stated in its latest report that, on the second day of the 2026 AI Infrastructure Summit, major vendors—including Micron, Samsung, SK Hynix, Broadcom, Marvell, Intel, Qualcomm, OpenAI, AWS, and Google—along with cloud computing giants, will make a collective appearance. The core agenda is heavily focused on strategies for addressing the "memory wall" issue. Participants generally agree that decoupling memory and storage through disaggregation has become the most widely accepted technological approach at present.
This trend has a direct impact on the semiconductor investment landscape: efficiency metrics are replacing raw computing power as the new industry benchmark, shifting from raw FLOPs to tokens per watt (tokens/W) or tokens per dollar (tokens/$); meanwhile, interconnect architectures are accelerating their move toward openness, and demand for custom chips is rising in tandem, drawing market attention to the product strategies of companies such as Broadcom, Marvell, and Astera Labs.
The Memory Wall: The Core Bottleneck of AI Scaling
The growth rates of memory bandwidth and capacity are far outpaced by the explosive expansion of model sizes—this is the central finding that this year's summit has reiterated from the Hot Chips conference in August.
The scale of Transformer models has grown by roughly 240-fold every two years, while memory bandwidth and capacity have increased by only about twofold over the same period, leading to a widening gap between the two. This structural imbalance is making GPU-centric conventional architectures increasingly unsustainable, driving the industry toward specialized, tiered memory solutions.
At the summit, Qualcomm unveiled its "HBC" (High-Bandwidth Computing) solution, which achieves a capacity-to-power ratio approximately 200 times that of SRAM and a bandwidth-to-power ratio about six times that of HBM by directly stacking LPDDR memory on top of the compute chip. Samsung's zHBM also follows a similar 3D DRAM stacking approach.
SK Hynix highlighted three specialized memory tiers: first, PIM (Processing-In-Memory), designed for memory‑intensive workloads, which delivers a 288‑fold capacity increase per rack compared to SRAM; second, HBF (High‑Bandwidth Flash), tailored for long‑context scenarios, offering roughly ten times the capacity of HBM; and third, the SALT‑KV software solution, enabling temperature‑aware KV cache scheduling across tiers.
Efficiency First: The New AI Scaling Law
The summit sent a clear industry signal: the era of brute‑force computing power is giving way to a new, efficiency‑driven paradigm.
A report by Bank of America Securities notes that as AI applications evolve toward agentic AI, the industry's evaluation framework is shifting from prioritizing raw FLOPs to emphasizing efficiency metrics such as tokens/W and tokens/$. This shift is directly driving demand for memory diversification and hierarchical decoupling: different types of workloads require memory tiers tailored to their specific characteristics, rather than relying on a single high‑performance memory solution.
This has a substantial impact on both data center operators' procurement strategies and chip manufacturers' product roadmaps, with memory architecture flexibility and energy efficiency emerging as new dimensions of competition.
Interconnectivity and customization are advancing in parallel.
At the network interconnection layer, Ethernet has largely cemented its dominant position in the scale-out domain, with openness and interoperability serving as the key factors behind its success.
Broadcom is advancing Ethernet to every layer of the interconnected architecture: on the horizontal scaling front, Tomahawk 6 (102.4T) has entered high-volume production and has been deployed among hyperscale cloud providers; on the vertical scaling front, Thor Ultra NIC and the ESUN solution are ready; and for cross-domain interconnectivity, Jericho 4 is taking charge—all built on an open architecture, without relying on vertical integration.
Meanwhile, demand for customization is also on the rise. Marvell offers an end-to-end solution that includes custom XPU attach, a full-optical interconnect suite, and multi-protocol support (UALink, NVLink Fusion, ESUN); Astera Labs, on the other hand, builds on the universal PCIe protocol and deploys specialized chips such as Scorpio and Leo to address interconnect and memory applications.
Production Speed and Reliability: Equally Important as Chip Design
At the summit, leading-edge laboratories and data center operators conveyed a unified message: the pace of rack mass production and system reliability have become as critical as chip design itself.
By aligning test capacity with production scale and strategically co-locating manufacturing processes with deployment sites, AWS has dramatically shortened the traditionally six- to nine-month post-silicon testing and stabilization cycle, significantly reducing the time lag between chip release and actual data center deployment.
Software co‑design is likewise regarded by hyperscale cloud providers as a key enabler for accelerating deployment. Features such as native PyTorch support and Hugging Face portability with minimal code modifications are seen as critical levers for shortening the end-to-end cycle from chip design to production‑ready applications.
Bank of America Securities' research team believes that a rapid product‑iteration cadence spanning more than one year is critical to token economics and market competitiveness. Achieving this pace no longer hinges solely on chip‑design capabilities; it increasingly depends on system‑level engineering integration and software–hardware co‑optimization.
Editor/lambor