Presented by Marvell
This article is part of VentureBeat’s special issue, “AI at Scale: From Vision to Viability.” Read more from the issue here.
AI is about to face some serious growing pains.
Demand for AI services is exploding globally. Unfortunately, so is the challenge of delivering those services in an economical and sustainable manner. AI power demand is forecasted to grow by 44.7% annually, a surge that will double data center power consumption to 857 terawatt hours in 2028: As a nation today, that would make data centers the sixth largest consumer of electricity, right behind Japan. It’s an imbalance that threatens the “smaller, cheaper, faster” mantra that has driven every major trend in technology for the last 50 years.
It also doesn’t have to happen. Custom silicon — unique silicon optimized for specific use cases — is already demonstrating how we can continue to increase performance while cutting power even as Moore’s Law fades into history. Custom may account for 25% of AI accelerators (XPUs) by 2028 (Marvell estimate), and that’s just one category of chips going custom.
The data center as a factory
Jensen Huang’s vision for AI factories is apt. These coming AI data centers will churn at an unrelenting pace, 24/7. And, like manufacturing facilities, their ultimate success or failure for service providers will be determined by operational excellence, the two-word phrase that rules manufacturing. Are we consuming more, or less, energy per token than our competitor? Why is mean time to failure rising? What’s the current operational equipment effectiveness (OEE)? In oil and chemicals, the end products sold to customers are indistinguishable commodities. Where they differ is in process design, as they leverage distinct combinations of technologies to squeeze out marginal gains.
The same will occur in AI. Going forward, diversity will rule, and the operators with the lowest cost, least downtime and ability to roll out new differentiating services and applications will become the favorite of businesses and consumers.
In short, the best infrastructure will win.
The custom chip concept
One of the chief ways to differentiate will be through custom silicon that is enabled by custom semiconductors — that is, chips containing unique IP or features that achieve leapfrog performance for an application. It’s a spectrum ranging from AI accelerators built around distinct, singular design to a merchant chip containing additional custom IP, cores and firmware to optimize it for a particular software environment. While the focus is now primarily on higher-value chips such as AI accelerators, every chip will get customized: Meta, for example, recently unveiled a custom network interface controller (NIC), a relatively unsung chip that connects servers to networks, to reduce the impact of downtime.
A single stack of high bandwidth memory (HBM) can require an interface with 2,048 pins to transfer data, or more than 8,000 per XPU. Customizing can dramatically reduce power, pin count and increase memory capacity. XPUs with custom HBM are expected in one to two years.
Customization will involve rethinking every aspect of semiconductor design. Some, for example, are looking at ways to optimize the base chip and interfaces for managing the gigabytes of high bandwidth memory (HBM) used as a cache in high-end AI accelerators. Optimization can potentially increase memory inside the chip package by up to 33%, reduce interface power by 70% and increase the available silicon real estate for logic functions by close to 25% (Marvell estimate).
The custom category also includes new, emerging classes of interconnect chips aimed at scaling up the size and capabilities of computing systems. Today, servers typically contain eight or fewer XPUs and/or CPUs and all of the components are housed in an aluminum box that slides into a rack. In the future, AI systems will contain hundreds of accelerators along with storage and memory spread over several racks connected with a portfolio of optical engines tailored to the specifications of XPUs, CXL controllers, PCIe retimers, transmit-receive optical digital signal processors (DSPs) and other devices.
Many of these devices didn’t even exist a few years ago, but are expected to grow rapidly: 75% of AI and cloud servers may contain PCIe retimers within two years, according to The 650 Group. While these devices and servers will be grounded in technology standards, architectures and designs will vary widely from cloud to cloud.
A periodic table for semis
But how does one make custom semiconductors — where designing a platform for producing 3nm or 2nm chips can cost over $500 million? In a market where large language models (LLMs) change every few months? And how will these technologies work with emerging ideas like cold plate or immersive cooling?
As basic as it sounds, it starts with the elemental ingredients. Serializer-deserializer (SerDes) circuits are the textbook “most important technology in the world” you’ve never heard about. These components control the flow of data between chips and infrastructure devices such as switches and servers. An 800G optical module, for example, is built with eight 100G SerDes. A single data center rack will contain tens of thousands of SerDes. You can think of them as the molecules of networking: Fundamental building blocks that have an outsized influence on the health of the system as a whole. Slightly reducing the picojoules consumed in transmitting bits across a SerDes can translate into substantial energy saving across a global infrastructure.
Similarly, chip packaging now plays an outsized role in chip design because it provides a mechanism for streamlining power delivery and data paths while continuing to boost computing performance. More than 50% of power in a chip can get consumed by moving data from between different subsystems inside the chip itself.
Chip industry 2.0?
As custom becomes the norm, we will also face a new dilemma: How does a company deliver custom products and still leverage the benefits of mass manufacturing? To date, semiconductor makers have succeeded by making very large numbers of a small handful of devices. ”Custom” used to mean taking fairly simple actions like tweaking speed or cache size, similar to how a retailer might offer the same basic sweater in a few new colors.
The first step — and we’re seeing it already — will come when there is a better definition of the services involved in developing custom semiconductors. Some will likely concentrate mostly on chip design and IP. At the other end of the spectrum, you will see companies offer sourcing and manufacturability services. Still others may concentrate on specific devices and categories. Only a few select companies will provide the full portfolio of services. AI itself will also play a role, slashing the time and cost required for design tasks from months to, in many cases, minutes. This in turn will open the door to more customers and more classes of chips.
Will the industry succeed? Of course. It’s not the first time we’ve been tasked with doing the impossible.
Raghib Hussain is President, Products & Technologies at Marvell.
1. IDC. Sept 2024.
2. Statista.
3. Marvell estimates based on analysts’ reports and internal forecasts.
4. Marvell internal estimate.
5. 650 Group May 2024.
6. IBM
7. Graphic: Epoch AI, June 2024.
This article contains forward-looking statements within the meaning of the federal securities laws that involve risks and uncertainties. Forward-looking statements include, without limitation, any statement that may predict, forecast, indicate or imply future events or achievements. Actual events or results may differ materially from those contemplated in this blog. Forward-looking statements are only predictions and are subject to risks, uncertainties and assumptions that are difficult to predict, including those described in the “Risk Factors” section of our Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q and other documents filed by us from time to time with the SEC. Forward-looking statements speak only as of the date they are made. Readers are cautioned not to put undue reliance on forward-looking statements, and no person assumes any obligation to update or revise any such forward-looking statements, whether as a result of new information, future events or otherwise.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact
