AWS re:Invent: AWS Unveils Next-Generation Graviton, Trainium Chips

The News: Amazon Web Services (AWS) launched the next generation of its Graviton and Trainium chips at re:Invent, improving performance that will help power in-memory databases, machine learning (ML) training, and generative AI applications. Read the full announcement on the AWS website.

AWS re:Invent: AWS Unveils Next-Gen Graviton, Trainium Chips

Analyst Take: AWS took the opportunity of its annual AWS re:Invent conference to double down on its custom silicon strategy with updates to its Graviton and Trainium offerings. AWS, Microsoft with Maia, and Google’s Tensor Processing Unit (TPU) offerings stand as pivotal players in the ever-evolving landscape of custom silicon for AI workloads. AWS’s relentless pursuit of innovation, evident in its Graviton processors and Inferentia chips since 2018, has empowered enterprises to optimize their AI workloads with cost-effective, high-performance solutions. Microsoft, with the Maia architecture, has demonstrated a commitment to providing versatile AI hardware that seamlessly integrates with Azure, offering businesses a comprehensive ecosystem for AI-driven endeavors. Google’s TPUs continue to shine as a testament to its dedication to AI acceleration, delivering exceptional processing power and efficiency for complex AI models. These tech giants are shaping the future of custom silicon, offering businesses the tools needed to excel in the realm of AI workloads.

AWS-designed Graviton4 and Trainium2 will be used in workloads and applications running in Amazon Elastic Compute Cloud (Amazon EC2). Both are designed by AWS. Graviton4 is a general-purpose microprocessor chip for large workloads, and Trainium2 accelerator chips are built for high-performance training of foundation models (FMs) and large language models (LLMs) with billions of parameters.

Performance Claims

The most recent iterations of these chips have witnessed remarkable advancements in both performance and power efficiency, as officially reported by AWS. Graviton4, in particular, stands out with its impressive capabilities, boasting up to a 30% enhancement in compute performance compared with its predecessor, the Graviton3 processors. Additionally, it incorporates 50% more cores, amplifying its processing potential, and an astounding 75% boost in memory bandwidth, facilitating faster data access and manipulation.

The advancements of Trainium from generation to generation is really impressive. Trainium2 represents a significant leap forward in the realm of AI model training. It has been meticulously designed to deliver a staggering fourfold increase in training speed when compared with the initial generation of Trainium chips. This transformative leap in speed is poised to revolutionize AI model training, allowing for quicker iterations and more agile development. Furthermore, AWS’s deployment of Trainium2 in EC2 UltraClusters, with the capability to scale up to 100,000 chips, opens the door to training Fine-Grained Model Search (FMS) and LLMs at unprecedented speeds while concurrently enhancing energy efficiency, a pivotal consideration in today’s environmentally conscious computing landscape. These remarkable advancements in both Graviton4 and Trainium2 underscore AWS’s commitment to pushing the boundaries of performance and efficiency in the custom silicon domain. While generational comparisons are interesting, we will be looking for more comparative tests from our Futurum Labs team before we pass judgement on the competitive landscape for these new offerings.

Use Cases

AWS claims to have built more than 2 million Graviton processors and has more than 50,000 customers—including the top 100 EC2 customers—using Graviton-based instances. These customers include Datadog, DirecTV, Discovery, Formula 1 (F1), NextRoll, Nielsen, Pinterest, SAP, Snowflake, Sprinklr, Stripe, and Zendesk. Graviton is supported by AWS managed services such as Amazon Aurora, Amazon ElastiCache, Amazon EMR, Amazon MemoryDB, Amazon OpenSearch, Amazon Relational Database Service (Amazon RDS), AWS Fargate, and AWS Lambda.

Graviton4 will be available in memory-optimized Amazon EC2 R8g instances, used for high-performance databases, in-memory caches, and big data analytics workloads. R8g instances provide up to 3x more vCPUs and 3x more memory than current generation R7g instances. Graviton4-powered R8g instances are available today in preview.

Trainium2 will be available in Amazon EC2 Trn2 instances, containing 16 Trainium chips in a single instance. Trn2 instances can help customers scale up to 100,000 Trainium2 chips in next-generation EC2 UltraClusters, interconnected with AWS Elastic Fabric Adapter (EFA) petabit-scale networking. Their compute and scale can cut LLM training time considerably, making them a good fit for generative AI.

Looking Ahead

In the fiercely competitive custom silicon arena, AWS maintains a substantial time in market lead over Microsoft and Google, with AWS claiming over 10,000 customers for Graviton alone. AWS’s Graviton processors, now in their fourth generation, showcase a commitment to ongoing innovation. This continuous improvement translates into superior performance, lower latency, and increased efficiency for AWS customers. AWS also stands out for its flexibility, offering chips from AMD, Intel, and NVIDIA for EC2 workloads. The mix of merchant and homegrown silicon will pivot to homegrown, because AWS AI silicon is so good. In contrast, Microsoft and Google are still catching up, with AWS’s established presence casting a long shadow. While they show promise, AWS’s multi-generation lead, consistent improvement, and diverse chip ecosystem solidify its position as the leader in custom silicon for AI workloads.

The ability to control the ecosystem, create greater economies of scale, and become stickier with its own chips is good business for AWS, and although it is not terrible for merchant silicon, it is hard to see a situation where this does not become more of a focus, a margin creator, and lead go-to-market for AWS.

Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.

Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.

Other insights from The Futurum Group:

AWS Chip Lab Tour

AWS Serves Up NVIDIA GPUs for Short-Duration AI/ML Workloads

Microsoft’s Customer Silicon: A Game-Changer for AI and Cloud Computing?