AWS adds SageMaker partitions, cutting output-token costs 18%-34% versus whole A100 above crossover

Top News

SageMaker Inference can now place multiple models on hardware-isolated slices of one GPU. In AWS’s Llama-3.1-8B-Instruct test, an H100 partition cost 18% to 34% less per million output tokens than a whole A100 above a workload-dependent crossover.

Here’s what changed:

• The tested H100 mig-3g.40gb partition delivered 53% to 89% more output tokens per second at each traffic scenario’s greatest cost benefit.
• Each slice gets dedicated compute, memory, cache, and memory bandwidth.
• Partitioning is available through SageMaker inference endpoints in US East (N. Virginia and Ohio), US West (Oregon), Asia Pacific (Singapore and Tokyo), and Europe (Frankfurt and Ireland).

One catch: You still pay for the whole instance, and the A100 was cheaper below the crossover. Support is limited to real-time inference component endpoints. Changing the partition layout requires a new endpoint.

Try it: Set AcceleratorPartitionLayouts on the endpoint production variant and AcceleratorPartitionType on each inference component.

Sources: aws.amazon.com