November 18, 2026 issue

Story 03 November 18, 2026 issue

Daily AI-generated issue

AWS reports 2.05× Dynamo throughput with disaggregated serving on SageMaker HyperPod Inference

Image: aws.amazon.com

Top News

NVIDIA Dynamo on SageMaker HyperPod Inference gives you managed endpoint setup or control over your own deployment. AWS reports 2.05× throughput for disaggregated versus aggregated serving in a Qwen3-235B-A22B-FP8 test with TensorRT-LLM.

Here's what you can configure:

  • InferenceEndpointConfig with dynamoSpec manages setup, defaults to vLLM, and supplies KV-aware routing and replica defaults.
  • Bring your own DynamoGraphDeployment for direct control over resources, weights, and endpoint management. Both paths support single-instance and multi-node deployments.
  • The managed path stages weights from S3 or FSx and handles TLS and endpoint registration. HyperPod manages node health and replacement.
  • The benchmark used 16 H200 GPUs across two p5en.48xlarge instances. Disaggregated serving delivered 207 output tokens/s/GPU versus 101, at concurrency 80 versus 32, while meeting P90 time-to-first-token below one second and median generation speed of at least 50 tokens/s per user.

One catch: Prefill and decode pools each load the full model weights, with KV cache transferred over EFA.

Try it: Configure InferenceEndpointConfig with dynamoSpec, then invoke the endpoint through the SageMaker Runtime API.

Sources: aws.amazon.com

This issue is researched and written by AI models, and every fact is checked against its cited source. No human edits it before it is sent.