Story 03 November 18, 2026 issue
Daily AI-generated issue
AWS reports 2.05× Dynamo throughput with disaggregated serving on SageMaker HyperPod Inference
Top News
NVIDIA Dynamo on SageMaker HyperPod Inference gives you managed endpoint setup or control over your own deployment. AWS reports 2.05× throughput for disaggregated versus aggregated serving in a Qwen3-235B-A22B-FP8 test with TensorRT-LLM.
Here's what you can configure:
InferenceEndpointConfigwithdynamoSpecmanages setup, defaults to vLLM, and supplies KV-aware routing and replica defaults.- Bring your own
DynamoGraphDeploymentfor direct control over resources, weights, and endpoint management. Both paths support single-instance and multi-node deployments. - The managed path stages weights from S3 or FSx and handles TLS and endpoint registration. HyperPod manages node health and replacement.
- The benchmark used 16 H200 GPUs across two p5en.48xlarge instances. Disaggregated serving delivered 207 output tokens/s/GPU versus 101, at concurrency 80 versus 32, while meeting P90 time-to-first-token below one second and median generation speed of at least 50 tokens/s per user.
One catch: Prefill and decode pools each load the full model weights, with KV cache transferred over EFA.
Try it: Configure InferenceEndpointConfig with dynamoSpec, then invoke the endpoint through the SageMaker Runtime API.
Sources: aws.amazon.com
This issue is researched and written by AI models, and every fact is checked against its cited source. No human edits it before it is sent.