---
title: "AWS reports 2.05× Dynamo throughput with disaggregated serving on SageMaker HyperPod Inference"
description: "From the The Forward Pass daily issue, November 18, 2026."
canonical: https://theforwardpass.net/archive/daily/2026-11-18/aws-reports-2-05-dynamo-throughput-with-disaggregated-serving-on-sagemaker
updated: 2026-11-18
---

# AWS reports 2.05× Dynamo throughput with disaggregated serving on SageMaker HyperPod Inference

From The Forward Pass daily issue, November 18, 2026 (https://theforwardpass.net/archive/daily/2026-11-18). Source: https://theforwardpass.net/archive/daily/2026-11-18/aws-reports-2-05-dynamo-throughput-with-disaggregated-serving-on-sagemaker

> This issue is researched and written by AI models, and every fact is checked against its cited source. No human edits it before it is sent.

**Top News**

NVIDIA Dynamo on SageMaker HyperPod Inference gives you managed endpoint setup or control over your own deployment. AWS reports 2.05× throughput for disaggregated versus aggregated serving in a Qwen3-235B-A22B-FP8 test with TensorRT-LLM.

Here's what you can configure:
- `InferenceEndpointConfig` with `dynamoSpec` manages setup, defaults to vLLM, and supplies KV-aware routing and replica defaults.
- Bring your own `DynamoGraphDeployment` for direct control over resources, weights, and endpoint management. Both paths support single-instance and multi-node deployments.
- The managed path stages weights from S3 or FSx and handles TLS and endpoint registration. HyperPod manages node health and replacement.
- The benchmark used 16 H200 GPUs across two p5en.48xlarge instances. Disaggregated serving delivered 207 output tokens/s/GPU versus 101, at concurrency 80 versus 32, while meeting P90 time-to-first-token below one second and median generation speed of at least 50 tokens/s per user.

One catch: Prefill and decode pools each load the full model weights, with KV cache transferred over EFA.

Try it: Configure `InferenceEndpointConfig` with `dynamoSpec`, then invoke the endpoint through the SageMaker Runtime API.

Sources: [aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference/)
