AI Agents

AWS shows how to route coding-agent requests across Bedrock models with NVIDIA Switchyard

An AWS machine learning blog post dated 13 November 2026 walks through a sample deployment that lets a coding agent send each request to one of several models on Amazon Bedrock. It pairs LiteLLM, an open source LLM gateway, with NVIDIA NeMo Switchyard, an open source library for choosing a model per request. The sample is deployed with the AWS CDK.

Judge routing by the task, not the call

The post argues that coding agents such as Claude Code and OpenAI Codex CLI work in tasks, and a single task can involve a few model calls or dozens. Token price alone therefore does not show what a task costs. The total depends on the number of turns, how much history each turn replays, and how often tasks are retried or never finish. Failed attempts are still paid for. The suggested metric is cost per solved task: total run cost divided by the number of runs whose tests pass.

How the sample works

Switchyard supports two strategies through its LiteLLM integration. Stage routing looks at recent tool activity and results in the request history and picks between two candidate models. Random routing chooses from the configured models, optionally weighted, for traffic splits and A/B comparisons. The post also notes that LiteLLM recently shipped a native Switchyard stage router as an auto-router strategy. Classifier-backed strategies such as Task and Composite routing need intermediate model calls, which this integration does not support.

The agent sends an OpenAI-compatible request with the model set to switchyard. LiteLLM and the plugin run in one container on Amazon ECS with AWS Fargate, in private subnets. An Application Load Balancer fronts it with AWS WAF rules, and it forwards only three paths: /v1/chat/completions, /v1/models and /health/liveliness. Other paths get a 404. Access is limited to an IPv4 CIDR range the deployer allows. LiteLLM calls Bedrock through the Converse or ConverseStream API using the task’s IAM role, via an interface endpoint in the VPC. The x-litellm-model-name response header shows which model answered, while the response body’s model field stays switchyard.

In the default Stage profile, the capable model is Claude Opus 5.5 and the efficient one is NVIDIA Nemotron 3 Super 120B. They must be listed in that order because Stage infers roles from position. The profile prefers the efficient model when history signals are inconclusive, so a new request with no tool results goes to Nemotron. Stage inspects the three most recent tool results. The sample also defines single-model baseline groups so the routed group can be compared against each model alone. A Random profile with 0.5/0.5 weights is provided for A/B tests.

Caveats the post states

  • The sample installs switchyard-server 0.3.0, which NVIDIA documents as a demo and evaluation component, and runs LiteLLM v1.102.0. The integration is experimental and may change.
  • It is sample code for non-production use, to be deployed in a non-production account.
  • The model pair was tested in US East (N. Virginia).
  • The litellm.yaml file is copied in plain text into the task definition and CloudFormation template, so credentials must not be placed in it.
  • The “global.” model prefix denotes a global cross-Region inference profile, which may process requests in supported commercial Regions worldwide. A geographic profile or in-Region model ID is the alternative if location matters.

Why it matters for agents

The post frames routing as a cost question that only makes sense at task level, which is a useful discipline for anyone running agent fleets. A cheaper model that needs more turns or fails more often may not save money, and the sample’s baseline groups are designed to test that on your own workloads. Because Stage reads tool results from history, agents must send full conversation history for it to work, a constraint that agent frameworks trimming context would need to respect. The experimental status, the demo-grade server component and the absence of benchmark figures in the post, which says a few sample runs “aren’t enough to draw conclusions”, mean the approach is best treated as an evaluation harness rather than a proven production setup.

Sources

This article was prepared with AI assistance and reviewed by an Emerge Times editor before publication. Licensed CC BY 4.0: credit “Emerge Times (emergetimes.com)”. Usage & License. Featured images are AI-generated illustrations; any logos shown are trademarks of their owners, and Emerge Times is not affiliated with them.