AWS adds traffic controls for Bedrock AgentCore Gateway
AWS has announced rate limiting support for Amazon Bedrock AgentCore Gateway, adding a new traffic control layer for AI applications that route calls to tools, models and agents. The feature lets operators set request, token and connection limits by user identity, target, model or tool. For teams running agentic systems, the change matters because a single agent call can trigger multiple downstream requests and consume shared model capacity quickly.What AWS added to AgentCore Gateway
Amazon Bedrock AgentCore Gateway now supports customer-defined rate limits for AI traffic, according to AWS. The gateway is positioned as a managed entry point for traffic to resources such as managed web search, knowledge bases, MCP servers, inference models, agents and HTTP endpoints.The new controls cover three traffic measures. Request limits are measured in requests per second or requests per minute and apply across target types. Token limits are measured in tokens per minute and apply to inference targets, with both input and output tokens counted. Connection limits are measured in connections per second and track long-lived sessions, which is relevant for streaming inference calls that hold a connection open while work continues.
The practical implication is that AI gateway policy can move closer to the traffic pattern of modern applications. A short request and a 90-second streaming request are both one request for request-rate accounting, but the latter holds a connection slot for longer. Token accounting adds another control plane for model use, where cost and capacity are often tied to token volume rather than request count alone.
How limits are scoped across users, targets and models
AWS describes the feature as a dimension-based system. Dimension keys define how incoming traffic is grouped into buckets, while entries define the allowed throughput for each bucket. Supported dimensions include targetName, toolName, qualifiedModelId, JWT claims, IAM principal and IAM source identity.This structure lets teams choose between broad and granular enforcement. A target-level rule can protect a busy MCP server or inference target. A user-level rule can use a JWT claim or IAM identity to prevent one caller from consuming a shared allowance. A combined rule can scope limits by both model and user group, making it possible to assign different token budgets to different roles for the same model.
AWS gives examples using role-based groups such as Basic, Advanced and Beta. In one pattern, group-level buckets set a shared ceiling for a role, while individual-user buckets stop one member of that group from exhausting the full group allowance. AWS says these limits are evaluated independently with AND semantics, meaning a request must pass every relevant limit before it proceeds.
The design also uses specificity. Named entries can take precedence over wildcard entries, and more detailed dimension combinations can be used when a shared target-level limit would be too blunt. For production operators, this reduces the trade-off between protecting a backend service and allowing a controlled subset of users to run heavier evaluation workloads.
Why agentic workloads need separate controls
Agentic workloads create a second rate-limit problem because one user invocation can cause an agent to call downstream tools and models. AWS uses an AWS Documentation Agent example in which an agent calls a Docs MCP target for retrieval and a BedrockMantle inference target for reasoning through a model.AWS separates this into two control layers. The first layer limits how frequently users or services can invoke the agent itself, using request and connection limits on the agent target. The second layer limits the downstream resources consumed by the agent after invocation.
Identity handling determines which limits apply downstream. If an agent uses an on-behalf-of token exchange based on the original user's JWT token, existing user-based limits can apply to downstream calls. If it uses a machine-to-machine grant, downstream requests carry the agent's identity rather than the caller's claims. In that case, AWS recommends adding limits that identify the agent itself, such as through an authorized-party claim, so that one agent cannot exhaust shared resources unnoticed.
Operational cautions for production teams
AWS frames rate limits as traffic management, not as a standalone security boundary. The blog says AgentCore Gateway uses fail-open semantics for rate-limit evaluation, so organizations should also rely on authentication, authorization and AWS WAF rules for security enforcement.There are several configuration details with operational consequences. AWS says customer-defined rate limits are evaluated before AgentCore Policy authorization. That means a request later denied by policy can still consume a rate-limit bucket unless a rate-limit entry explicitly assigns a zero rate to users or groups that should be blocked at the rate-limit layer.
The announcement also warns against high-cardinality or unbounded JWT claims, such as request IDs, nonce values or token IDs, because they can create too many buckets and weaken enforcement. AWS recommends stable identifiers such as subject, role, team or tier. It also advises adding catch-all entries, because a caller that matches no entry can fall through to service-managed quotas without customer-defined enforcement.
For observability, AWS says AgentCore Gateway can emit rate-limiting information through application logs using OpenTelemetry span attributes. That gives operators a way to investigate throttling decisions, bucket consumption and limit behavior before a capacity issue becomes visible to users.
Conclusion
The AgentCore Gateway update gives AWS customers a more detailed way to govern AI traffic as applications move from single model calls toward multi-step agentic workflows. The most useful part is not only the existence of throttling, but the ability to combine identity, target, tool and model dimensions.The feature still requires careful design. Shared buckets can starve peers, missing wildcard entries can leave gaps, and fail-open behavior means rate limits should sit beside authorization and security controls rather than replace them. Used carefully, the new controls can help teams preserve backend availability while assigning AI capacity more deliberately across users, agents and models.
Sources
Editorial Team - CoinBotLab
