# Scale your AI system for real traffic

When traffic grows, an AI system fails in three predictable ways: it hits a
provider rate limit, it piles up work faster than it can finish, and individual
calls get slow enough to time out. The fixes are four patterns you can layer in
order: batch what is not urgent, queue what is, cache what repeats, and degrade
gracefully when you still run out of room.

This guide is about the system around the model, not the model call itself. It
assumes you already have a working feature, like the summariser in an online
store, and now ten times the users want it at once.

## What breaks when an AI system gets real traffic?

A single request looks instant in a demo. Multiply it by a thousand concurrent
users and three limits show up at the edges.

- **Rate limits.** Providers cap how fast you can call them. The Claude API
  enforces three separate ceilings per model: requests per minute (RPM),
  input tokens per minute (ITPM), and output tokens per minute (OTPM).
  Cross any one of them and you get an HTTP `429` response.
- **Backpressure.** If work arrives faster than you finish it, something has to
  hold the overflow. With no queue, the overflow lands on the user as a timeout
  or a crash.
- **Slow calls.** A long generation can take many seconds. Hold an HTTP
  connection open the whole time and your server runs out of connections long
  before it runs out of CPU.

The rest of this guide is the four patterns that turn that failure picture into a
system that bends instead of breaking.

## Which rate limit am I hitting?

You cannot fix a limit you have not identified. The `429` response tells you
which one tripped, and the response carries a `retry-after` header with the
number of seconds to wait before trying again. Honour that header rather than
guessing.

The Claude API replenishes capacity with a token bucket algorithm. Your
allowance refills continuously up to your maximum, rather than resetting on a
fixed clock. The practical effect: a 60 RPM limit behaves closer to 1 request
per second, so a sudden burst can trip the limit even when your per-minute
average looks fine. Smooth your traffic and you get more headroom for free.

One detail changes how you design everything downstream: on most Claude models,
tokens read from the cache do not count toward your ITPM limit. So caching is not
only a cost lever, it is a throughput lever. More on that below, and in the
dedicated [prompt caching](/guides/prompt-caching) guide.

| Pattern | What it solves | Cost to you |
| --- | --- | --- |
| Batching | Throughput on non-urgent work | Latency (results later) |
| Queuing | Bursts and backpressure | Complexity (a worker tier) |
| Caching | Repeated work, rate-limit pressure | Staleness risk |
| Graceful degradation | The case where you still run out | A weaker but valid answer |

## Pattern 1: batch the work that can wait

Not all work is interactive. Re-summarising every product description overnight,
scoring a backlog of support tickets, or running an evaluation set are all jobs
where the answer is needed eventually, not now. Batching trades latency for
throughput and cost.

The Claude API offers a Message Batches API for exactly this. You submit many
requests as one asynchronous batch and poll for results. Batch requests run at
50% of the standard cost. Most batches finish in under an hour, and a batch has a
24-hour processing window: you can collect results when every request completes
or after 24 hours, whichever comes first.

The rule of thumb: if no human is waiting on the result this second, batch it.
You take pressure off your per-minute rate limits and cut the bill in half. Drive
this from your own backend on a schedule. You can ask Claude Code to scaffold the
submit-and-poll job for you.

```text
Set up a nightly batch job for our support-ticket summarizer.

Context:
- tickets land in a table, status new, we summarize each one
- summaries are read the next morning, not in real time

Do this:
- plan a job that collects pending tickets, submits them as one async batch, and stores results
- handle the case where the batch is not finished yet on the next poll
- do not write the model-calling code, just the job structure and storage
- show me the plan before writing anything
```

## Pattern 2: queue the work that cannot

Interactive work cannot wait until tomorrow, but it can wait a few seconds. A
queue is the buffer that absorbs a burst so your model calls go out at a steady
rate instead of all at once.

The shape is always the same. Your app accepts the request, drops a job on a
queue, and returns immediately with a job id. A separate pool of workers pulls
jobs at a controlled rate and calls the model. The user polls the job id or gets
notified when it is done.

A queue buys you four things at once:

- **Backpressure handling.** The queue holds the overflow instead of your web
  server. Bursts flatten into a steady stream.
- **Concurrency control.** You tune how many workers run, which sets how fast you
  call the model. Keep that rate just under your RPM and token limits.Sizing workers against RPM alone is a classic trap: a few workers sending long prompts can blow past your ITPM token ceiling while sitting well under the request cap, and you get 429'd anyway. Cap concurrency against whichever limit you hit first, which for big prompts is usually tokens, not requests.
- **Retries with backoff.** When a call returns `429` or fails, the worker waits
  and [retries instead of dropping the job](/guides/handle-ai-failures-gracefully). Exponential backoff means each
  retry waits longer than the last, with a little random jitter so all your
  workers do not retry in lockstep.
- **Decoupling.** Slow model calls no longer hold an HTTP connection open. The
  user got their job id in milliseconds.

When a worker hits a `429`, the correct move is to read the `retry-after` header,
sleep for that long, and put the job back. That single behaviour is the difference
between a queue that recovers from a spike and one that melts down trying to push
through it.A fixed backoff with no jitter makes a fleet worse, not better: every worker trips the same 429 at once, sleeps the same amount, and retries in lockstep, so the burst that broke you reforms a few seconds later. Add a small random jitter to each wait so workers spread out, and back off per worker rather than pausing the whole pool.

## Pattern 3: cache what repeats

The fastest call is the one you never make. Two kinds of caching matter, and they
solve different problems.

**Response caching.** If users ask the same thing, store the answer keyed on the
input and serve it on a repeat. A product FAQ or a "summarise this fixed
document" feature can have a high hit rate. This removes the call entirely, so it
saves latency, cost, and rate-limit budget all at once. Set a sensible
expiry so answers do not go stale, and only cache work that is
deterministic enough to reuse.

**Prompt caching.** Most calls are not identical, but they share a large, stable
prefix: a long system prompt, a big reference document, your tool definitions.
Prompt caching stores that prefix on the provider side so repeat calls skip
reprocessing it. As noted above, on most Claude models cached input tokens do not
count toward your ITPM limit, so a high cache hit rate raises your effective
throughput well past the headline number. The full mechanics, breakpoints, and
TTLs are in [prompt caching](/guides/prompt-caching).

Order your prompt so the stable parts come first and the variable parts last.
That maximises how much of each request hits the cache. The same ordering
discipline shows up in [context engineering](/guides/context-engineering-for-ai-agents),
and it pays off twice here.

## Pattern 4: degrade gracefully

You will still hit moments where every lever is pulled and demand exceeds
capacity. At that point, aim to degrade gracefully: never
hand the user a blank error.

Graceful degradation means having a [defined fallback](/guides/guardrails-for-production-ai) for each failure, ranked
from best to worst:

- **Smaller model.** If the primary model is saturated or slow, route to a
  faster, cheaper one. A slightly weaker answer beats no answer.
- **Cached or partial result.** Serve a recent cached answer, or the part of the
  work that did finish, with a note that it is partial.
- **Non-AI fallback.** Fall back to keyword search, a templated response, or a
  static help page when the AI path is fully unavailable.
- **Honest queueing.** As a last resort, accept the job, tell the user it is in
  progress, and notify them when it is ready.

Two guardrails make degradation safe. Set a **timeout** on every model call so a
slow response cannot pin a worker forever, and trip the fallback when the timeout
fires. Add a circuit breaker so that once a dependency is clearly
failing, you stop hammering it for a short cooldown and serve fallbacks instead,
then test with a trickle of traffic before reopening. A circuit breaker turns a
slow cascade into a fast, bounded failure.

## How do the four patterns fit together?

The four patterns stack. Each one shrinks the load that reaches the next.

Read it left to right. Batching removes non-urgent work from the live path.
Caching removes repeat work. Queuing smooths what remains to a rate your limits
allow. Graceful degradation catches whatever still overflows. Most systems do not
need all four on day one, so add them in that order as load grows.

## What should I check before scaling to production?

Run this before you call a system production-ready under load.

- [ ] You handle `429` by reading `retry-after` and backing off, not by retrying
  immediately.
- [ ] Non-urgent jobs go through a batch path, not the live request path.
- [ ] A queue sits between intake and your model calls, with worker concurrency
  tuned under your rate limits.
- [ ] Repeated inputs are response-cached, and long stable prefixes use prompt
  caching.
- [ ] Every model call has a timeout and a defined fallback.
- [ ] You [log enough](/guides/observability-for-ai-systems) to see which limit you hit and how often.

You do not have to write the plumbing by hand. Describe your traffic shape and
your limits to Claude Code and have it scaffold the queue, the worker loop, and
the retry logic, then review the result against this checklist. The same
discipline that makes a single agent reliable, covered in
[test and debug your agent](/guides/test-and-debug-your-agent), is what keeps a
fleet of them responsive once real traffic arrives.