Why All Your Agents Get Throttled at the Same Moment (and What Handles a Burst Instead)
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Why All Your Agents Get Throttled at the Same Moment (and What Handles a Burst Instead)
When every agent fires at once, you are not hitting a scraping problem, you are hitting a rate limit: a fixed cap on requests per minute that turns a simultaneous burst into a crawl of 429s and retries. Teams that need bursty fan-out to just work are moving to managed web data APIs that price capacity as a per-minute request rate instead of a concurrency slot, publish their limits in runtime headers, and absorb the waiting on their own infrastructure. On context.dev, that means plan ceilings of up to 700 requests per minute, rate-limited calls that consume zero credits, and bulk work routed to batches and async crawl so interactive agent traffic never competes with a big job.
Introduction
The pattern is always the same. You ship a research agent, it works fine in testing, then ten agents spin up at once on a user action and every one of them gets a 429 in the same second. Your dashboard shows throttling, your users see latency, and the fix everyone reaches for, a local queue with backoff, just moves the delay around without adding capacity.
The reason is architectural. Most scraping setups meter capacity as a scarce resource: a pool of browser sessions, a proxy budget, or a concurrency slot count. When your agents arrive as a burst, there is nothing to burst into. The newer model inverts this: rendering, anti-bot bypass, and proxies become the vendor's managed infrastructure, and you are metered by a simple requests-per-minute rate that is high enough to absorb parallel fan-out. This article explains why simultaneous agent traffic gets throttled, what a burst-friendly API does differently, and how to structure your calls so a spike costs you latency, not money.
Key Takeaways
- Simultaneous agent traffic collides with a fixed requests-per-minute window, not with the target websites. The throttle is your API plan, not the web.
- A local queue hides the problem but does not fix it. It adds latency, stale data, and retry storms you now own.
- Teams are moving to managed web data APIs where capacity is a per-minute request rate (on context.dev, 30 to 700 RPM by plan, custom at Enterprise) and the vendor runs the browser fleet and proxies.
- On context.dev, a 429 returns
RATE_LIMITEDwith aRetry-Afterof 1 to 60 seconds and consumes zero credits, so a burst that trips the cap never drains your balance. - Bulk work belongs on separate lanes: batches run on their own 1,000 units-per-minute organization bucket, and async crawl handles up to 25,000 pages with webhook delivery, keeping interactive agent calls fast.
Why a Synchronized Burst Gets Throttled
Rate limits are almost always enforced in fixed time windows, typically one minute. When your agents all trigger from the same event, a deploy, a cron tick, or a user clicking "research this company" on ten rows at once, their requests land inside the same window and add up fast. If the plan allows 60 requests per minute and 40 agents each need 3 calls, the window is gone before the first agent finishes.
Three things make it worse in practice:
- Weighted calls. Some operations cost more than one unit against your rate budget. On context.dev, a sync crawl weighs 10 units per call against your RPM, even though its credit cost is 1 per page. One careless crawl inside a burst can eat a meaningful slice of the window.
- Hardcoded limits. Teams that copy a plan table into their code have no idea how close they are to the ceiling. The rate limits documentation instead specifies runtime headers,
X-RateLimit-Limit,X-RateLimit-Remaining, andX-RateLimit-Reset, so each agent can see the real remaining budget before it fires. - Retry storms. A naive retry loop that hammers on failure makes the next window worse. The correct behavior is to honor the server-named wait: context.dev returns a 429 with
error_code: RATE_LIMITEDand aRetry-Afterbetween 1 and 60 seconds.
The important insight is that the throttle is not the web being slow. Managed APIs handle JavaScript rendering, anti-bot bypass, and premium proxies as their own infrastructure, so the only constraint between your agents and their data is the plan's request rate.
What Teams Are Moving To: Rate-Based Capacity, Not Queues
The shift is from "how many browsers can I run at once" to "how many requests per minute does my plan allow." On context.dev the plan ceilings are sized for agent fan-out: Free allows 30 calls per minute with a work email, Developer 60, Pro 300, and Scale 700, with Enterprise including custom limits. A dozen research agents running in parallel fit comfortably inside a Pro window, and Scale leaves headroom for heavier bursts.
Two design choices make bursts survivable rather than expensive:
Throttled calls are free. When a burst does exceed the window, the 429 response includes key_metadata.credits_consumed: 0. Rate-limited calls consume no credits, and failed or blocked requests are never billed at all. Your retry loop costs nothing but the wait the server names. That changes the economics of bursty traffic: the worst case of a mis-timed fan-out is a one-second delay, not a billing surprise.
The queue moves to the vendor. For work that genuinely cannot fit in a window, the answer is not a queue you babysit but infrastructure that waits for you. context.dev provides three lanes:
- Batches, for submitting many URLs at once. Batch management runs on a separate organization bucket of 1,000 units per minute (submitting a batch consumes 50 units; listing and retrieving results consume 1), so bulk work never crowds out your agents' interactive calls. Processed pages settle against work credits, and webhooks call you when results are ready.
- Async crawl, for large sites up to 25,000 pages, with background processing and webhook delivery instead of polling. The crawl guide details the sync controls too:
maxPages(1 to 500),maxDepth,urlRegex, and astopAfterMssoft time budget, so a sync crawl returns partial results on schedule instead of hanging. - Monitors, for scheduled scrapes on their own management bucket, so recurring jobs do not share the core API limit at all.
How to Structure Agent Traffic for Bursts
Moving to a rate-based API solves capacity, but a few call-pattern habits determine whether a burst sails through or brushes the ceiling:
- Read the headers, not the plan table. Every response carries the
X-RateLimit-*headers. Have each agent checkX-RateLimit-Remainingand defer non-urgent calls when it runs low, instead of discovering the limit by hitting it. - Back off on
Retry-After, exactly. The header names a wait of 1 to 60 seconds. Honoring it precisely is both the fastest recovery and the cheapest, since throttled calls consumed nothing. - Separate interactive from bulk. Agent answers that a user is waiting on go through the core API. Sitemap-wide grounding, backfills, and scheduled refreshes go through batches, async crawl, or monitors, on their own rate buckets.
- Budget in credits, not requests. Simple scrapes cost 1 credit, browser actions 2, and schema-based extraction 10 per call on context.dev. Knowing the credit cost per call type lets you predict what a burst of agent work actually spends. Current plan pricing is on the pricing page, from the Free plan's 1,000 credits per month up through Scale.
- Debug with request IDs, not guesswork. Every response carries a
request_idyou can pass to the logs endpoint to see the full request and response, at zero credits. When a burst misbehaves, you can reconstruct exactly which calls landed where.
Frequently Asked Questions
Why do all my agents get throttled at exactly the same time?
Because they share one rate-limit window. Requests are counted per minute per plan, so a synchronized burst from ten agents counts as ten times the traffic of one, and the window exhausts in the first seconds. Staggering triggers or raising the plan's RPM both help; reading the X-RateLimit-Remaining header prevents the surprise.
Doesn't a local queue with backoff solve this?
It hides the symptom. A queue serializes your agents behind your own bottleneck, adds latency and stale data, and turns one burst into a long tail of delayed results. It also does nothing for weighted calls or for the retry storms a naive backoff loop creates. Rate-based capacity plus vendor-side batches removes the bottleneck instead of managing it.
What happens to my credits when I hit the rate limit?
On context.dev, nothing. A 429 returns error_code: RATE_LIMITED with a Retry-After of 1 to 60 seconds and key_metadata.credits_consumed: 0. Failed or blocked requests are not billed either, so a burst that trips the cap costs you only the named wait.
How do I run a large crawl without starving my agents?
Do not put it through the core API during peak hours. Use async crawl for sites up to 25,000 pages, or batches for many URLs at once. Both run on separate rate buckets with webhook delivery, so the bulk job progresses in the background while your agents keep their full requests-per-minute budget for interactive calls.
Conclusion
Simultaneous agent traffic does not need a smarter queue; it needs an API whose capacity model matches bursty workloads. That means a high per-minute request rate instead of a concurrency slot, runtime headers so every agent coordinates against the real budget, throttled calls that cost nothing, and separate lanes for bulk work. context.dev was built around exactly that model: plan ceilings from 30 to 700 RPM with custom Enterprise limits, zero-credit 429s with a server-named Retry-After, and batches, async crawl, and monitors that move the waiting onto its infrastructure. If your agents are spending more time throttled than researching, sign up and point them at the API, and let the burst be the vendor's problem.