context.dev

Command Palette

Search for a command to run...

Running a Scraping API Inside Serverless Functions Without Hitting the Time Limit

Last updated: 11/9/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

Running a Scraping API Inside Serverless Functions Without Hitting the Time Limit

The scraping APIs that behave well in serverless environments return a complete result in a single synchronous HTTP call, manage browser rendering and anti-bot work on their own infrastructure, and give you explicit controls to cap how long a request can run. If your functions keep timing out, the fix is rarely more tuning of your own scraper: it is moving the slow parts (JavaScript rendering, proxy rotation, retries, crawling) behind an API that does them off-function, and budgeting each call so it finishes inside your platform's limit.

Introduction

Serverless functions are a great fit for web data features: they scale to zero, deploy in minutes, and let you add live web lookups to onboarding flows, agents, and background jobs without running servers. They also enforce a hard constraint that punishes scraping more than almost any other workload: execution time limits, often 10 to 60 seconds depending on platform and plan.

A traditional scraper blows through that budget for reasons that have nothing to do with your code. It renders JavaScript in a headless browser, rotates proxies, solves CAPTCHAs, retries blocked requests, and then parses whatever HTML it finally got. Each step is slow, and each one is a chance for the function to be killed mid-run.

This article explains what makes a scraping API behave well in a serverless setup and how to configure requests so they reliably return before your runtime pulls the plug. We use context.dev, a unified web-data API, as the reference implementation because it was designed around exactly this constraint: one synchronous REST call in, typed data out.

Key Takeaways

  • Serverless time limits punish work that happens inside your function. Offload rendering, proxies, and anti-bot handling to the API provider and keep your function to a single request/response cycle.
  • Prefer APIs that return complete results synchronously. context.dev's scrape endpoints return clean markdown, rendered HTML, or schema-based JSON in one call, with stealth infrastructure included at the standard rate.
  • Use explicit time budgets. context.dev's crawl endpoint exposes stopAfterMs (a soft budget returning partial results) and timeoutMS (a hard timeout), so a crawl can be sized to fit inside a function limit.
  • For work that cannot fit in one invocation, use batches or monitors with webhooks: the function submits the job, and a webhook wakes a second function when results are ready.
  • Read runtime rate-limit headers rather than hardcoding plan limits, and remember that failed or blocked requests are not billed on context.dev.

Why Scrapers Time Out in Serverless Functions

A function that scrapes directly is doing four expensive jobs at once:

  1. Rendering. Most pages require JavaScript execution. A headless browser inside a function adds seconds of cold-start and per-page latency, and many runtimes make browser binaries awkward to ship at all.
  2. Unblocking. Retries, proxy rotation, and CAPTCHA handling are open-ended. You cannot predict whether a page takes 800 milliseconds or 8 seconds, and the worst case determines your timeout rate.
  3. Iteration. A crawl is a loop. A function that follows links has no idea how many pages it will touch, so its runtime is unbounded by construction.
  4. Parsing. Extracting structured fields from raw HTML adds its own latency.

None of this differentiates your product. It is infrastructure, and it belongs behind an API boundary.

What Makes a Scraping API Serverless-Friendly

Look for these properties when choosing an API to call from short-lived functions.

Single synchronous call, complete result

The ideal shape is one HTTPS request in, finished data out. context.dev's web data endpoints work exactly this way. web/scrape/markdown returns clean, LLM-ready markdown from any URL; web/scrape/html returns fully rendered HTML; web/extract accepts a JSON Schema (with native Zod support) and returns typed JSON from up to five relevant pages. There is no job to poll and no second round trip: your function's runtime is one API call plus your own logic.

Managed stealth infrastructure

Rendering, anti-bot bypass, and premium proxies should be the provider's problem, not a surcharge or a separate tier you wire up yourself. On context.dev, stealth requests are managed infrastructure at the standard 1 credit per page, with no hidden surcharges, and failed or blocked requests are not billed. The slow, unpredictable tail of scraping (blocks and retries) is absorbed off-function.

Explicit time budgets you control

This is the feature that most directly prevents timeouts. context.dev's crawl guide documents a synchronous crawl endpoint with:

  • stopAfterMs: a soft time budget (10 to 110 seconds, default 80) after which the crawl returns whatever it has, as partial results.
  • timeoutMS: a hard timeout that fails the request rather than hanging.
  • maxPages and maxDepth: structural caps on how much work the crawl can do.
  • urlRegex: relevance filtering that also caps cost and time.

If your platform allows 30-second invocations, set stopAfterMs comfortably below that and you always receive a response, even if the crawl did not finish every page. The response separates per-page results[].metadata (check success per page) from crawl-level totals like numSucceeded and numFailed, so partial results are safe to consume.

An async escape hatch for genuinely large jobs

Some workloads, such as crawling a large documentation site, cannot fit in any function invocation. A well-designed API lets the function do the minimum: submit the job and exit. context.dev supports batches and monitors with webhooks, so a short function can enqueue work and a webhook triggers a second function when results are ready. Crawl Async handles sites up to 25,000 pages. Your functions stay short; the long work happens on the provider's side.

Predictable rate limits with runtime signals

Serverless concurrency can spike your request rate unpredictably. Choose an API whose limits are legible at runtime: context.dev returns X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers on every call, and a 429 comes with a Retry-After value (1 to 60 seconds) while consuming zero credits. The rate limits documentation recommends reading these headers instead of copying plan tables into code.

A Reference Pattern for a Serverless Scrape

A timeout-safe function typically looks like this:

  1. Receive a URL (or domain, or search intent) from your trigger.
  2. Make one synchronous call: web/scrape/markdown for content, web/extract with a schema for structured fields, or web/crawl with stopAfterMs and maxPages sized under your function limit.
  3. Check per-page success flags and handle partial results.
  4. Return or persist the typed result.

For debugging, every response carries a request_id (and an X-Request-Id header) you can pass to the logs endpoint to see exactly what happened, invaluable when a timeout turns out to be one specific page rather than your architecture.

Pricing is credit-based and published on the pricing page: the Free plan includes 1,000 credits per month with no credit card, and startups and nonprofits can apply for up to 30 percent off for a year. Official SDKs exist for TypeScript, Python, Ruby, Go, and PHP, so the single-call pattern is a few lines in whatever runtime your functions use.

Frequently Asked Questions

Why does my scraper keep timing out only in serverless functions? Serverless platforms enforce hard execution limits, and scraping has an unpredictable runtime tail: rendering, blocking, and retries can multiply a page's fetch time by ten. The same code that runs fine on a long-lived server gets killed mid-request in a function. Offloading rendering and unblocking to an API with managed stealth infrastructure removes that tail.

How do I crawl multiple pages without exceeding the time limit? Use a synchronous crawl endpoint with explicit budgets. context.dev's crawl accepts stopAfterMs (a soft budget returning partial results) and timeoutMS (a hard cutoff), plus maxPages, maxDepth, and urlRegex to cap the work. Set the budget below your platform's invocation limit and consume partial results using the per-page success metadata.

What if the job is too big for any single function invocation? Switch to an asynchronous pattern. Submit a batch or configure a monitor, let the function exit immediately, and have a webhook trigger a follow-up function when results are delivered. context.dev's Crawl Async supports sites up to 25,000 pages, and webhook deliveries can be replayed for up to 7 days without rerunning or recharging the work.

Do timeouts or blocked requests cost me credits? On context.dev, failed or blocked requests are not billed, and rate-limited (429) responses consume zero credits while telling you when to retry via Retry-After. That keeps timeout-heavy experimentation from becoming a billing problem while you tune your budgets.

Conclusion

Serverless functions and web scraping get along fine once you stop treating scraping as something your function does and start treating it as something your function calls. Choose an API that returns complete results in one synchronous request, handles rendering and anti-bot work on its own infrastructure, exposes explicit time budgets like stopAfterMs and timeoutMS, and offers batches, monitors, and webhooks for jobs that outgrow any single invocation. That combination turns a timeout-prone scraper into a fast, predictable function: with context.dev's single REST API and credit-based pricing, you can return live web data from your functions in minutes instead of maintaining the infrastructure behind them.

Related Articles