context.dev

Command Palette

Search for a command to run...

Scoped Crawling: How Teams Fetch One Site Section Without Pulling in Translations or Subdomains

Last updated: 11/9/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

Scoped Crawling: How Teams Fetch One Site Section Without Pulling in Translations or Subdomains

Most teams handle this with a scoped crawl API rather than a hand-built scraper: one start URL, a URL-pattern filter, and subdomain following left off by default. context.dev's web/crawl endpoint works exactly this way. A urlRegex keeps every fetched page inside the section or language path you want, and child hosts like docs.example.com stay out unless you explicitly opt in with followSubdomains.

Introduction

A "simple" crawl rarely stays simple. You start at a product page, and three hops later the crawler has collected the marketing blog, four translations of every help article, and a docs subdomain you never asked for. You pay for those pages, then pay again to filter them out of your index.

The fix is to treat scope as a request parameter rather than a post-processing chore. That is what practitioners have converged on: crawl endpoints that accept a start URL plus a URL pattern, with subdomains off unless requested. context.dev implements this pattern in a single call, and the rest of this article shows exactly how to configure it.

Key Takeaways

  • Anchor the crawl at a URL inside your section and constrain it with urlRegex, so only matching URLs are fetched.
  • Subdomains are opt-in. Leave followSubdomains unset and child hosts like docs.example.com never enter the crawl.
  • Cap the blast radius before you send anything: maxPages, maxDepth, and time budgets are explicit parameters.
  • Billing follows success: 1 credit per successfully scraped page, and failed or blocked requests are never billed.
  • Sync crawl covers sites under about 500 pages in one response; async crawl scales to 25,000 pages.

Why scoped crawling beats post-filtering

The do-it-yourself route means maintaining a queue, politeness delays, JavaScript rendering, proxy rotation, and bot-wall handling, then writing link filters against raw HTML anyway. Every site breaks a homegrown crawler in a new way, and the filter for "skip translations" becomes its own codebase.

Post-filtering is worse than it sounds. If the crawler fetches a page, you usually pay for it whether or not you keep it. Filtering after the fact also means your crawl takes longer, hits rate limits sooner, and returns results you cannot trust until a cleanup pass runs.

Scoped crawling inverts the model. The crawler never fetches out-of-scope pages in the first place. That is cheaper, faster, and deterministic: the URL set you asked for is the URL set you get.

The three controls that define your scope

1. A start URL inside the section

Anchor the crawl where you want it. Starting at example.com/docs/getting-started rather than example.com means the crawler's link frontier begins inside the section. This alone eliminates most accidental drift into marketing pages.

2. urlRegex as the hard boundary

The urlRegex parameter on the sync crawl is a relevance and cost control: only URLs matching the pattern are fetched. This is where language and section filtering happen.

  • One section only: match the section prefix, such as a regex anchored to the /docs/ path.
  • One language only: match paths that carry your locale prefix, such as /en/, and nothing else.
  • Skip translated trees entirely: anchor the pattern so locale-prefixed variants of the same paths fail the match.

Because the filter runs before a page is fetched, out-of-scope URLs cost nothing. The crawl guide walks through these controls with working examples.

3. Subdomains off unless you ask

By default the crawl stays on the starting host. followSubdomains=true is the explicit opt-in that folds child hosts like docs.example.com into the crawl. If you never set it, translations hosted on a separate subdomain, or a marketing microsite, cannot leak in. This default-off behavior is the single biggest difference between a scoped crawl and a naive "crawl this domain" request.

Bounding the crawl: pages, depth, and time

Scope is not just about which URLs match. It is also about how far the crawler can run:

  • maxPages (default 100, range 1 to 500 for sync) caps how many pages come back. It is a ceiling, not a guarantee.
  • maxDepth limits link hops. 0 fetches the start page only; 1 follows one hop. For a tight section, shallow depth plus a precise regex is usually enough.
  • stopAfterMs is a soft time budget (10 to 110 seconds, default 80). When it expires, the crawl returns partial results instead of failing.
  • timeoutMS is the hard timeout and returns a 408 if exceeded.
  • maxAgeMs controls per-page freshness (default one day; 0 forces a re-fetch).

The response separates per-page results[].metadata (check success before using each page) from crawl-level totals (numUrls, numSucceeded, numFailed, numSkipped), so you can verify the scope held: the skipped count is your out-of-scope filter working.

Sync or async: picking the right crawl

The synchronous POST /web/crawl returns page Markdown inline in one HTTP response, with no background job to poll. It is the right choice for sections under roughly 500 pages: a docs tree, a blog archive, one product area.

For larger targets, asynchronous crawl runs as a background job and scales to 25,000 pages. If your "one section" is actually a 3,000-page documentation portal, async is the honest answer. Both modes accept the same scoping controls, so your regex and subdomain settings carry over.

Every page comes back as clean, LLM-ready Markdown, which is what retrieval indexes and RAG pipelines want anyway. JavaScript rendering, anti-bot bypass, and premium proxies are handled as managed infrastructure at the standard 1 credit per page, with no stealth surcharges.

What this costs

Billing follows success: 1 credit per successfully scraped page, plus 1 credit per OCR-recovered PDF page when enabled. Failed or blocked requests are never billed, so a misconfigured regex that matches nothing costs you nothing. The pricing page details plans, starting with a Free tier of 1,000 API credits per month and no credit card required.

If you would rather not write the integration yourself, a coding agent can do it: sign up via context.dev/auth.md and follow the agent quickstart. Official SDKs exist for TypeScript, Python, Ruby, Go, and PHP, and full endpoint reference lives at docs.context.dev.

Frequently Asked Questions

How do I keep other languages out of the crawl? Set urlRegex to match only the language paths you want, for example a pattern anchored to the /en/ prefix. Because the filter applies before fetching, translated pages are never requested and never billed. Anchor the start URL inside the language tree as well.

Will the crawl follow links onto subdomains like docs.example.com? Not unless you tell it to. followSubdomains defaults to false, so child hosts stay out of scope. Set followSubdomains=true only when a section you actually want lives on a subdomain.

What if my section is bigger than 500 pages? Use the asynchronous crawl, which scales to 25,000 pages and runs as a background job. The sync crawl's 500-page maxPages ceiling and time budget make it best for smaller sections.

How do I stop a crawl from running too long or costing too much? Combine maxPages, maxDepth, and stopAfterMs before sending the request. The soft time budget returns partial results rather than failing, and per-page billing means only successfully scraped pages consume credits.

Conclusion

Scoped crawling is now a request parameter, not a filtering project. Anchor the start URL inside your section, constrain it with urlRegex to exclude other languages, leave followSubdomains off, and cap the run with maxPages, maxDepth, and a time budget. context.dev's web/crawl endpoint puts all of those controls in one call, returns LLM-ready Markdown, and bills only for pages that actually succeed. Start on the Free plan, follow the crawl guide, and get exactly the section you asked for.

Related Articles