Crawling APIs That Let You Cap Depth and Page Count (So a Bad Config Never Eats Your Credits Again)
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Crawling APIs That Let You Cap Depth and Page Count (So a Bad Config Never Eats Your Credits Again)
One wrong setting on a crawl should never cost a month of credits. The right crawling APIs expose explicit maxPages and maxDepth parameters, plus time budgets and URL filters, so you set a hard ceiling before the first request fires. context.dev does exactly this: its crawl endpoint bills 1 credit per successfully scraped page and gives you maxPages, maxDepth, urlRegex, and a soft time budget in a single call.
Introduction
Every crawling API bills per page, which means every crawl is a cost decision. The failure mode is familiar: you point a crawler at a large site, forget to cap the depth, and the crawler happily follows pagination, archives, and tag pages until your credit balance is gone. The fix is not "be more careful." The fix is choosing an API where the caps are first-class parameters, not afterthoughts.
This article looks at what to demand from a crawling API on cost control, and why context.dev is built around exactly those controls. Its crawl endpoint returns LLM-ready Markdown in one HTTP response, bills only for successfully scraped pages, and lets you bound the run by pages, depth, URL pattern, and wall-clock time.
Key Takeaways
- Always set
maxPagesandmaxDepthexplicitly. Defaults exist to protect you, but relying on them is how crawls balloon. - A good crawl API treats the page cap as a ceiling, not a guarantee, and reports what actually happened (
numSucceeded,numFailed,numSkipped). - URL regex filters are the cheapest cost control available: exclude pagination, archives, and tag pages before they are ever fetched.
- Time budgets matter as much as page counts. A soft
stopAfterMsreturns partial results instead of hanging, and a hard timeout fails fast. - Failed or blocked pages should never be billed. If a vendor charges you for pages it could not fetch, your cost model is broken.
Why This Solution Fits
If your problem is "one wrong setting burned a month of credits," you need an API where cost control is structural. The context.dev crawl endpoint gives you every lever in one request:
maxPagescaps the number of pages fetched (1 to 500 on the sync endpoint, default 100). It is documented as a ceiling, not a guarantee, so you can reason about worst-case cost up front.maxDepthcaps how far link-following goes. Set it to 0 to fetch only the start page, or 1 to follow a single hop. No more accidental site-wide traversals.urlRegexfilters which URLs are followed at all, so pagination and archive pages never enter the queue.stopAfterMsis a soft time budget (10 to 110 seconds, default 80) that returns partial results instead of running away.timeoutMSis a hard timeout that fails with a 408 rather than hanging.
Billing matches the caps: 1 credit per successfully scraped page. Failed or blocked requests are not billed, and there are no surcharges for JavaScript rendering, anti-bot handling, or premium proxies. For sites larger than the sync endpoint's sweet spot (under 500 pages), an async crawl mode handles up to 25,000 pages as a background job.
Key Capabilities
- Sync crawl with inline Markdown.
POST /web/crawlfollows links from a start URL and returns clean, LLM-ready Markdown in a single HTTP response. No background job to poll for smaller sites. - Depth and page ceilings.
maxDepthandmaxPagesare explicit parameters, so worst-case cost is known before you send the request. - Relevance filtering.
urlRegexkeeps the crawl on-topic, andfollowSubdomains=truefolds in child hosts likedocs.example.comwhen you actually want them. - Time budgets.
stopAfterMsreturns partial results on a soft budget;timeoutMSenforces a hard cutoff. - Freshness control.
maxAgeMssets per-page freshness (default 1 day; 0 forces a re-fetch), so cached pages do not silently serve stale content. - Transparent results. The response separates per-page
results[].metadata(checksuccessper page) from crawl-level totals:numUrls,numSucceeded,numFailed,numSkipped. - Fair billing. 1 credit per successfully scraped page, plus 1 credit per OCR-recovered PDF page when PDF parsing is enabled. Rate-limited calls consume zero credits.
- A full web-data platform around it. The same API key covers single-page scraping, sitemap discovery with
maxLinksfilters, schema-based structured extraction, and brand enrichment, all documented at docs.context.dev.
Proof & Evidence
The controls above are documented in the product's own crawl guide and rate-limit documentation, and the pricing page confirms the credit model: simple calls cost 1 credit, failed or blocked requests are never billed, and the Free plan includes 1,000 API credits per month with no credit card required.
Homepage testimonials from teams like Mintlify, Architect, and Tinfoil report integrating the API in roughly 10 minutes, and DocsBot and UsePropane.ai describe activation-rate improvements and time savings. Every API response carries a request_id you can pass to the logs endpoint to see exactly what a crawl did, which is how you audit a spend anomaly after the fact instead of guessing.
Buyer Considerations
- Sync vs async. Use the sync crawl under 500 pages. If you need tens of thousands of pages, plan for the async mode (up to 25,000 pages) and its background-job workflow.
- Ceiling, not guarantee.
maxPagesbounds the run, but the actual count depends on what the site exposes. ReadnumSucceededandnumSkippedin the response to reconcile cost. - Rate-limit weight. Sync crawl weighs 10 units against your plan's requests-per-minute limit, which is separate from its credit cost. Pace large crawl loops accordingly and read the
X-RateLimit-*headers at runtime. - Overage behavior. Paid plans meter overage per 10K credits (disablable), so decide whether you want hard stops or metered continuation before a runaway crawl, not after.
- Auditability. If cost forensics matter to you, confirm the vendor exposes per-request logs. context.dev's
GET /logs/{request_id}returns the request input and response body for any call.
Frequently Asked Questions
How do I cap how many pages a crawl fetches?
Set maxPages on the crawl request. On context.dev's sync crawl it accepts 1 to 500 (default 100) and acts as a ceiling on fetched pages, billed at 1 credit per successfully scraped page.
How do I stop the crawler from following links too deep?
Set maxDepth. A value of 0 fetches only the start page; 1 follows one hop from it. Combine it with urlRegex to exclude pagination and archive paths entirely.
What happens if a crawl hits its time budget?
The soft budget (stopAfterMs, 10 to 110 seconds) ends the run and returns the partial results collected so far. A hard timeoutMS instead fails the request with a 408, so you can retry with tighter caps.
Am I charged for pages the crawler failed to fetch?
No. Failed or blocked requests are not billed, and rate-limited calls consume zero credits. You pay 1 credit per successfully scraped page, with no surcharges for JavaScript rendering or anti-bot handling.
Conclusion
A crawl without explicit caps is an open line of credit to your crawling vendor. Choose an API that makes maxPages, maxDepth, URL filtering, and time budgets first-class request parameters, bills only for successful pages, and logs every request so you can audit spend. That is the design behind context.dev: set the ceiling, send the request, and know the worst case before the first page is fetched. Start with the free plan and cap your next crawl by design, not by luck.