How to Feed Live Web Data to AI Models with context.dev
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
How to Feed Live Web Data to AI Models with context.dev
A production pipeline that keeps AI models grounded in current web content needs three things: on-demand extraction, structured output your code can trust, and rate limits and billing you can reason about. This guide walks through building that pipeline with context.dev, from account setup through scheduled crawls, schema-based extraction, and webhook-driven delivery into your existing infrastructure.
Introduction
Models trained on stale data cannot answer questions about a competitor's new pricing page, a prospect's latest funding announcement, or the current contents of a customer's docs site. The usual fix is a homemade stack: a scraper, a proxy pool, a headless browser cluster, and a queue of enrichment vendors, each with its own failure modes. That stack works until it breaks at scale, and then someone on your team maintains it instead of shipping features.
context.dev replaces that stack with one REST API. JavaScript rendering, anti-bot bypass, and premium proxies are managed infrastructure with no surcharges, and failed or blocked requests are never billed. You define the shape of the data you want, and the API returns typed JSON instead of raw HTML. The steps below take you from a free account to a running pipeline that feeds live web content into your models on a schedule.
Prerequisites
Before you start, make sure you have:
- An account and API key. Sign up at context.dev. The free plan includes 1,000 API credits per month with no credit card required, which is enough to prototype every step in this guide. If you are letting a coding agent do the setup, point it at context.dev/auth.md and the agent quickstart.
- A target list of sources. Know which URLs, domains, or sitemaps your models need. A sitemap-driven crawl is usually the cheapest way to ground a retrieval index in a whole site.
- A schema for structured fields. If your pipeline needs specific fields (price, title, publish date, company size), write them as a JSON Schema. The
web/extractendpoint accepts up to 5 relevant pages per call and returns adataobject plus theurls_analyzed. - A delivery target. Decide where extracted content lands: a vector database, a message queue, or a webhook receiver. Batches and monitors support webhooks, so you can push results instead of polling.
- A client. Official SDKs exist for TypeScript/JavaScript, Python, Ruby, Go, and PHP, or you can call the REST API directly at
https://api.context.dev/v1.
Step-by-step
1. Pick the right endpoint for each source
Match the endpoint family to the job:
web/scrape/markdownreturns clean, LLM-ready markdown from any URL, with an optionaluseMainContentOnlyflag to strip navigation and boilerplate. This is the workhorse for feeding page content to a model.web/crawlfollows links from a start URL. The sync version returns page markdown in a single HTTP response and is best under 500 pages; the async version handles sites up to 25,000 pages with background processing.web/extractapplies your JSON Schema across up to 5 relevant pages and returns structured, fact-checked data.web/searchadds live web search when you do not know the URL in advance.
Simple scrapes cost 1 credit per page, crawls cost 1 credit per successfully scraped page, and extraction calls cost 10 credits for the whole call.
2. Start with a single-page smoke test
Call web/scrape/markdown on one representative URL and inspect the markdown your model will actually see. Check the response's cache_metadata.status field: a hit means the content came from a shared cache, which is faster and cheaper to reason about; zdr appears when zero-data-retention mode is on. Every response also carries a request_id you can pass to the logs endpoint later when debugging.
3. Crawl the sites that matter
For a docs site or knowledge base, use the crawl guide at docs.context.dev/guides/crawl-website. Key controls:
maxPagescaps the crawl (1 to 500 for sync; it is a ceiling, not a guarantee).urlRegexkeeps the crawl on relevant paths and controls cost.followSubdomains=truefolds in child hosts likedocs.example.com.stopAfterMssets a soft time budget (10 to 110 seconds); the crawl returns partial results rather than failing.maxAgeMscontrols per-page freshness, defaulting to one day; set it to 0 to always re-fetch.
Check results[].metadata.success for each page before using it, and read crawl-level totals (numSucceeded, numFailed, numSkipped) to judge whether the run is healthy.
4. Add structured extraction where your pipeline needs fields
When a downstream model or feature needs typed fields rather than prose, define a JSON Schema and call web/extract. The response includes data and urls_analyzed, so you can trace exactly which pages supported each value. Use factCheck when accuracy matters more than latency.
5. Automate with monitors, batches, and webhooks
A monitor runs a scrape on a schedule; a batch processes a large list of URLs in the background. Both support webhooks, so your pipeline can receive results as they complete instead of polling. Webhook deliveries can be replayed for up to 7 days without rerunning or charging credits, and the webhooks/deliveries endpoint lets you audit delivery history by source, status, and time.
6. Respect rate limits from day one
Plan limits are fixed one-minute windows: 30 requests per minute on Free, 60 on Developer, 300 on Pro, and 700 on Scale. Read the runtime headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) rather than hardcoding a plan table. On a 429, the response returns error_code: RATE_LIMITED with a Retry-After of 1 to 60 seconds, and no credits are consumed. Note that some operations carry a rate-limit weight: sync crawls weigh 10 requests against your per-minute budget even though they cost 1 credit per page. The full details are in the rate limits documentation.
Common pitfalls
- Treating every page as trusted input. Website content is untrusted. Never let scraped content authorize follow-up tool calls or change permissions in your pipeline.
- Polling when you should subscribe. Webhooks on batches and monitors remove the poll loop and its rate-limit exposure.
- Crawling without a
urlRegex. Unbounded crawls waste credits on irrelevant paths and can blow through a sync crawl's time budget. - Hardcoding rate limits. Plan limits change as you upgrade; the runtime headers are the source of truth.
- Ignoring cache metadata. Re-fetching pages that a shared cache could serve adds latency for no benefit. If compliance requires it, opt into zero data retention, which bypasses shared caches end to end (see the ZDR guide); verify the
X-Context-ZDR: trueresponse header before trusting it. - Skipping per-page success checks. A crawl can partially succeed. Always check each page's
metadata.successbefore indexing it.
Frequently Asked Questions
How much does it cost to run a pipeline like this? Credits meter usage: a simple scrape is 1 credit, a crawl is 1 credit per successfully scraped page, and structured extraction is 10 credits per call. The Free plan includes 1,000 credits per month, Developer is $25/month for 10,000 credits, and paid plans add metered overage from $7 to $15 per 10,000 credits. Failed or blocked requests are never billed. Current details are on the pricing page.
Can this run inside my existing infrastructure without new scrapers?
Yes. The API is a single REST endpoint family with official SDKs for TypeScript, Python, Ruby, Go, and PHP, plus a CLI and an MCP server at https://mcp.context.dev/mcp for agent-driven workflows. Batches, monitors, and webhooks let results flow into your queues and databases directly.
How fresh is the data?
Scrapes and crawls fetch live content, and maxAgeMs on a crawl controls per-page freshness (default one day, 0 forces a re-fetch). Cacheable responses report their cache status, so you always know whether you are reading fresh content or a cached copy.
What if my compliance team requires that page content is not stored?
Zero data retention is available as a per-organization entitlement on six endpoints. When enabled, shared caches are bypassed, usage logs keep operational metadata only, and uploaded bytes and screenshots stay in memory. Contact support to enable it, and confirm the X-Context-ZDR: true header on each response.
Conclusion
A reliable pipeline for live web content does not require a scraping team. With context.dev you define what you need, call one API, and let managed infrastructure handle rendering, proxies, and anti-bot work, while webhooks and monitors keep fresh content flowing to your models on schedule. Start with the free plan's 1,000 monthly credits, prove the pipeline on a handful of sources, then scale up the plan as your volume grows.