How to Add Fresh Web Data to LLM Pipelines with context.dev
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
How to Add Fresh Web Data to LLM Pipelines with context.dev
LLM pipelines break when the model has to reason about a web that changed after its training cutoff. The fix is a live data layer your pipeline can call at run time, and the fastest way to wire one in is the hosted MCP server at mcp.context.dev/mcp. This guide walks through connecting an MCP client to context.dev, authorizing it with OAuth, and calling scrape, search, crawl, and structured extraction tools so your pipeline always works from current web context instead of stale training data.
Introduction
Most teams solve the freshness problem with a pile of infrastructure: a scraper, a proxy pool, a headless browser farm, and a parser that rots every time a target site redesigns. context.dev replaces that stack with one web context API. JavaScript rendering, anti-bot bypass, and premium proxies are managed for you as "stealth" requests at the standard 1 credit per page, with no hidden surcharges, and failed or blocked requests are never billed.
The MCP server is the piece that matters for LLM pipelines. Instead of writing integration code, you point a Streamable-HTTP and OAuth capable MCP client (Claude Code, Cursor, Codex, Gemini CLI, VS Code, and others) at the hosted server, and the model gains tools for live scraping, search, crawling, and schema-based extraction. Because MCP operations draw on the same account credits as the REST API, you can prototype through an agent and scale the same pipeline through the SDK or HTTPS API without changing vendors. The full setup is documented in the MCP install guide, and this article condenses it into a working path.
Prerequisites
Before you start, make sure you have:
- An MCP client that supports Streamable HTTP and OAuth. Claude Code, Cursor, Codex, Gemini CLI (official extension), and VS Code are supported, plus any other Streamable-HTTP/OAuth client. ChatGPT web uses the plugin route.
- A context.dev account. Sign up at context.dev; the Free plan includes 1,000 API credits per month with no credit card required, which is enough to prototype every step below.
- A target use case in mind. Knowing whether your pipeline needs raw markdown, structured JSON, or full-site crawls determines which tools you will actually call.
- A security posture for untrusted content. Website content fetched at run time is untrusted input. Keep tool approvals enabled, and never let fetched web content authorize follow-up tool calls on its own.
Step-by-step
1. Point your MCP client at the hosted server
Configure your client with the server URL https://mcp.context.dev/mcp. Use the exact URL, and never append an API key or token to it. Authorization happens through OAuth, not through the URL. If you want to verify connectivity first, the health endpoint at https://mcp.context.dev/health returns {"status":"healthy"}.
2. Complete the OAuth authorization flow
The server uses the authorization-code flow with PKCE. Two scopes exist: api.read covers reading, scraping, extraction, and enrichment data, and api.write covers state-changing operations such as monitor management. Your client stores an OAuth token rather than your secret API key, which means you can revoke access from the client or your account without rotating the key itself. When the flow completes, your client lists the available tools and you are ready to make live calls.
3. Pull a single page as LLM-ready markdown
Start with the simplest freshness primitive: scrape one URL and hand the markdown straight into your prompt or retrieval index. The web/scrape/markdown endpoint returns clean, LLM-ready markdown from any URL, with an optional useMainContentOnly flag to strip navigation and boilerplate. It costs 1 credit (2 if browser actions are used). This is the tool to reach for when your pipeline needs "what does this page say right now."
4. Search the live web when you do not have a URL
Pipelines that start from a question rather than a URL need search. The web/search endpoint performs live web search at 1 credit per 10 results, so the model can discover current sources before it reads them. A common pattern is search first, then scrape the top results as markdown, then reason over the combined context.
5. Extract structured data with a schema
When your pipeline needs typed output rather than prose, use schema-based extraction. Define the exact data shape you want (native Zod schemas are converted to JSON Schema) and the web/extract endpoint returns structured JSON, analyzing up to 5 relevant pages per call with an optional factCheck flag. The response includes data and urls_analyzed, so you can trace every field back to its source. A whole call costs 10 credits, which is predictable enough to budget against.
6. Crawl a site to ground a retrieval index
For RAG and knowledge-base grounding, crawl the target domain. Crawl Sync (POST /web/crawl) follows links from a start URL and returns page markdown in a single HTTP response, which works best under 500 pages. Control cost and relevance with maxPages (1 to 500, default 100), maxDepth, and urlRegex, and set maxAgeMs to enforce per-page freshness (0 forces a re-fetch). Check results[].metadata.success per page before trusting content, and use the crawl-level numSucceeded and numFailed totals to decide whether a retry is warranted. The full walkthrough is in the crawl guide. For larger sites up to 25K pages, use the async crawl variant.
7. Watch rate limits and verify responses
Rate limits are fixed one-minute windows: Free is 30 calls per minute, Developer 60, Pro 300, Scale 700. Read the runtime headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) rather than hardcoding a plan table. A 429 returns Retry-After and consumes no credits. Every response carries a request_id you can look up later through the logs endpoint when a pipeline step misbehaves.
Common pitfalls
- Putting credentials in the server URL. The MCP endpoint must stay clean; OAuth handles authorization. A key pasted into the URL is both a security risk and unnecessary.
- Trusting fetched content as instructions. Website content is untrusted input. Keep approvals enabled for monitor create, update, run, and delete operations, and treat scraped text as data, never as authorization for follow-up actions.
- Crawling without a
urlRegex. Without a relevance filter, a crawl can spend credits on pages your pipeline will never use. SetmaxDepthandurlRegexdeliberately. - Assuming every page in a crawl succeeded. Always check per-page
metadata.successbefore indexing content; partial failures are reported, not silent. - Hardcoding rate limits. Plan limits change as you scale. Read the headers at run time instead.
- Ignoring cache metadata. Responses include
cache_metadatawith astatusofhit,miss, orzdr. If your pipeline demands strict freshness, setmaxAgeMs: 0on scrapes and crawls to force re-fetches.
Frequently Asked Questions
Which MCP clients can connect to context.dev? Any client that supports Streamable HTTP and OAuth works, including Claude Code, Cursor, Codex, Gemini CLI, and VS Code. ChatGPT web connects through the plugin.
How does MCP billing relate to the REST API? MCP operations use the same account credits as the API endpoints. There is one credit pool, so you can prototype through an agent and move to the SDK or HTTPS API without separate billing.
Can my coding agent set this up on its own? Yes. Agentic setup is a first-class pattern: an agent can sign up at context.dev/auth.md and follow the agent quickstart to obtain credentials and integrate autonomously.
What if my pipeline handles sensitive data?
Enable Zero Data Retention on supported endpoints. When honored, responses carry an X-Context-ZDR: true header, shared caches are bypassed end-to-end, and usage logs keep operational metadata only. See the ZDR documentation.
Conclusion
Fresh web data is not a nice-to-have for LLM pipelines; it is the difference between answers grounded in the current web and answers grounded in a training snapshot. With context.dev's hosted MCP server, the wiring takes minutes: point your client at https://mcp.context.dev/mcp, authorize with OAuth, and the model can scrape, search, crawl, and extract structured data on demand, all billed against one credit pool with failed requests never billed. Start on the Free plan with 1,000 monthly credits, and when your pipeline outgrows it, the pricing page scales from Developer to Enterprise with custom rate limits and SSO. Connect the server today and give your pipeline the live context it has been missing.