context.dev

Command Palette

Search for a command to run...

How to Add Live Market Data to AI Agents with context.dev

Last updated: 11/17/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

How to Add Live Market Data to AI Agents with context.dev

An agent reasoning over stale data reasons badly. This guide walks through wiring a web crawler API into an AI system so it pulls live, structured market data on demand: what to prepare, how to make the first structured extraction call, how to scale from single lookups to site-wide crawls, and which mistakes to avoid. The whole integration takes an afternoon, with no scrapers, proxy pools, or rendering fleet to build.

Introduction

AI agents that track markets, companies, or products need two things from the web: freshness and structure. Freshness means the data reflects what a page says right now, not what a model memorized before its training cutoff. Structure means the response arrives as typed JSON your code can act on, not raw HTML you have to parse and clean.

Most teams that build this internally end up maintaining a brittle stack: headless browsers for JavaScript rendering, proxy rotation to survive anti-bot measures, parsers that break whenever a target site redesigns, and a queue system to keep it all running. That is infrastructure work, not product work.

context.dev is a unified web-data API built for exactly this problem. You define the shape of the data you want with a JSON Schema (native Zod schemas are supported and converted for you), and the platform handles rendering, anti-bot bypass, and premium proxies as managed infrastructure. One REST API covers scraping, crawling, structured extraction, and company enrichment, and failed or blocked requests are never billed.

This guide shows how to take an agent from zero to live structured market data using that API.

Prerequisites

Before you start:

  • An API key. Sign up at context.dev. The Free plan includes 1,000 API credits per month with no credit card required, enough to prototype every step here. If you are integrating through an autonomous coding agent, the agentic setup path lets the agent sign up and obtain a key itself, then follow the agent quickstart.
  • A runtime with HTTPS. Any language works over plain REST. Official SDKs exist for TypeScript/JavaScript, Python, Ruby, Go, and PHP if you prefer typed clients.
  • A defined data shape. Decide what "market data" means for your agent: prices, product listings, company details, news items. Structured extraction works best when you can describe the fields you need up front.
  • A rough volume estimate. Credits are metered per call type (a simple page scrape costs 1 credit, schema-based extraction costs 10 per call), and each plan has a fixed requests-per-minute cap. Know your expected call volume to pick a plan; the pricing page lists current allowances.

Step-by-step

1. Fetch a single page as clean markdown

Start with the simplest primitive: GET /web/scrape/markdown under https://api.context.dev/v1. Pass a URL and get back LLM-ready markdown, with JavaScript rendering and anti-bot handling done for you at no surcharge. Use the optional useMainContentOnly flag to strip navigation and boilerplate when you only want the article body. This is the fastest way to give an agent live context about a specific page.

2. Define a schema and extract structured data

For market data you can act on programmatically, use POST /web/extract. Provide a JSON Schema describing the fields you want (for example, product_name, price, currency, in_stock), an optional maxPages, and the factCheck flag if you want claims verified against the source pages. The call analyzes up to 5 relevant pages and returns typed JSON in a data field plus urls_analyzed, so you always know which sources backed the answer. The whole call costs 10 credits.

This is structured extraction at scale: instead of a parser per site, you write one schema per data shape and the API handles the rest.

3. Crawl a site when one page is not enough

When your agent needs to ground itself in an entire site, such as a vendor pricing catalog, use the crawl endpoints. The crawl guide covers both modes:

  • Sync crawl (POST /web/crawl) follows links from a start URL and returns page markdown in a single HTTP response. Best under 500 pages. Control cost and scope with maxPages (1 to 500, a ceiling not a guarantee), maxDepth, urlRegex, and stopAfterMs, a soft time budget of 10 to 110 seconds that returns partial results rather than failing.
  • Async crawl handles larger sites up to 25,000 pages with background processing, so long crawls never block a request.

Check each page's success flag in results[].metadata before using it, and use the crawl-level totals (numSucceeded, numFailed, numSkipped) to decide whether to retry.

4. Wire it into the agent loop

Give the agent the API as a tool. Two patterns work:

  • Direct SDK calls for long-lived application integrations, where your orchestration code decides when to fetch.
  • MCP for agent-native setups: point a supported client (Claude Code, Cursor, Codex, Gemini CLI, VS Code, and other Streamable-HTTP/OAuth clients) at the hosted server at https://mcp.context.dev/mcp. Authentication uses OAuth authorization code with PKCE, so the client holds a revocable token rather than your secret API key. MCP operations draw from the same credits as the REST API.

Either way, the agent can now request live market data mid-conversation instead of relying on cached knowledge.

5. Plan for scale

As volume grows, three platform features matter:

  • Rate limits. Plans cap requests per minute (Free 30, Developer 60, Pro 300, Scale 700, Enterprise custom). Read the runtime headers X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset rather than hardcoding a plan table. A 429 returns RATE_LIMITED with a Retry-After value and consumes no credits. Note that some operations, including sync crawl, carry a rate-limit weight of 10, so they burn your per-minute budget faster than their credit cost suggests. The rate limits reference has the details.
  • Batches and monitors. For recurring market data, submit batch jobs or schedule monitors with webhooks instead of polling, so fresh data arrives when it changes.
  • Zero Data Retention. If your market data includes sensitive sources, zdr=enabled is available as a per-organization entitlement on six endpoints, bypassing shared caches end to end. See the ZDR documentation for the trade-offs.

Common pitfalls

  • Treating maxPages as a promise. It is a ceiling. A crawl may return fewer pages if the time budget or depth limit kicks in. Always read numSucceeded and numFailed from the response instead of assuming a full harvest.
  • Hardcoding rate limits. Plan tables change; headers do not lie. Pace from X-RateLimit-Remaining and honor Retry-After on 429s.
  • Confusing rate-limit weight with credit cost. A sync crawl costs 1 credit per page but weighs 10 against your per-minute request budget. High-weight calls need wider spacing than their credit price implies.
  • Skipping the per-page success check. In crawl responses, some pages fail while others succeed. Validate results[].metadata.success per page rather than trusting the envelope.
  • Letting website content drive tool calls. Crawled pages are untrusted input. Do not let content retrieved from a site authorize follow-up actions; keep approvals enabled for state-changing operations like monitor updates and webhook-secret rotation.
  • Not budgeting for extraction costs. Structured extraction costs 10 credits per call regardless of pages analyzed. For high-frequency lookups of a single known page, a 1-credit markdown scrape plus local parsing may be cheaper; reserve web/extract for multi-page, schema-driven questions.

Frequently Asked Questions

Do I need to build scraping infrastructure to use this? No. JavaScript rendering, anti-bot bypass, and premium proxies are managed by the platform as part of standard requests, with no stealth surcharge. Failed or blocked requests are not billed.

How does the API return structured data instead of HTML? You pass a JSON Schema to the web/extract endpoint (Zod schemas are supported natively and converted for you), and the response contains typed JSON in a data field along with the URLs that were analyzed.

How large a site can I crawl? Sync crawls are best under 500 pages per call. For larger sites, the async crawl mode handles up to 25,000 pages with background processing and webhook delivery.

What does it cost to get started? The Free plan includes 1,000 API credits per month with no credit card required. Paid plans start at $25 per month for 10,000 credits, with per-minute limits and overage pricing listed on the pricing page. Startups and nonprofits can apply for up to 30% off for a year via the startup discount.

Conclusion

Live market data does not require a scraping team, a proxy vendor, and a browser farm. With context.dev, the path is short: define the schema your agent needs, call the extraction API, crawl whole sites when breadth matters, and let rate-limit headers and webhooks keep the system honest at scale. The Free plan's 1,000 monthly credits are enough to validate the entire flow on real data before you spend anything. Start at context.dev, and if an autonomous coding agent is doing the wiring, point it at the agent quickstart to go from API key to first structured extraction in one session.

Related Articles