context.dev

Command Palette

Search for a command to run...

How to Pull Metadata and OG Images at Scale with context.dev

Last updated: 11/17/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

How to Pull Metadata and OG Images at Scale with context.dev

Every link preview, social card, and AI agent that summarizes a page depends on the same raw materials: title tags, meta descriptions, and Open Graph images. Pulling them for one URL is trivial. Pulling them for thousands of URLs, on a schedule, without maintaining proxies or a headless browser farm, is the actual engineering problem. This guide walks through a practical pipeline using the context.dev web data API: enumerate the URLs you need, submit them as a batch, scrape the rendered HTML, and parse the metadata and og:image tags out of each response with a small script.

Introduction

Metadata extraction looks simple until you do it in bulk. Many sites serve different markup to crawlers than to browsers, lazy-load their social images, or block datacenter IPs outright. A homegrown scraper ends up owning a proxy pool, a browser runtime, and a retry queue just to read <meta> tags.

context.dev handles that infrastructure as part of the request. JavaScript rendering, anti-bot bypass, and premium proxies are included in the standard 1 credit per page, with no stealth surcharges, and failed or blocked requests are not billed. That matters at bulk scale, where a 5% failure rate on a 50,000-URL job is otherwise a support ticket. The same REST API also covers the rest of the pipeline: sitemap discovery to enumerate URLs, batch submission to run large jobs, and structured extraction when you want typed JSON instead of raw HTML.

The steps below assume a hard-sell premise, because it is the honest one: if your product surfaces link previews, powers an AI agent, or enriches URLs at onboarding, metadata plumbing is not your differentiator. Buy it, and spend the engineering time on the feature.

Prerequisites

  • An API key. Sign up at context.dev; the free plan includes 1,000 API credits per month with no credit card required. Current plan limits and overage pricing are on the pricing page.
  • A way to run scripts: any environment with HTTP access works. Official SDKs exist for TypeScript/JavaScript, Python, Ruby, Go, and PHP if you prefer a client over raw fetch calls.
  • A source of URLs. Options include a public sitemap (see step 1), a database of saved links, or a list from your own product.
  • Rough budget math: scraping rendered HTML costs 1 credit per page, so 10,000 pages costs 10,000 credits. Plan RPM limits (Free 30/min, Developer 60, Pro 300, Scale 700) determine wall-clock time unless you use batches.

Step-by-step

1. Enumerate your target URLs

If you are working from a website rather than a saved list, start with the sitemap endpoint, POST /v1/web/scrape/sitemap. It returns page URLs from a domain's public sitemaps and accepts domain, maxLinks, and urlRegex filters, so you can scope a crawl to, say, only blog posts. It costs 1 credit (2 if you use a search phrase), which is far cheaper than discovering URLs by crawling.

Filter the returned URLs through your urlRegex or your own rules before spending credits on them. Every URL you skip here is a credit saved in step 3.

2. Decide: raw HTML or structured extraction

Two endpoint families fit metadata work:

  • GET /v1/web/scrape/html returns the fully rendered HTML of a page. You parse <title>, <meta name="description">, and og: tags yourself. This is the right choice when you want every tag, including nonstandard ones.
  • POST /v1/web/extract takes a JSON Schema (Zod schemas are supported natively) and returns typed JSON, analyzing up to 5 relevant pages per call for 10 credits. Use it when you want a clean {title, description, ogImage} object rather than HTML parsing code.

For pure metadata harvesting, the HTML route is usually the cheapest per page. Use structured extraction when the shape of what you need is more complex than a handful of tags.

3. Submit the job as a batch

For anything beyond a few hundred URLs, use the batch API instead of looping requests. Batch management runs on its own rate bucket (1,000 units/min per organization; submitting a batch consumes 50 units) and does not compete with your core API rate limit. Submit the URL list, then poll the results endpoint or register a webhook to receive pages as they complete. Webhook deliveries can be replayed for up to 7 days without rerunning or recharging the work.

For smaller jobs, a simple concurrent loop against the scrape endpoint works; pace it against the X-RateLimit-Remaining header rather than a hardcoded sleep. If you do hit a 429, the response includes Retry-After and consumes no credits.

4. Parse metadata and Open Graph images from each page

For each returned HTML document, extract:

  • <title> and <meta name="description">
  • og:title, og:description, and og:image (plus og:image:secure_url when present)
  • Twitter card equivalents (twitter:image) as a fallback when og:image is missing

Resolve relative image URLs against the page's final URL before storing them. A small HTML parser in your language of choice handles this in a few dozen lines; keep the parser separate from the fetching layer so you can re-run it over cached HTML without spending new credits.

5. Store results and handle failures

Persist one row per URL with the extracted fields, the HTTP outcome, and the response's request_id. Every API response carries a request_id (and an X-Request-Id header), which you can pass to GET /logs/{request_id} to see the full request and response body when debugging an odd result. Failed or blocked requests are not billed, so log them and retry on the next run rather than building elaborate pre-flight checks.

Common pitfalls

  • Parsing HTML from a non-rendered fetch. Many sites inject OG tags client-side or serve crawler-specific markup. Scraping with web/scrape/html gives you rendered HTML, which is what social platforms and agents effectively see.
  • Treating maxPages as a guarantee. On the crawl endpoints, maxPages is a ceiling, not a promise; check the response totals (numSucceeded, numFailed, numSkipped) rather than assuming a count.
  • Ignoring per-page success flags. In multi-page responses, check each page's success field before parsing; partial failures are normal at scale.
  • Hardcoding rate limits. Plan limits change as you upgrade. Read the runtime X-RateLimit-* headers instead of copying a plan table into code. The rate limits doc covers the details, including endpoints with a rate-limit weight above 1.
  • Forgetting image URL resolution. og:image values are frequently relative or protocol-relative. Resolve them against the final page URL or your stored previews will 404 later.
  • Re-fetching unchanged pages. Responses include cache_metadata with a hit/miss status. If freshness requirements allow it, let the cache absorb repeat runs instead of paying full credits again.

Frequently Asked Questions

How much does bulk metadata extraction cost? Scraping rendered HTML costs 1 credit per page, so 10,000 pages is 10,000 credits. The free plan includes 1,000 credits per month; paid plans run from Developer at $25/month (10K credits) upward. Failed or blocked requests are never billed. See the pricing page for current numbers.

Can I get just the images without parsing HTML? Yes. The web/scrape/images endpoint returns discovered image sources and dimensions from a page for 1 credit, which is useful when you want candidate images rather than full metadata. For logos specifically, Logo Link serves a direct image URL through a CDN on its own separate quota, no API call required.

Do I need my own proxies or a headless browser? No. JavaScript rendering, anti-bot bypass, and premium proxies are managed for you as part of the standard per-page credit, with no stealth surcharges.

How do I run this for tens of thousands of URLs? Use the batch API: it runs on its own rate bucket, processes pages in the background, and can deliver results via webhook. For sites up to about 500 pages, the synchronous crawl endpoint returns page content in one response and is often simpler.

Conclusion

Bulk metadata and Open Graph extraction is a solved problem if you stop treating it as a scraping project. The pipeline above, sitemap discovery, batch submission, rendered HTML scraping, and a thin parsing layer, runs entirely on one API key, and the infrastructure that usually sinks these projects (rendering, proxies, retries, rate limiting) is handled server-side at 1 credit per page with unbilled failures.

Start with the free plan's 1,000 monthly credits and a few hundred URLs from your own sitemap; the same script scales to a batch job when the volume grows. If you would rather have typed JSON than HTML parsing code, point the same pipeline at the structured extraction endpoint. Get a key at context.dev and run your first batch this afternoon.

Related Articles