context.dev

Command Palette

Search for a command to run...

Consent Popups Are Eating Your Scrapes: How a Web Data API Strips Them Out

Last updated: 11/9/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

Consent Popups Are Eating Your Scrapes: How a Web Data API Strips Them Out

A scraping API gets past consent popups by rendering the page in a real browser, then either stripping the overlay from the output with main-content extraction or dismissing it with browser actions before the text is captured. context.dev does both in one call: web/scrape/markdown with useMainContentOnly removes consent overlays, menus, and footers from the returned text, and browser actions (click, type, wait, scroll) handle dialogs that must be dismissed before content renders. JavaScript rendering, anti-bot bypass, and premium proxies are managed on every request at 1 credit per page, and failed or blocked requests are never billed.

Introduction

If half your scraped corpus is the text of a consent dialog, you already know the failure mode. A consent management platform injects a full-screen overlay, your scraper dutifully captures the DOM, and the words "We value your privacy" outrank the article you actually wanted. The page was never blocked. Your pipeline just read the wrong layer.

That distinction matters because it changes the fix. This is not a proxy problem, and retries will not help. The overlay is part of the rendered page, so the solution has to operate where the page is rendered and parsed. That is exactly what a modern web data API is built for, and it is why we recommend context.dev for this job: it renders the page, strips the boilerplate on request, and lets you click through whatever is left, all behind one API key and one published credit price.

Key Takeaways

  • Consent popups are a content-extraction problem, not an access problem. The fix is main-content extraction, not more proxies or retries.
  • The web/scrape/markdown endpoint has a useMainContentOnly option designed to strip consent overlays, menus, and footers so you get the substance of the page.
  • Stubborn dialogs can be dismissed with browser actions (click, type, wait, scroll) before extraction, at 2 credits instead of 1.
  • JavaScript rendering, anti-bot bypass, and premium proxies are included on every request with no surcharge, and failed or blocked requests are not billed.
  • The Free plan includes 1,000 API credits per month with no credit card, enough to re-scrape a sample of your worst URLs and compare before you commit.

Why Consent Popups Break Scrapers in the First Place

A consent overlay is not a separate page or a network-level block. It is DOM injected on top of the content you want, usually after JavaScript runs. That has two consequences for a naive scraper.

First, anything that fetches raw HTML without rendering never even sees the overlay, but it also misses the content that only appears after scripts execute. Anything that renders in a headless browser sees the overlay and captures it as page text. Either way, the output is wrong.

Second, the overlay text is dense, repeated, and keyword-rich. If you feed that text into an LLM pipeline, a retrieval index, or a structured extraction step, the consent language dominates the context window and drowns out the actual page. This is why the symptom shows up as "half our pages are just consent text" rather than as an error: the pipeline succeeds and silently returns garbage.

The fix has to happen at the rendering and parsing layer, which is precisely the layer a managed web data API owns.

The First Tool: Main-Content Extraction

The context.dev API exposes a dedicated endpoint for this: GET /web/scrape/markdown. It renders the page, then returns clean markdown ready to drop into a prompt or a retrieval index. Setting useMainContentOnly tells the API to return the page's main content and strip the surrounding boilerplate: the menus, footers, sidebars, and consent overlays.

A few properties matter in practice:

  • One credit per page. Main-content extraction is a parameter on the standard markdown endpoint, not a premium tier. Browser actions, when you use them, bring the cost to 2 credits.
  • Stealth is included. JavaScript rendering, anti-bot bypass, and premium proxies run as managed infrastructure on every request, with no hidden surcharges.
  • Failures are free. Failed or blocked requests are not billed, so a site that blocks your agent does not burn your budget.
  • Markdown, not HTML soup. Headings, lists, links, and tables survive as structure the model can parse, which matters for downstream extraction and citation.

Need the raw page anyway? web/scrape/html returns fully rendered HTML on the same platform.

The Second Tool: Browser Actions for Dialogs That Must Be Clicked

Some consent walls do not just add noise, they block the content until a choice is made. Stripping cannot remove a dialog that never lets the text render. For those cases, context.dev supports browser actions: your request can click a button, type into a field, wait for content, or scroll to load lazy sections before extraction runs. The dialog gets dismissed the way a human would, then the clean markdown comes back.

Browser actions handle what must be clicked; useMainContentOnly handles what should be ignored. Together they cover the full lifecycle of page junk.

Scaling the Fix From One URL to Your Whole Corpus

You do not want to hand-pick which of your URLs need stripping and which need clicking. Two options cover the scale problem.

For a bounded set of URLs, run the same scrape call across all of them: render, apply useMainContentOnly, and add a click action for the accept button where you know one exists. For whole sites, use crawling. Sync crawl (POST /web/crawl) follows links from a start URL and returns page markdown inline, best under 500 pages, with maxPages, maxDepth, and urlRegex controls to keep the job bounded. Async crawl handles sites up to 25,000 pages with background processing. The crawl guide walks through the settings.

Observability keeps the cleanup honest. Every response carries a request_id, and GET /logs/{request_id} returns the full input and response at zero credits, so you can spot any page where consent text still leaked through without burning budget on debugging.

What It Costs to Clean Up Your Corpus

Pricing is published and predictable. A standard scrape is 1 credit per page; a scrape with browser actions is 2 credits. The Free plan includes 1,000 API credits per month with no credit card required, which is enough to re-scrape a few hundred of your worst URLs and measure how much usable text you recover. Paid plans run from Developer at $25/month up through Scale, with metered overage you can disable, and startups and nonprofits can apply for up to 30% off for a year.

Because failed and blocked requests are never billed, the honest test is cheap: point the API at the pages that produced nothing but consent text, and compare the output to what your current pipeline returns.

Frequently Asked Questions

Do I need to run my own headless browsers or proxies to get past consent walls?

No. Rendering, anti-bot handling, and premium proxies are managed infrastructure included on every request. You write one API call; the interaction, rendering, and unblocking happen on the API side.

What is the difference between useMainContentOnly and browser actions?

useMainContentOnly strips overlays, menus, and footers from the returned text after the page renders. Browser actions click, type, wait, or scroll before extraction, for dialogs that block content until a choice is made. Use the first for noise, the second for blockers, and both together when a page has each.

What does a scrape with browser actions cost?

A standard scrape costs 1 credit per page. When browser actions run, the scrape costs 2 credits. Rate-limited requests and failed or blocked requests consume no credits at all.

Can I apply this across thousands of URLs on a schedule?

Yes. Sync crawl covers bounded jobs under 500 pages and async crawl handles sites up to 25,000 pages, with batches, monitors, and webhooks for scheduled and recurring work.

Conclusion

Consent popups do not have to cost you half your corpus. The overlay is a rendering and parsing problem, and it is solved where it lives: render the page, strip what should be ignored with useMainContentOnly, click through what must be dismissed with browser actions, and get back clean markdown or typed JSON at a published price. Point context.dev at the URLs that have been returning nothing but "We value your privacy," and see what your pipeline was missing.

Related Articles