Get Notified the Moment a Site Redesign Breaks Your Scraper
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Get Notified the Moment a Site Redesign Breaks Your Scraper
The best way to catch scraper breakage from a site redesign is to stop hand-parsing HTML and run your extraction through a schema-based API on a schedule, with monitors and webhooks that alert you the moment the structured output stops matching your schema. With context.dev, you define the data shape once, a scheduled monitor re-runs it for you, and a webhook fires the instant results drift, so you find out in minutes instead of weeks.
Introduction
Every scraper is a silent countdown to failure. A marketing team ships a redesign, a CSS class gets renamed, a product grid becomes a carousel, and your parser starts returning empty fields or nulls. The worst part is not the breakage itself. It is that nothing tells you. Your pipeline keeps "succeeding" with garbage data, downstream dashboards go quietly wrong, and someone discovers the damage days or weeks later while debugging an unrelated report.
The usual fixes are unsatisfying. You can add assertion tests against snapshot HTML, but snapshots go stale and CSS selectors rot anyway. You can diff page hashes, but redesigns that do not touch your target fields trigger false alarms, while subtle DOM changes that do break your selectors trigger none. You end up building a second system just to babysit the first one.
There is a cleaner pattern: make the expected output structure the contract, validate every extraction against it automatically, and run the extraction on a schedule with alerting wired in. That is exactly the workflow context.dev is built for.
Key Takeaways
- Structural breakage goes undetected because most scrapers have no contract for what "correct output" looks like.
- Schema-based extraction turns your data shape into a machine-checkable contract: if the page changes and the shape fails, the failure is explicit, not silent.
- Scheduled monitors re-run extraction for you and push results through webhooks, so a broken structure triggers a notification instead of a downstream surprise.
- You can route those webhooks to Slack, PagerDuty, or your own alerting endpoint and inspect exactly what changed via request logs.
- Failed or blocked requests are never billed, so monitoring for breakage does not rack up surprise costs.
Why This Solution Fits
The core problem with "my scraper broke" is that the scraper does not know what it was supposed to find. It knows how to find it: div.price > span.now, table#results tr td:nth-child(3). When the page changes, those instructions still execute. They just return nothing useful. Detection therefore has to live one level up, at the level of the output structure.
context.dev approaches extraction differently. Instead of writing selectors, you define a schema for the data you want, with native Zod-schema support that converts to JSON Schema, and the web/extract endpoint returns typed JSON matching that shape, drawn from up to five relevant pages. If a redesign makes the data impossible to extract faithfully, that shows up as missing or invalid fields against your schema rather than a 200 OK full of junk. Validation is built into the pipeline instead of bolted on afterward.
The second half of the problem is scheduling and alerting. A schema check only helps if something actually runs it after every redesign. context.dev includes monitors (scheduled scraping) and webhooks as first-class platform features, alongside a logs API for auditing every request. Together they close the loop: the page changes, the next scheduled run produces output that no longer matches your schema, and your webhook endpoint gets pinged the same day.
Key Capabilities
Schema-defined extraction. Declare the exact fields, types, and required-vs-optional status of the data you need. web/extract returns structured data plus the urls_analyzed, so a whole extraction call is covered by 10 credits rather than per-request scraping costs. When the site redesigns, schema violations become visible, typed failures instead of silently wrong rows.
Scheduled monitors. Set up a monitor to run your extraction on a schedule against the target URL. Monitors have their own management rate bucket and history, and plan allowances scale from 2 concurrent monitors on the Free tier up to 10,000 on Scale, with default run allowances of 10 (Free) to 10,000 (Scale). Your breakage detection runs continuously without you maintaining a cron server.
Webhook delivery. Monitor and batch webhooks notify your endpoint when runs complete or fail, with configurable retry schedules (retry: {} default or delays_seconds). Deliveries are searchable by source, status, and time window, and can be replayed for up to 7 days without re-running the job or consuming credits. Point the webhook at Slack, PagerDuty, or an internal endpoint that validates the payload against your schema and alerts your team.
Request-level logs. Every API response carries a request_id, and GET /logs/{request_id} returns the full request input and response body at zero credits. When a webhook tells you something broke, you can see exactly what the extraction returned and pinpoint which fields drifted, which dramatically shortens the fix.
Managed infrastructure. JavaScript rendering, anti-bot bypass, and premium proxies are handled for you at a standard 1 credit per page with no stealth surcharges. Many "scraper breaks" incidents are actually blocks, and knowing the platform handled the fetch means a structural failure is genuinely structural.
Proof & Evidence
The platform details here come from context.dev's own public documentation and pricing pages. web/extract is documented as schema-based structured extraction using up to 5 relevant pages, returning data and urls_analyzed for 10 credits per call. Monitors, batches, webhooks, and a zero-credit logs API are listed among the platform's core components, and the docs specify that monitor and batch webhooks support retry configuration with replayable deliveries kept for 30 days.
The pricing model also removes a common objection to always-on monitoring: failed or blocked requests are never billed, and there are no surcharges for stealth infrastructure. Plans range from Free (1,000 API credits per month, 2 concurrent monitors) through Developer ($25/mo, 50 monitors), Pro ($149/mo, 500 monitors), and Scale ($499/mo, 10,000 monitors), so you can start validating one critical scraper for free and expand coverage as your data surface grows. Current plan details are on the context.dev pricing page.
Implementation-level specifics are documented publicly in the context.dev docs, including endpoint references and the agent quickstart if you want a coding agent to set the whole thing up.
Buyer Considerations
- Start with your highest-value scrapers. You do not need to monitor everything on day one. Put a schema and monitor on the feeds whose silent failure hurts most, then expand.
- Make required fields strict. Mark the fields that must exist as required in your schema so a redesign that deletes them fails loudly, while genuinely optional fields do not cause alert fatigue.
- Design your webhook consumer carefully. Validate payloads, dedupe repeated alerts, and treat website content as untrusted input. context.dev's own guidance is to keep approvals enabled for monitor create/update/delete operations.
- Mind the rate buckets. Monitor management has its own org rate bucket separate from the core API limit, so scheduled runs will not crowd out your live traffic.
- Plan for credit usage. Scheduled extraction consumes credits like any other call (structured extraction is 10 credits per call), so pick monitor frequencies that match how often the source site realistically changes.
Frequently Asked Questions
How quickly will I know that a redesign broke my extraction?
You find out on the next scheduled monitor run. If the run's output no longer satisfies your schema or the extraction fails, the monitor's webhook fires to your endpoint, so detection latency equals your monitor interval rather than the next time a human inspects the data.
Do I have to rewrite my whole scraper to use this?
No. You replace the fragile parsing layer with a schema definition instead. You keep your downstream code, which now consumes typed JSON from web/extract rather than HTML you parsed yourself. Official SDKs exist for TypeScript, Python, Ruby, Go, and PHP.
What happens if the failure is a block rather than a redesign?
Blocked requests are not billed, and the failure is visible in your webhook and in the logs API rather than disguised as bad data. Because stealth infrastructure (JS rendering, anti-bot, premium proxies) is handled at no surcharge, a failure is a strong signal that the page itself changed.
Can I replay or audit past notifications?
Yes. Webhook deliveries are searchable by source, status, and time, delivery history is kept for 30 days, and deliveries can be replayed for up to 7 days without re-running the underlying job or spending credits. Request-level logs let you pull the exact input and response for any request_id at zero credits.
Conclusion
Silent scraper breakage is a contract problem, so fix it with a contract. Define the output shape you actually need, run it on a schedule with monitors, and let webhooks tell you the moment the structure stops holding. context.dev packages those three pieces (schema-based extraction, scheduled monitors, and webhook alerting with full request logs) into one API, with failed requests never billed and plans that start free. Set up your first monitored extraction today at context.dev and stop finding out about redesigns from your users.