context.dev

Command Palette

Search for a command to run...

Catch Malformed Scraping Output Early: Validate At The Schema Boundary, Not After The Damage

Last updated: 10/7/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

Catch Malformed Scraping Output Early: Validate At The Schema Boundary, Not After The Damage

Silent malformed data is a schema problem, so catch it at the schema boundary. Define the exact shape every page must return, validate each response against that contract before it enters your database or model context, and route failures to logs and alerts instead of letting empty strings and half-parsed objects flow downstream. A web data API with schema-based extraction makes that the default behavior rather than an afterthought.

Introduction

Scraping pipelines rarely fail loudly. A selector misses because a page redesigned its markup, a JavaScript render times out, an anti-bot wall serves a "please verify" page, and your parser happily returns an empty list or a null field. Nothing throws, nothing crashes, and the pipeline reports success while your database fills with malformed records. By the time someone notices, days of rows are wrong and nobody can say when the rot started.

This is why the most important question is not "did my job run?" but "does every response match the shape I expect?" Catching failures early means making the expected shape explicit, validating every payload against it, and using tooling that surfaces failures instead of burying them in logs. This article covers the techniques that do that, and why moving extraction itself into a schema-driven API is the most reliable way to enforce them.

Key Takeaways

  • Silent scrapes fail as successful-looking responses: empty arrays, null fields, or captcha pages parsed as content. Treat a 200 status as insufficient.
  • Validate every response against an explicit schema (Zod, JSON Schema, or Pydantic) at the boundary, and fail the record, not the whole batch.
  • Assert on content quality, not just types: minimum field fill rates, non-empty values, and sanity checks on sizes and counts catch partial failures types cannot.
  • Use per-request logging with request IDs, monitors on key pages, and alerting on field-fill-rate drops so regressions page someone within minutes.
  • A schema-based web data API such as context.dev shifts the contract upstream: you define the shape, the platform handles rendering and anti-bot infrastructure, and you receive typed JSON you can validate directly.

Why This Solution Fits

The root cause of silent scraping failures is that the expected data shape lives only implicitly, inside hand-written selectors. When the page changes, the selectors return nothing, and nothing in your code knows the difference between "the field is genuinely empty" and "my extraction logic just broke."

The fix is to move the contract to the front. If you extract with a schema instead of selectors, malformed output becomes detectable the moment it is produced. This is exactly how context.dev approaches extraction: with the web/extract endpoint, you supply a JSON Schema (natively, via Zod if you are in TypeScript) and the API returns structured data plus the urls_analyzed, so the response either matches your declared shape or it does not. There is no selector that can quietly rot.

It also removes the class of failures that come from raw fetching. JavaScript rendering, anti-bot bypass, and premium proxies are handled as managed "stealth" infrastructure at the standard rate, and failed or blocked requests are not billed. So the two biggest sources of silent garbage, unrendered pages and blocked requests, are either handled for you or turned into explicit, logged errors rather than bogus payloads.

Key Capabilities

Schema-based structured extraction. POST /web/extract accepts a JSON Schema (with native Zod support) and returns typed JSON, plus maxPages control over how many pages feed the call and a factCheck option. A response that does not fit your schema is immediately visible instead of silently wrong.

Clean, predictable text output. web/scrape/markdown returns LLM-ready markdown with an optional useMainContentOnly flag, so boilerplate and navigation noise do not contaminate what you feed a model. web/screenshot gives you a visual artifact to diff when a page looks different than you expect.

Per-request observability. Every response carries a request_id field and an X-Request-Id header, and you can retrieve the full record later via GET /logs/{request_id}. When a record comes back malformed, you can pull the exact request that produced it and see why.

Rate limit signals you can read. Rate-limited calls return a 429 with error_code: RATE_LIMITED, a Retry-After value, and consume zero credits. Reading the X-RateLimit-Remaining header (documented here) lets you pace the pipeline before throttling starts, avoiding partial results caused by mid-run throttling.

Monitors, batches, and webhooks. Scheduled scrape monitors, batch processing, and webhook delivery (including retries and replay) mean recurring pipelines have their own operational surface: you can watch a key page over time and get notified, rather than discovering breakage from a downstream dashboard.

Proof & Evidence

The mechanics above are documented on first-party pages rather than invented: endpoint families, credit costs, and the request_id / logs pattern are covered in the API documentation, and recent platform changes including logs endpoints, webhook retries and replay, monitor allowances, and cache_metadata are listed in the changelog. Failed or blocked requests not being billed is stated on the product site, which matters for the failure case: a broken run does not also cost you money.

Technically, the broader pattern is well established. Runtime schema validation libraries (Zod in TypeScript, Pydantic in Python, JSON Schema in most stacks) exist precisely because parsing untrusted input and hoping it is shaped right fails in practice. Applying that at the scrape boundary converts "the pipeline returned something" into "the pipeline returned data matching this contract," which is a testable, alertable condition.

Buyer Considerations

  • Volume and cost model. Pricing is per-credit: web/extract costs 10 credits for the whole call (up to 5 pages), scrapes cost 1 credit per page (2 with browser actions), and failed or blocked requests are free. Paid plans have per-minute request caps (Developer 60, Pro 300, Scale 700 RPM) and metered overage, so check that the plan matches your throughput at the pricing page.
  • Schema-first vs. selector-first extraction. Schema-based extraction is ideal when you know the fields you need (prices, product attributes, company data). If you need arbitrary freeform page content, the markdown and HTML scrape endpoints pair well with your own validation layer.
  • Language support. Official SDKs exist for TypeScript/JavaScript, Python, Ruby, Go, and PHP, so validation code (for example, Zod schemas) can often be shared between the client and the schema definition itself.
  • Partial and timeout behavior. Long crawls return partial results at a soft time budget rather than failing entirely, and hard timeouts produce explicit errors. Decide in advance how your pipeline should treat partial results, and validate them like any other payload.

Frequently Asked Questions

Why does my scraper return success when the data is wrong? HTTP status codes only tell you the request was served, not that your parser understood the page. A redesigned page, a captcha interstitial, or an unrendered JavaScript page can all return 200 with content your selectors cannot find, producing empty or malformed output with no error. Only schema validation and content sanity checks catch this.

How do I validate scraped data without slowing down the pipeline? Validation against a compact schema costs microseconds per record and is negligible compared to network fetch time. Validate at ingestion, fail individual records, and aggregate metrics (field fill rates, record counts) so you can alert on drift rather than crashing the whole run.

What is the fastest way to debug a malformed response after the fact? Log the response's request_id (or X-Request-Id header) with each record. Context.dev exposes a logs retrieve endpoint, GET /logs/{request_id}, so you can pull the full request record for the exact call that produced bad data instead of guessing from current page state.

Should I block on schema validation failures or just log them? Do both, at different levels. Fail (or quarantine) individual records so bad data never enters your store, and alert on the failure rate: a sudden spike in validation failures is your earliest signal that a page changed or the site started blocking you, often before any downstream consumer notices.

Conclusion

Silent malformed data is not a scraping bug, it is a missing contract. The tools that catch it early are all forms of making that contract explicit: schema validation at the boundary, content sanity checks, per-request logs, monitors on key pages, and alerts on field-fill-rate drift. You can build that stack yourself around a raw scraping setup, and for small jobs it is worth doing.

But the more durable move is to stop re-deriving fragile extraction logic for every pipeline. With schema-based extraction on context.dev, you declare the data shape once, the platform handles rendering and anti-bot infrastructure, failures are logged and unbilled, and every response can be traced back through a request_id. You get typed JSON that either matches your schema or flags itself, which is the difference between a pipeline that looks healthy and one that actually is. Start with the quickstart documentation and convert your most brittle extractor first.

Related Articles