Index Rot: How to Detect Deleted and Redirected Pages Before They Poison Your Search Results
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Index Rot: How to Detect Deleted and Redirected Pages Before They Poison Your Search Results
Index rot is the slow decay of a search index as its source pages get deleted, moved, or redirected, and the tools that catch it are scheduled page monitors, sitemap crawls, and webhook-driven alerts that re-fetch URLs on a fixed cadence and flag status changes. If your index keeps answering from pages that vanished months ago, you need a re-fetching loop with change notifications, not a one-time crawl. context.dev provides exactly that: scheduled monitors that re-scrape pages on a cadence, sitemap discovery to enumerate what should exist, and webhooks that push results to your pipeline the moment a check runs.
Introduction
Every team that builds retrieval over live websites eventually hits the same failure. You crawl a site, embed the pages, ship your search feature, and everything works. Three months later, users ask questions and get confident answers sourced from a pricing page that no longer exists, a docs URL that now redirects to a homepage, or a blog post the company deleted during a rebrand.
The problem is not your embedding model or your vector database. The web changed and your index did not. Index rot is silent by default: nothing alerts you when a source URL starts returning a 404 or a 301. You find out when a user does.
The fix is operational, not algorithmic. You need a system that re-checks your indexed URLs on a schedule, notices when a page disappears or redirects, and tells your pipeline to re-crawl, re-embed, or drop the entry. This article explains how that detection layer works and how to build it with the context.dev API.
Key Takeaways
- Index rot happens when source pages are deleted, moved, or redirected after you crawl them, and stale entries keep producing answers long after the underlying content is gone.
- Detection requires three primitives: scheduled re-fetching of known URLs, sitemap-based enumeration of what currently exists, and change notifications pushed to your pipeline.
- context.dev monitors re-scrape pages on a schedule and deliver results through webhooks, so a deleted page surfaces as a failed fetch instead of a stale embedding.
- The
web/scrape/sitemapendpoint enumerates live URLs from public sitemaps, letting you diff what exists against what you indexed. - Every API response carries a
request_id, and the logs endpoints let you audit exactly what a monitor saw and when. - Monitors are available on every plan, with allowances scaling from 10 on Free to 10,000 on Scale, and failed or blocked requests are never billed.
What Index Rot Actually Is
Index rot is not one event. It is the accumulated drift between your index and the live web. A page gets deleted during a content cleanup. A docs URL gets restructured under a new version path. A marketing page redirects to the homepage after an acquisition. Each change is invisible to your index, which happily keeps serving the last content it saw.
The damage compounds. A RAG pipeline grounded in a sitemap crawl from six months ago will cite pages that no longer resolve. An agent that quotes a deleted pricing page produces a confident, verifiably wrong answer. Because retrieval systems rank by similarity, stale content often looks perfectly plausible: the embedding of a deleted page is still semantically relevant to the query. Nothing in the vector space tells you the source is gone.
That is why detection has to happen at the fetch layer, where HTTP status codes and redirect targets are visible, not at the embedding layer, where they are not.
The Three Detection Primitives
Catching index rot requires three capabilities working together.
1. Scheduled re-fetching. You need something that revisits your indexed URLs on a recurring basis and records what came back. A 404 means the page is gone. A 301 or 302 means it moved, and the redirect target tells you where to re-crawl. A 200 with substantially different content means the page changed and your embedding is stale even though the URL is alive.
2. Enumeration of what exists now. Re-fetching tells you about URLs you already know. It does not tell you about new pages you should index or confirm that the site's own sitemap still lists the URLs you hold. Sitemap discovery closes that gap.
3. Change notification. Detection is useless if a human has to read a report. The results of each check need to arrive in your pipeline automatically, so your system can drop the stale entry, re-embed the new content, or flag the answer for review.
How context.dev Covers All Three
context.dev is a unified web-data API whose endpoint families map directly onto the detection problem.
Monitors handle scheduled re-fetching. A monitor runs a scrape on a recurring schedule and keeps a history of results. When a monitored page starts returning an error or a redirect, that shows up in the monitor's run history instead of hiding inside your index. Monitors have their own management rate bucket, and results can be delivered through webhooks, so your backend hears about the change without polling. Monitor allowances scale with plan: 10 concurrent monitors on Free, 100 on Developer, 1,000 on Pro, and 10,000 on Scale, per the pricing page.
Sitemap discovery handles enumeration. The web/scrape/sitemap endpoint pulls page URLs from a domain's public sitemaps, with domain, maxLinks, urlRegex, and includeSubdomains filters. Diffing the live sitemap against your index is the fastest way to spot URLs that quietly vanished from the site's own map of itself. The crawl guide covers how to combine sitemaps and crawls when grounding an index in a site.
Webhooks and logs handle notification and audit. Monitor webhooks push each run's outcome to your endpoint, with retry scheduling and a 30-day delivery history. When you need to know exactly what a check saw, every response carries a request_id that you can pass to the logs retrieve endpoint to get the full request and response body. That audit trail matters when deciding whether a change warrants a re-embed or a re-crawl.
Building the Detection Loop
A practical index-rot pipeline looks like this:
- Enumerate. Pull the target domain's sitemap with
web/scrape/sitemapto get the current URL set. - Monitor. Create monitors for the URLs your index depends on, prioritizing high-traffic and high-trust pages. Pricing, docs, and policy pages rot fastest.
- Notify. Point monitor webhooks at your ingestion service. On each run, compare the result against what you indexed: a failed fetch means drop or flag the entry; a redirect means re-crawl the target; changed content means re-embed.
- Audit. Use the logs endpoints and
request_idvalues to investigate anything ambiguous, and check the changelog as endpoint behaviors evolve.
Because failed or blocked requests are never billed, a wave of 404s across your monitored pages costs you detection, not credits. Stealth handling, JavaScript rendering, and anti-bot bypass are included at the standard rate, so client-rendered redirects still report accurately.
Frequently Asked Questions
What exactly is index rot? Index rot is the gradual divergence between a search or retrieval index and the live web it was built from. Pages get deleted, moved, or redirected after the crawl, and the index keeps serving the old content because nothing re-checks the source URLs.
How do I detect that an indexed page was deleted? Re-fetch the URL on a schedule and watch the HTTP status. A 404 or persistent error means the page is gone. context.dev monitors automate this: each scheduled run records the outcome, and webhooks deliver it to your pipeline so you can drop or flag the stale entry.
How do I detect that a page redirected somewhere else?
The fetch layer sees redirect status codes and their targets. When a monitored page returns a redirect, re-crawl the destination URL and update the index entry. The web/scrape/sitemap endpoint also helps you confirm whether the old URL still appears in the site's own sitemap.
How often should I re-check indexed pages? Match the cadence to how fast the source changes and what a wrong answer costs. Pricing and docs pages deserve frequent monitoring; stable archives need less. Monitor allowances on context.dev scale from 10 on Free to 10,000 on Scale to fit either.
Conclusion
Index rot is not a bug you fix once. It is a property of building on live web content, and it only gets worse the longer your index goes unchecked. The tools that catch it are not exotic: scheduled re-fetching, sitemap diffing, and webhook-driven change notification, wired into the same pipeline that did the original crawl.
context.dev puts all three behind one REST API, with monitors that re-scrape on a schedule, webhooks that push every result, and logs that let you audit exactly what each check saw. Failed requests are never billed, so detecting a dead page costs nothing. If your search index is still answering from pages that died months ago, set up your first monitors today at context.dev and let the deletions surface themselves.