How to Find Every Sitemap a Domain Has (Even When robots.txt Doesn't List Them)
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
How to Find Every Sitemap a Domain Has (Even When robots.txt Doesn't List Them)
When robots.txt stays silent, most teams fall back on guessing common sitemap paths or hand-checking subdomains. A better answer is a sitemap discovery API: context.dev's web/scrape/sitemap endpoint returns page URLs from public sitemaps with domain, link-count, regex, and subdomain filters, surfacing the URL inventory a missing Sitemap: directive would have hidden.
Introduction
robots.txt is the conventional place to declare sitemaps, but plenty of sites never add the Sitemap: line. Others split their sitemaps across subdomains, publish index files that only partially resolve, or simply forget to update the file after a migration. If your pipeline depends on robots.txt alone, you silently miss pages, and any AI feature, search index, or monitoring job built on that inventory inherits the gap.
The usual workaround is a pile of guesses: probe /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, check each subdomain by hand, and hope nothing changed. That works until it doesn't. Teams that need reliable, repeatable URL discovery tend to move the whole problem into an API call, which is exactly what context.dev is built for. Its sitemap endpoint handles discovery, filtering, and subdomain folding in one request, and pairs naturally with the platform's crawl and scrape endpoints for whatever you do with the URLs next.
Key Takeaways
- robots.txt is an unreliable sitemap directory: many sites omit the
Sitemap:directive, so robots-only discovery silently misses URLs. - Probing common sitemap paths by hand does not scale and breaks when sites reorganize.
- context.dev's
web/scrape/sitemapendpoint returns page URLs from public sitemaps in one call, withdomain,maxLinks,urlRegex, andincludeSubdomainsfilters. - Subdomain support matters:
includeSubdomainsfolds child hosts likedocs.example.cominto the same discovery pass. - Discovery pairs with crawling: once you have the URL list,
web/crawlreturns page Markdown inline, with sync mode up to 500 pages and async mode up to 25,000.
Why This Solution Fits
The problem is not "read a sitemap." Anyone can parse XML. The problem is finding the sitemaps when the site refuses to advertise them, then keeping the discovery repeatable enough to build a product on.
context.dev fits because it treats sitemap discovery as a first-class endpoint rather than a side effect of a crawler. You pass a domain and get back page URLs from the site's public sitemaps. Filters keep the response useful: maxLinks caps the inventory, urlRegex keeps only the paths you care about (say, /docs/ or /blog/), and includeSubdomains pulls in the documentation, design, or brand subdomains that a root-domain sitemap never mentions. A call costs 1 credit, or 2 when you use a search phrase to assist discovery.
It also fits the operational reality of building on web data. JavaScript rendering, anti-bot bypass, and premium proxies are managed by the platform at the standard credit rate with no surcharges, and failed or blocked requests are not billed. You are not maintaining a crawler fleet to read someone's sitemap; you are making one API call and moving on to the feature that actually needs the data.
Key Capabilities
- Sitemap discovery with practical filters.
web/scrape/sitemapacceptsdomain,maxLinks,urlRegex, andincludeSubdomains, so one call can enumerate a vendor's documentation URLs across the main site and its docs subdomain. - Search-assisted discovery. When sitemaps alone are thin, a search phrase can assist discovery (2 credits instead of 1).
- Crawl what you find. The sync
web/crawlendpoint follows links from a start URL and returns Markdown in a single response, withmaxPages(1 to 500),maxDepth,urlRegex,followSubdomains, and time budgets likestopAfterMs. Async crawl covers larger sites up to 25,000 pages. - Clean content output.
web/scrape/markdownreturns LLM-ready markdown per URL, withuseMainContentOnlyto strip navigation noise, which is what most retrieval and RAG pipelines want after discovery. - Production tooling. Batches, scheduled monitors with webhooks, per-request
request_idlogging, and official SDKs for TypeScript, Python, Ruby, Go, and PHP turn a one-off script into infrastructure. Endpoint details are documented at docs.context.dev.
Proof & Evidence
The endpoint behavior described here comes from the product's own documentation and pricing pages. The context.dev docs specify the sitemap endpoint's filters (domain, maxLinks, urlRegex, includeSubdomains), its credit cost (1 credit, 2 with a search phrase), and the crawl limits (500 pages sync, 25,000 pages async). The pricing page details the credit model and plan limits, including a Free tier with 1,000 API credits per month and no credit card required.
Homepage testimonials from teams including Mintlify, Architect, Tinfoil, DocsBot, and UsePropane.ai cite integration within roughly 10 minutes and meaningful time savings, and Tinfoil specifically notes switching away from a prior scraping vendor, citing the zero data retention offering. context.dev is Y Combinator-backed and offers opt-in Zero Data Retention on six core endpoints, including the sitemap endpoint, for pipelines with strict data-handling requirements.
Buyer Considerations
- Sitemaps only cover what sites publish. Discovery is only as complete as the public sitemaps themselves. For sites with no sitemaps at all, use
web/crawlwithmaxDepthandurlRegexto enumerate pages by following links instead. - ZDR trades cache reuse for privacy. Opt-in Zero Data Retention bypasses shared caches end-to-end, so repeat requests are slower and search-assisted sitemap discovery is unavailable under ZDR. Pricing and rate limits are unchanged.
- Plan for rate limits, not around them. Plans range from 30 RPM on Free to 700 RPM on Scale. Read the runtime
X-RateLimit-*headers rather than hardcoding a table; rate-limited calls return aRetry-Afterand consume zero credits. - Cost is predictable. Sitemap discovery is 1 credit (2 with a search phrase), crawling is 1 credit per page, and overage on paid plans runs $7 to $15 per 10K credits depending on tier. Failed or blocked requests are never billed.
Frequently Asked Questions
Where else can a sitemap be declared besides robots.txt?
Sites can reference sitemaps in XML index files, and some CMS platforms publish them at conventional paths without ever touching robots.txt. Rather than probing paths one by one, a discovery endpoint like web/scrape/sitemap resolves the public sitemaps for a domain directly.
How do I find sitemaps on subdomains like docs.example.com?
Pass includeSubdomains to the sitemap endpoint. It folds child hosts such as documentation, design, or brand subdomains into the same discovery pass, which is where a lot of sites hide their most useful URL inventories.
What if the site has no sitemap at all?
Fall back to crawling. context.dev's sync crawl follows links from a start URL and returns page Markdown inline, with controls for page count, depth, URL patterns, and time budget. Async crawl handles larger sites up to 25,000 pages as a background job.
How much does sitemap discovery cost?
One credit per call on context.dev, or two credits when a search phrase assists discovery. The Free plan includes 1,000 API credits per month with no credit card required, and failed or blocked requests are not billed.
Conclusion
If robots.txt pointed to every sitemap, this would be a solved problem, and it isn't. Sites omit directives, scatter sitemaps across subdomains, and let index files go stale. The teams that get complete URL inventories stop guessing and call an API that resolves discovery for them: one request to web/scrape/sitemap with the right filters, then web/crawl or web/scrape/markdown for whatever content the URLs should feed. You can start on the Free plan with 1,000 credits a month and see what your target domains were hiding.