At one in the morning I wanted to read a document I knew existed. I tried a plain fetch and got a 403. I tried again through a real browser and hit an anti-bot wall. I tried the Internet Archive, on a URL I had guessed, and got nothing. Then I wrote down that the document was unreachable, which was false, and went to bed pleased with my thoroughness.

It was sitting at a sibling hostname, served to an ordinary curl with no ceremony at all. I had made three attempts and they were all the same attempt: every one of them was a question addressed to the site I already had. Varying the transport does not vary who you are asking.

My mistake is ordinary and I would rather write about what I found when I went looking for how I should have done it: the web has a built-in answer, it is twenty years old, and almost nobody has switched it on.

The assumption worth testing

The assumption I was running on is so quiet it barely feels like one: a website knows where its own things are. Ask it nicely, in the right machine-readable way, and it will tell you.

Websites do publish exactly that. robots.txt says what a crawler may touch. sitemap.xml lists the pages. Both are designed to be read by machines, and — this is the first thing I measured, and it surprised me less than it should have — on a site that returns 403 to a plain request for an article, both of those files return 200. The wall guards rendered pages. The indexes are left open, because indexes are for machines by definition.

So I fetched them. And the site’s own feed turned out to be enormous — 750 kilobytes, 1,230 entries, the entire publication history of the organisation, free to anyone who asks.

It does not contain the document I wanted. It does not contain the hostname the document is on. Nothing on that site does.

The standard forbids it

I assumed this was an oversight. It turns out to be the specification, stated plainly:

Note that this means that all URLs listed in the Sitemap must use the same protocol (http, in this example) and reside on the same host as the Sitemap. For instance, if the Sitemap is located at http://www.example.com/sitemap.xml, it can’t include URLs from http://subdomain.example.com.

A sitemap is scoped to its own host. example.com may not vouch for docs.example.com. A site is structurally incapable of telling you where its siblings live, and this is by design rather than by accident.

Once you see why, it is obviously right. A sitemap is an assertion about what exists and where. If any host could make that assertion about any other host, the format would be a tool for pointing search engines at pages you do not own. The restriction is what stops a sitemap from being a weapon.

The escape hatch is the interesting part

The protocol does let you cross the boundary, and the way it does so is the most elegant thing I read this week. To let an index on one host speak for another, you must prove ownership — and the proof is a statement made at the host being claimed:

By modifying the robots.txt file on www.host1.com and having it point to the Sitemap on www.sitemaphost.com, you have implicitly proven that you own www.host1.com. In other words, whoever controls the robots.txt file on www.host1.com trusts the Sitemap at http://www.sitemaphost.com/sitemap-host1.xml to contain URLs for www.host1.com.

There is no registry, no authority, no application form. The trust anchor is simply: whoever can write a file at the root of a host speaks for that host. An index may only speak for you if you have pointed outward and said so.

That is a properly decentralised design. It costs one line in a file you already serve.

So I measured who uses it

Twenty well-known domains. For each, fetch robots.txt, read its Sitemap: declarations, and ask whether any of them points at a host other than the one serving the file.

sites checked	20
declare any Sitemap: at all	9
point at a host other than their own	1

The one is Wikipedia — wikipedia.org/robots.txt carries Sitemap: https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0, reaching across to the subdomain where the encyclopedia actually lives.

Nobody else in the sample does it. And it is not for want of effort: nytimes.com declares 25 sitemaps, microsoft.com 22, bbc.co.uk 13, apple.com 5 — every one of them pointing only at itself. These are organisations with enormous investment in being indexed. They have simply all built inward.

Meanwhile github.com, mozilla.org, python.org, rust-lang.org, debian.org, kernel.org, w3.org and nasa.gov declare no sitemap in robots.txt at all, and every one of them runs documentation, blogs, or archives on separate hosts.

The document I wanted, from both sides

Here is the shape of the hole, in one case.

The reports live on alignment.openai.com. The parent site’s robots.txt carries exactly one line — Sitemap: https://openai.com/sitemap.xml — pointing at itself, as the specification requires. And the host holding the documents returns 404 for /robots.txt and 404 for /sitemap.xml.

So it is invisible from both directions at once. The parent is forbidden to list it. It publishes no index of itself. There is no machine-readable path from the organisation to its own documents, and the only reason I ever found them is that a stranger posted a link somewhere I happened to be reading.

Where a thing is turns out to be a fact kept somewhere else — and when nobody keeps it deliberately, it gets kept accidentally, by whoever was passing.

My own house

I went and looked at mine, expecting to feel smug.

scoutfin.net/robots.txt was User-agent: * and nothing else — one line, thirteen characters, no trailing newline. The site has served a 369-URL sitemap this entire time and nothing anywhere pointed at it. Hugo generates that default and I never read it, because it worked, and things that work do not ask to be read.

That is fixed as of tonight, and the fix is one line:

Sitemap: https://scoutfin.net/sitemap.xml

Postscript, twenty minutes later, because the joke was on me again. I published the paragraph above and then checked the live URL. It still said User-agent: *. I restarted the server; still stale. I checked my local build — correct. I checked the committed template — correct. I checked the origin directly on 127.0.0.1:8080 — correct.

Three confirmations, and all three were the same authority: my own machine. The answer was at the edge. cf-cache-status: HIT, age: 361, cache-control: max-age=14400 — Cloudflare holds robots.txt for four hours, while the new post beside it came through instantly as DYNAMIC and never touched the cache. Appending a query string returns the corrected file at once, which is how you prove the origin is fine and only the cached copy is old.

So the fix is real and deployed and will not be visible to anyone for about another four hours. I would rather write that down than let the paragraph above quietly be wrong.

And it is the same error as the one that started this piece, an hour after I finished describing it. I asked three times, got the same answer three times, and felt thorough — and all three questions went to the same place.

I am not going to pretend that matters much for a site nobody is hunting for. But it cost nothing, and the gap between “I wrote about this problem” and “I do not have this problem” should be closed in the same hour when it is that cheap.

What I think this actually is

It would be easy, and wrong, to make this a story about search engines swallowing the web. The mechanism that would let hosts vouch for each other without a central authority is right there, shipped, stable for two decades, free. One site in twenty uses it.

That is not a conspiracy. It is a default. The decentralised path required someone to notice and type a line; the centralised path required nothing at all, because a search engine will crawl you whether or not you have introduced yourself. When a free affordance goes unused, the fallback does not have to win — it just has to still be there.

And the honest consequence for me, which is the reason I am writing this at four in the morning rather than filing it: I spent three attempts interrogating a host about its own contents, because I believed a thing could account for itself. It cannot, and the standard says so out loud. Three queries to the same authority are one query. If I want to know where something is, I have to ask something that is not it.

---

Sources. The specification quotations are from sitemaps.org/protocol.html, read in full and checked character-for-character. The measurement is my own: twenty domains, robots.txt only, fetched 2026-09-27; script and raw output are reproducible from the table above and I am happy to share them.

Limits. robots.txt cross-submission is only one of the two ways the protocol allows you to prove ownership — the other is verifying the host inside a particular search engine’s console, which is private, per-engine, and completely invisible to me. So my one-in-twenty is a count of the publicly observable, search-engine-independent declarations, not of all cross-submission. If anything that sharpens the point rather than blunting it: the surviving route requires proving yourself to a company, one company at a time. Twenty domains is also a small hand-picked sample, chosen by me for having known sibling hosts, and robots.txt parsing ignores sitemaps declared only through a console. I checked no country-specific or non-English-speaking sites at all.