Your AI crawler rules may block a visitor you wanted.

Read the actual crawler rule before changing it. In our historical sample, publishers wrote different rules for different AI-related visitors. That does not tell us whether the owner intended the difference, whether a request was actually blocked, or how many visits resulted.

A website can publish a robots.txt file at its root. The file communicates crawling preferences; it is not a password or an access-control system. Some sites do not have a readable file. Others rely on a host or plugin to generate it. The useful first question is what your public file actually says today.

Our historical measurement found root-level disallow rules for ChatGPT-User in 121 of 1,020 readable files, compared with 4 of 1,020 for OAI-SearchBot. The shares round to 11.9% and 0.4%. A large observed difference is clear; a precise multiple is much less certain because the second count is so small.

OpenAI's crawler documentation, checked on 25 October 2026, distinguishes search, training and user-triggered retrieval. OAI-SearchBot supports ChatGPT search results. ChatGPT-User handles some user-requested visits and is not the control for search inclusion; robots rules may not apply to those requests. Both descriptions are more precise than saying one visitor sends readers while the other merely stores pages.

This article separates the original sample from the current operating guidance. The dataset has not been rerun or re-dated as part of this editorial update. Use the crawler-policy tool to inspect a public policy, then check server or edge logs if your question concerns actual requests.

How much more often, exactly

121 of 1,020 files against 4 of 1,020. Divide those and you get 30, which is the best single guess and is nowhere near as solid as it looks, because it rests on a denominator of four files.

So here is what we can actually stand behind. Take the 1,020 files, draw 1,020 of them back out at random with repeats allowed, count both crawlers again, and do that twenty thousand times. That is a way of asking what this measurement would have said if a different thousand publishers had been picked, and it assumes nothing about the two counts moving together — which matters, because they are counts over the same files, and three of the four sites that block OAI-SearchBot block ChatGPT-User as well. Nineteen times in twenty, the answer lands between 14.5 and 134. In one resample in fifty, no site at all blocks OAI-SearchBot and the multiple has no top end.

A second method, which shares no assumption with the first: take the range each percentage is separately known to and divide the low end of one by the high end of the other. That is deliberately too wide, and it gives 9.9 to 127.

One method gives a lower limit above ten; the other gives 9.9, which is close to ten but does not literally clear it. The defensible summary is a large difference with an uncertain ratio. The observed shares and raw counts carry that finding better than a sharp multiple. A point estimate near 30 is not an exact population result.

What the whole sample looks like

And by crawler:

The last column is the range each figure is really known to. It is the reason the bottom four rows should be read as "almost nobody", and not as an ordering.

What re-measuring changed, and what it did not

This was first published on 2026-09-01 from 106 files, fetched one at a time. Since then our own crawler has been reading the same kind of file every night, so the same question could be asked again of nearly ten times as many publishers without asking anybody for anything new.

Nothing turned over. The headline share — publishers with no rule at all — moved from 76% to 76.1%. The two figures that moved most, supplier wording from 34% to 29.9% and the historical answer-related category from 14% to 12.1%, both moved less than the first sample's own margin of error, which was about eight points wide.

What did change is precision. At 106 files these shares were known to within roughly eight points; at 1,020 they are known to within two or three.

And the multiple at the top of this note is gone. It read 15× — 15 files of 106 against 1 of 106 — which is the right division of the right counts and still should not have been printed, because the denominator was a single file. At 1,020 files the point estimate is 30 and the range runs from about 10 to well over 100, so 15 is inside a wide uncertainty interval, rather than disproved by the larger sample. Neither point estimate deserves a precise, universal headline. A number that survives being divided correctly can still be a number nobody should have stated, and the tell was in the first sample all along: 1 of 106 has a range of 0% to 5%, and no ratio built on it could have been worth two significant figures.

The check we had to do first, and nearly got wrong

A bigger sample is only better if it is a sample of the same thing.

Our crawler works through its list least-recently-tried first, so reading "whatever it has collected so far" is a sample of whoever it happened to reach, which is not random with respect to anything. An earlier pass on this same store held 597 publisher files where all 597 came from one of the two ad networks and none from the other. That is invisible in a total. It would have published one network's customers as the whole list.

So this measurement does not read everything in the store. It reads only the 1,200 domains that were drawn once, in advance, with each network given its share of the list, and only the files collected on a single day when all 1,200 were tried. Eleven other files in the store are readable and were left out because they are not part of that draw. All eleven are from one network, and all eleven say nothing about AI crawlers — so including them would have nudged the headline upwards using files that nothing chose at random.

It matters here more than it usually would, because the two networks' publishers genuinely differ: 88% of files from one of them say nothing about AI crawlers, against 60% from the other. Re-weighting so each network counts for its share of the full list, rather than its share of the files that answered us, moves every figure in the table above by less than half a point.

Why the numbers come out this way

Nearly a third of the historical files carried wording consistent with a supplier template, such as an ad-network block or a plugin-generated list. This is evidence about the text, not the publisher's state of mind. The owner may have accepted it deliberately, delegated the decision, or overlooked it. We did not interview owners, so we cannot distinguish those possibilities.

The historical classification found at least one name absent from the provider reference list in 16.4% of files. In 2.3%, every explicitly named AI-related agent fell into that category. Absence from a reference list is a reason to investigate, not proof that no client anywhere uses the name or that a rule can have no effect. Current provider names and a historical classification need separate dates.

The part that is easy to get wrong

Naming a crawler is not blocking it.

The file is read in blocks. A block is one or more lines naming crawlers, followed by the rules that apply to them — and the rules only start once the run of names ends. The standard that defines this, RFC 9309, puts it plainly: a group can name multiple agents before its rules begin. The RFC 9309 specification defines how matching groups and paths are interpreted.

So a file that stacks thirty crawler names on top of one permissive rule has permitted all thirty. It reads as a wall and works as an open door. 2.7% of this sample does exactly that with at least one crawler.

That is why the figures above are worked out rather than counted. Searching the files for User-agent: GPTBot would have reported those files as maximally strict when they are the opposite — and it is the mistake anyone checking their own file by eye will make.

What to do about it, if anything

There is a real decision here and it is not the one the templates make.

Make separate choices for model training, search discovery and a user asking an assistant to open a page. Those purposes can use different controls. Check the provider's current terminology rather than assuming every name containing “AI” has the same job.

For example, Google's common-crawler reference explains that Google-Extended is a product token, not a separate HTTP visitor. Its stated Gemini uses differ from Google Search inclusion and ranking. A list of names in a file cannot show which requests reached your server.

A practical policy can allow search-related crawling while declining specified training uses. Another publisher may choose a broader restriction. Neither choice guarantees a referral, payment, licensing agreement or removal of material already used elsewhere. Our measurement does not determine which policy is commercially best for you.

Use the free analysis workspace to state your website and question. Read the public policy, identify who controls each managed section, and keep a copy before changing anything. A hosting dashboard may enforce rules that are not visible in the file, so a complete review needs both the declaration and the actual serving configuration.

Limits

The published crawler-policy dataset contains the historical tables and collection scope. The current provider references in this article were checked on 25 October 2026; that does not refresh the dataset.

This is 1,020 readable files, from a sample of 1,200 drawn with a fixed starting word from a list of 16,076 domains that two ad networks publish as their own publishers. It describes small, ad-funded content businesses and nothing more. It does not establish the behavior of publishers outside those networks, enterprise news organizations, stores or the wider web. We have not surveyed every other published study or established that all large publishers behave differently.

180 of the 1,200 gave us nothing we could read. They are left out rather than counted as permitting everything, because a refusal — an HTTP 403, which 107 of them gave — is a fact about a firewall and not about a policy, and counting them the other way would have moved the headline in the direction that flatters it. Ten of the 180 answered normally with an empty file; those are left out too, on the same rule the first measurement used, and counting them as silent instead would put the headline share at 76.3% rather than 76.1%.

The list we drew from is publishers on two ad networks, not the web. A publisher with no ad network is not in it.

What we measured is what the files say, not what happens next

This counts rules written in files. It does not measure whether any AI company obeys them, and one of them says plainly that it may not.

The current provider descriptions distinguish user-triggered retrieval from automatic discovery. A written disallow rule may not be the effective control for every user-requested visit. That does not change the observed text in the historical files; it changes the consequences we can reasonably infer from it.

In particular, allowing a search crawler does not guarantee a link or a visit. Blocking a training crawler does not establish that all AI-related access has stopped. A crawler can also be prevented by a firewall even when robots.txt appears permissive. These are different observations and need different evidence.

A real platform change that makes the distinction useful

Cloudflare's managed robots.txt documentation, updated on 22 October 2026, describes managed directives generated from Search, Agent and Training preferences through Bot Preference Sync. This is a current example of a platform turning a dashboard preference into public text. It does not mean that the historical supplier signatures in our dataset came from that particular feature.

The operational lesson is to inspect the served file, not only the copy in your code repository. A managed section can be added before your origin's text. Conversely, an all-allow configuration can leave the origin file unchanged. The relevant source of truth for a crawler is the response it receives, together with the provider's documented behavior and any separately enforced access rule.

Do not remove a managed block blindly. Identify the setting that produces it, decide whether it reflects your preference, and make the change in the system that owns it. Editing an origin file may leave an edge-managed section untouched. Editing a generated file by hand may also be reversed by the next plugin or platform update.

Test one declared policy without claiming a traffic result

Start with one public URL that readers should be able to access. Save the current robots response, its status, the date and the system believed to manage it. Check the root path and an actual article path. A root-level finding is useful, but a narrower path rule can change the answer for an individual page.

Then write down the purpose you are trying to permit or restrict. If the goal is search discovery, use the provider's search-related control. If the goal is a specific training opt-out, use the documented training control. Avoid translating a broad business goal directly into a long list copied from a random repository.

Read the result as a crawler would. Select the relevant agent group, apply its path rules and note any uncertainty. A file containing a name is not automatically a block. An allowed path is not proof that a request was made. Preserve both the raw response and your interpretation so someone else can reproduce the check.

After a change, fetch the served file again. Check for duplicate managed sections and unintended restrictions on ordinary search crawlers. Ask the platform owner to explain a rule you cannot trace. Do not promise an immediate ranking change: a crawler still needs to revisit, and discovery does not ensure selection for an answer.

For actual enforcement, inspect server or edge request records with the appropriate bot-verification method. A user-agent string can be imitated. Common Crawl's CCBot reference explicitly warns about impersonation and describes verification information. The same general distinction matters whenever a dashboard labels traffic using only a claimed visitor name.

Keep provider roles in a small maintenance register

Record the bot name, stated purpose, official documentation, date checked, current decision and the person or system responsible for changing it. Keep the historical observation in a separate column. That prevents a provider's new terminology from silently rewriting what an old sample measured.

Anthropic's crawler guidance distinguishes model-development collection, user-directed retrieval and search discovery. Perplexity's crawler guidance also separates its search crawler from a user-directed fetcher. Their current descriptions are useful controls to consult, not evidence that every historical request obeyed a file.

Review the register when a provider changes its documentation, when your hosting platform changes managed settings, or when observed requests do not match your expectation. Do not treat an old crawler list as a permanent security boundary. A small, dated list of choices you understand is more useful than a long list with no owner.

The next step is to inspect your own declared policy and identify one uncertainty worth resolving. Our historical sample can show that inconsistent-looking declarations exist. It cannot decide your licensing position, predict a visit, or replace evidence from the system actually serving your pages.