Your AI crawler list may still contain outdated names.

A name in robots.txt does not necessarily block anything. Among 1,020 readable files in our historical publisher sample, 244 named AI crawlers. Of those, 167 also carried names classified as legacy by the registry used for that measurement. That is 167 / 244 = 68.4%, rounded to 68%.

The denominator matters. This is not 68% of all publishers, all sampled sites, or even all files that actively block a current crawler. It is the share within the 244 files that named crawlers.

Names and effective rules are different measurements

The saved sample separates three groups:

The last two rows add to 244. Their names alone would not tell you which group a file belongs to. A block listing several user agents above an Allow: / can name all of them while permitting access.

These are root-path evaluations. A file may also contain different rules for another path. We did not test every URL or observe every crawler’s requests.

The downloadable dataset retains named tokens, blocked tokens, network and legacy-name flags separately. Its method identifies the historical frame, seed, exclusions and collection date. A roster frame of advertising-funded publishers is not a representative sample of the whole web.

Anthropic’s documented names

Anthropic’s own crawler documentation, checked again on 20 October 2026, distinguishes three uses:

The same page does not list anthropic-ai or claude-web. The saved registry flagged those strings as legacy. That observation does not prove no software still uses them, or establish whether a particular publisher’s work reached a model.

In this snapshot, 163 files carried the first legacy string and 138 carried the second. These categories overlap: adding them would double-count files. The resolved current-token count for ClaudeBot is 74, not the 101 previously stated in this note.

Blocking a legacy string also does not imply that a file omitted the current token. Both can occur in the same file. We therefore withdraw the earlier inference that most owners believed they had blocked Anthropic but had left it open. Neither owner beliefs nor crawler compliance were measured.

What this says about referrals

A training control and a user-directed retrieval control serve different purposes. The historical sample blocked ChatGPT-User in 121 files and OAI-SearchBot in four. That comparison is between two retrieval/search tokens; it does not show that retrieval crawlers were blocked more often than every training crawler. For example, GPTBot was blocked in 206 files.

A robots.txt policy is not a measured traffic loss. A successful request is not proof that content was licensed. A missing token is not proof that every bot can enter: a firewall, authentication or another rule can also control access.

Google additionally provides a Search Console control for Search generative AI features. It is separate from Google-Extended, and its private property setting cannot be determined from this public file. The crawler tool explains that distinction without guessing your setting or estimating missing clicks.

A practical check

Read the token against the operator’s current documentation, evaluate the rules for the path you care about, and inspect actual server logs if you need to know what fetched it. Record those as separate observations. Do not translate a name list into a universal AI opt-out, licensing readiness score or revenue forecast.

Evidence grade: A. The historical primary-data snapshot and current operator documentation support the stated counts and definitions. They do not establish present-day adoption across the web, user intent or economic impact.

A legacy flag is a research classification

The word legacy describes how the saved crawler registry classified a token for this study. It is not a claim that a string can never appear in a request or that every publisher using it made a mistake.

An operator may rename a crawler, keep an old compatibility name or change a product's role. Third-party templates can preserve old strings. Software outside the operator's documented system may also send a familiar user-agent value. The public name alone does not identify the requestor.

Check the current operator reference and the purpose of the rule before deleting it. Adding a current documented token may be appropriate for a chosen policy, but removing an old one without understanding the surrounding configuration can change behavior you intended to preserve.

The useful research claim is narrower: these historical files contained strings classified in this stated way. The useful maintenance question is whether the live rules still express the publisher's actual current choices.

The same name can appear in an allowing rule

A name count detects an explicit mention. It does not evaluate whether the relevant crawler can fetch a particular path. A group can name several agents and permit the root, block a narrower directory or include more specific rules.

The Robots Exclusion Protocol defines groups and path matching. A parser needs to evaluate the applicable rules rather than treat the text User-agent followed by a known name as a block.

For a chosen example, a site names a documented crawler but uses an allow rule for the root. The name appears in a text search while the evaluated root policy permits access. Another site can omit that explicit name but apply a wildcard block. A name-only search would miss that effective restriction.

Those examples explain why named tokens and resolved root blocks are separate dataset fields. A useful interface should preserve that distinction, especially when a visitor expects “found in robots.txt” to mean “blocked.”

Root access is not every URL

A root-path test asks about the site's / path. A publication can permit the root while restricting an archive or another directory. It can also use a more specific exception for a path below a broader restriction.

If the decision concerns an article, evaluate that article's path. If it concerns a collection, inspect representative paths and explain the sample. Do not turn one root evaluation into a statement that every page has the same policy.

The historical study intentionally reports root-level blocks for comparison. That makes its unit reproducible. It also limits what can be inferred about individual content URLs. A crawler-policy score that hides the tested path would lose an important part of the method.

Path behavior can change when a site migrates its URL structure. A rule that once applied to /blog/ may no longer cover articles moved elsewhere. Keep the intended content scope in the maintenance record rather than only the agent list.

Search discovery, training and user fetching need separate choices

OpenAI's crawler reference distinguishes GPTBot, OAI-SearchBot and ChatGPT-User by purpose. Search discovery is not the same operation as collecting training material or retrieving a page at a person's request.

Perplexity's crawler documentation likewise distinguishes its search crawler from user-requested fetching. Operators can describe different handling for user-triggered requests, so do not assume every token follows an identical control model.

A publisher may deliberately choose different policies for those uses. Blocking one token and allowing another is not automatically contradictory. It may reflect a desire to remain discoverable while limiting another use.

The research does not choose that policy for every site. It helps identify which declared rule applies to which documented purpose. A list copied from another publication cannot supply your own decision about discoverability, licensing or public access.

Keep declaration, observation and enforcement separate

A robots file states a preference for clients that follow the relevant rules. Server logs show observed requests under the logging conditions. An edge or application access rule can deny or permit requests. These are related layers, not interchangeable evidence.

If you need to know whether a crawler visited, inspect logs and verify the requestor where the operator provides a method. User-agent text alone can be spoofed. A request carrying a famous name is not automatically an authenticated visit from that company.

If you need to know whether access was denied, examine the actual response or enforcement logs. A declared disallow does not prove an edge block, and an edge block does not necessarily reveal a corresponding robot preference.

Finally, none of those observations alone proves what a model later used internally. A successful fetch is not a training receipt. A denied request is not proof that no other source contains similar material. Keep claims at the level the available evidence supports.

Reproduce the percentage with its actual denominator

The headline uses one hundred sixty-seven files divided by two hundred forty-four files that explicitly named evaluated AI crawlers. That gives about 68.4%. Dividing the same numerator by all one thousand twenty readable files gives about 16.4%.

Both arithmetic results can be calculated, but they describe different groups. The article's headline is conditional on the naming subset. It is not the share of the whole publisher frame.

The sample also began with one thousand two hundred drawn domains, not all of which yielded a readable robots file. A count based on readable responses does not establish the policy of excluded domains. Do not assign a missing response to the allowing group simply because no block was observed.

The same care applies to overlapping legacy names. One file can carry both strings, so their individual counts cannot be added to recover the number of distinct files carrying either. Use the row-level union or the published distinct count.

What a shared template can and cannot explain

Repeated text across files can be evidence of a common template or generator. It does not establish the exact installation path, whether the owner reviewed it or why it remains in place.

A publisher may have knowingly accepted a vendor default. Another may have edited it. A third may have copied part of the file from a guide. Without interviews, change history or other direct evidence, assigning intent would overstate the snapshot.

Use template patterns to decide where maintenance guidance may help. For example, a provider could make a control's product scope clearer or show the current documented token beside the setting. Those are possible improvements, not measured causes of the historical counts.

If you manage several sites, inspect the final deployed files rather than assume the same setting creates identical outputs everywhere. Different plugin versions, theme code or hosting rules can affect the result. Record the generator and version where known.

A practical crawler-policy maintenance record

List the product or use you want to manage, the operator's current token, the intended paths, the source URL and the date checked. Add the effective public rule and any separate enforcement layer you use.

Mark unknowns. If you do not know who generates a section, investigate before editing it manually. If the operator documentation is unclear about a user-directed fetch, record that ambiguity rather than claim a universal opt-out.

After changing a rule, inspect the live response. Check that it is the intended file rather than an HTML page or a stale cached version. Confirm that unrelated search, image or application resources were not accidentally restricted.

Keep a copy of the prior configuration and a rollback path. A broad wildcard change can affect more than the named AI product. A narrow policy goal should not unintentionally break ordinary discovery or page rendering.

Do not estimate lost traffic from a token count

The historical difference between blocked tokens does not tell us how many visits would otherwise have arrived. To estimate referral effects, you would need actual traffic observations, a suitable comparison and assumptions about other changes.

A later increase or decline in visits can coincide with a policy edit while search demand, content, ranking or the product itself also changes. An ordinary before-and-after chart cannot isolate the policy effect automatically.

Keep impressions, clicks, sessions and revenue distinct. Being mentioned in an answer is not the same as receiving a visit, and a visit is not the same as an ad-earning session. A crawler request is another unit again.

The public crawler tool can help inspect visible policy and historical context. It cannot recover private Search Console settings, authenticate every request or calculate missing revenue from a robots list. Those limits should travel with any finding you share.

Turn the audit into a clear next action

If the token is current and the effective rule matches your goal, keep the evidence record and review it after relevant operator changes. If a legacy token is the only expression of a current goal, check the documented replacement and test the intended policy. If the file is generated elsewhere, change the owning configuration and verify deployment.

Do not remove controls merely to make an audit score greener. Decide the purpose first. A deliberate block can be correct for the publisher even if it reduces a discovery route. A deliberate allowance can also be correct when it fits the publication's aims.

The strongest outcome is an understandable policy: what is allowed, what is restricted, for which use and what the test actually observed. That is more useful than a long inherited name list presented as complete protection from every AI system.

Keep the source versions so future checks can reproduce that policy.