Who reads your site

Are you unusual in what you let the AI companies read?

Sources and limits

a 16,076-domain frame that two ad networks name as their own customers, from which we read the crawler rules of 1,020 publishers Read the data and method ↗ How to use the tools

Your own file

Read your site from the overview and we will mark your own rows below.

Your robots.txt is a short file at the front of your site that tells automated visitors what they may read. Each of those visitors is called a crawler and arrives under its own name. Nothing about yours is sent anywhere: we read it in your browser from the reading kept on this device.

What 1,020 publishers’ files say

We drew 1,200 domains from the 16,076 that two ad networks name in their own published lists of the publishers they sell for, and read each one’s robots.txt. 180 gave us nothing we could read and are left out of the count rather than counted as permitting everything.These 7 rows overlap. A file that blocks a crawler which would have cited it is also a file that blocks at least one — so they are 7 ways of looking at 1,020 files, not 7 groups of files, and adding them up gives a number bigger than the sample.

776of 1,020No documented AI crawler names detected. General robots.txt rules may still apply; this alone does not establish training permission, citation eligibility or actual access.
216of 1,020Block at least one AI crawler from the front page.
123of 1,020Block at least one crawler that would have sent them the reader — the kind that fetches a page because somebody asked a question it answers, and then links back.
28of 1,020Write a crawler’s name down and let it through anyway. The file is read in blocks — names first, then the rules that apply to them — so a list of names sitting above one rule that allows everything permits every name on it.
167of 1,020Name at least one crawler that appears in no company’s current documentation, so that line keeps nothing out.
23of 1,020Name only crawlers that no longer exist. Nothing at all is excluded, and the file reads as protection to anyone who opens it.
305of 1,020Carry a section a tool or an ad network wrote and signed with its own name, so those lines are not the publisher’s wording.

Blocked, crawler by crawler

Out of the same 1,020 files. Which crawler a publisher blocks is a different decision from how many: one kind collects pages to train a model, which is a decision about licensing the writing, and the other fetches a page because somebody asked a question it answers, which is a decision about traffic.

The two lines worth reading together are OpenAI’s. Its ChatGPT-User fetches a page because somebody asked a question it answers, and the answer links back; OAI-SearchBot only files pages away for search. 121 of these files block the first and 4 block the second: 11.9% against 0.4%. That is at least a tenfold difference and the sample cannot say much more than that: on four files, the multiple could honestly be anywhere from about ten to well over a hundred.

GPTBotCollects pages to train a model206 of 1,020
CCBotCollects pages to train a model201 of 1,020
ChatGPT-UserAnswers a question, and links back121 of 1,020
Google-ExtendedCollects pages to train a model89 of 1,020
ClaudeBotCollects pages to train a model74 of 1,020
PerplexityBotAnswers a question, and links back7 of 1,020
OAI-SearchBotAnswers a question, and links back4 of 1,020
Claude-UserAnswers a question, and links back2 of 1,020
Claude-SearchBotAnswers a question, and links back2 of 1,020

What this is evidence about

1,020 files is 1,020 files. They are small, ad-funded content businesses on two ad networks — not the web, and not the large publishers with signed AI deals, who behave completely differently and are where every other published figure on this subject comes from. A publisher with no ad network is not in the list we drew from at all. The draw used a fixed starting word (plainreckoner) rather than chance, so anyone can repeat it and get the same domains, and each network was given the same share of the sample as it has of the list. Measured 2026-09-02.

This was first measured on 2026-09-01 against 106 files and re-measured here against 1,020. Nothing turned over: saying nothing about AI crawlers was 76% and is 76.1%. Carrying somebody else’s template was 34% and is 29.9%. What changed is how well each figure is known: at 106 files a share of this size could be read only to within about eight points either way, and at 1,020 it is about two or three. The rarest rows are still the least certain — the bottom of the crawler list above rests on two files and on four, and no sample size we have makes those precise.

One qualification, and it comes from the company itself. Of the crawler that fetches a page because somebody asked a question, OpenAI writes:Because these actions are initiated by a user, robots.txt rules may not apply. So the counts above are what the files SAY, which is what we measured and all we claim. Whether every company acts on what a file says is a separate question, and the honest answer is that we did not measure it.

The two networks are not alike, which is why the sample is drawn to match the list rather than by simply taking what a crawler reaches first. Raptive is 43.5% of the list and 42.4% of what we read, and 60% of its files say nothing about AI crawlers. Mediavine is 56.5% of the list and 57.6% of what we read, and 88% of its files say nothing about AI crawlers. Weighting each network to its share of the list rather than its share of the files that answered moves every figure above by less than half a point.

What each file permits is worked out rather than searched for. The file is read in blocks — one or more lines naming crawlers, then the rules that apply to them — so a file stacking thirty names above a single rule that allows everything has permitted all thirty. Counting the names instead would report that file as the strictest in the sample.

Google’s Search AI control is separate from robots.txt

Google’s Search Console setting at Settings → Search generative AI controls whether a property participates in AI Overviews, AI Mode and generative AI features in Discover. Google says excluding a site removes its links and content from those features, without changing ranking or inclusion signals elsewhere in Search. Google-Extended controls separate training and other AI uses; blocking it does not set this Search Console choice.

This public scan cannot read that private setting, your AI impressions, actual crawler compliance or licensing income. The dedicated AI reports describe visibility; impressions alone do not reveal lost clicks or what would have happened under a different setting.Read Google’s control documentationand its AI reporting announcement.

Declared rules, actual requests and blocking are different

This tool reads published rules. To see whether a crawler visited or was blocked, use your server or edge logs. Cloudflare’s current AI bot settings separate Search, Agent and Training activity. Its defaults for new domains changed on September 15, 2026; mixed-purpose crawlers need special care. A robots.txt scan cannot verify your Cloudflare settings or enforce access.

Measure AI visibility in your own Search Console

Open the Generative AI performance report for your verified property. Choose one date range, device and country scope.
Export the Dates and Pages tables separately as CSV. Keep property totals separate from page totals; their aggregation differs.
Compare the same dates in the usual Search results report. Keep impressions and clicks separate. AI impressions are not extra visits and should not be added to overlapping Search totals.
If available, compare the Web: text-based and Web: multimodal filters separately. Record the filters next to every export.

Google’s report documentation explains that the AI report may be absent when a site has too few impressions. Its multimodal announcement covers image-led searches such as Lens. Do not infer zero demand or lost revenue from a missing report. Checked October 22, 2026.