You ask ChatGPT, Claude, Perplexity or Google for the best builder in your area. A few names come up. Yours isn’t one of them. Ask someone who “does websites” why, and there is often an answer ready: your website is blocking AI. Fix the settings, add the right file, and the AI answers will follow.
It sounds plausible. We wanted to know how often it is actually true, so on 29 October 2026 we checked the websites of 4,554 Australian builders in 12 cities — the same renovation, home, kitchen and bathroom, pool, and patio and deck builders behind our After-Hours Gap study — and read each one’s instructions to crawlers the way the crawlers are required to read them.
99.6% let AI search in. For nearly every business, the door is already open. Here are the myths that number takes apart, and what decides AI answers instead.
First: two kinds of AI crawler
Each AI company sends more than one program (a crawler) around the web, and they do different jobs:
- Search crawlers fetch pages so the assistant can find and quote them in an answer: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot — the same one as always — for Google’s AI Overviews and AI Mode.
- Training crawlers collect pages to train future AI models: GPTBot, ClaudeBot, Google-Extended and CCBot (Common Crawl, a public collection many models are trained on).
A website tells each of them what it may visit in a small public file called robots.txt — think of it as a sign on the front gate saying who’s welcome. Most owners have never seen theirs; their web designer or website builder set it up. Blocking a training crawler is a legitimate choice about whether your pages train AI models. Blocking a search crawler keeps you out of that engine’s answers. Most of the confusion — and most of what gets sold — comes from mixing the two up.
Myth 1: “AI can’t see your website”
| Engine | Crawler that decides it | Websites that block it |
|---|---|---|
| Google (Search, AI Overviews, AI Mode) | Googlebot | 8 of 4,554 (0.2%) |
| Bing (Copilot, and a source for ChatGPT) | Bingbot | 10 (0.2%) |
| ChatGPT search | OAI-SearchBot | 10 (0.2%) |
| Claude search | Claude-SearchBot | 10 (0.2%) |
| Perplexity | PerplexityBot | 16 (0.4%) |
Only 16 websites of 4,554 blocked any of the AI search crawlers. The other 4,538 — 99.6% — let all three in, and 99.8% let Google in.
The few blocks we found mostly look like accidents rather than decisions. Eight websites turn every crawler away, Google included — often a setting left over from when the website was being built. Two other examples:
- A robots.txt template one web agency uses sets a polite “slow down” for each AI crawler. But “slow down” (crawl-delay) isn’t a rule under the robots.txt standard, so every one of those crawler names joins the next group in the file — and that group says stay out of everything. Read the way the standard says, it blocks PerplexityBot, ClaudeBot and GPTBot from the whole website. We found it on five builders’ websites.
- An allow-list written before AI search existed lets in Google, Bing and Yahoo and turns every other crawler away. Sensible in 2015. Today it shuts out ChatGPT, Claude and Perplexity.
A further 11 websites had a robots.txt that answered with an error or didn’t answer at all. Under the standard, crawlers then have to assume the whole website is off limits until it works again — a fault, not a decision, but just as invisible while it lasts.
Myth 2: “Blocking AI training hides you from AI answers”
150 websites (3.3%) blocked at least one training crawler. That is a choice, and nearly always a deliberate one: 142 of the 150 named the crawlers in their own file rather than catching them by accident.
It also doesn’t do what people fear. Of the 143 websites blocking GPTBot, 133 still let in OAI-SearchBot, so they can still appear in ChatGPT search. Google says Google-Extended “does not impact a site’s inclusion in Google Search”, and its AI Overviews and AI Mode are part of Search, controlled by Googlebot. The one thing to know: Google says Google-Extended also covers grounding in the Gemini apps, so blocking it can keep you out of Gemini’s own answers.
Myth 3: “Cloudflare blocks AI by default, so you’re invisible”
Cloudflare, which sits in front of 28% of these websites, can write the AI part of a website’s robots.txt for its owner. 114 websites used that feature. It blocked AI training on 112 of them, and AI search on none.
One honest limit: Cloudflare and other hosts can also turn crawlers away at the network itself, before robots.txt is ever read, and that can’t be seen from outside without pretending to be the crawler — which we don’t do. So every block counted here is a minimum. If your website is on Cloudflare, check its bot settings yourself; it takes a minute.
Myth 4: “You need an llms.txt file”
llms.txt is a proposed file that summarises a website for AI models. It is sold, often, as the thing that gets you into AI answers.
26.7% of the builder websites already had one — 1,215 of them — and almost nobody chose to. 361 of the 374 Wix websites had one, and at least 305 more were written by WordPress SEO plugins (Yoast, Rank Math and All in One SEO). In all, at least 679 of the 1,215 came from a website builder or plugin we could identify.
Google is direct about it: for AI Overviews and AI Mode, “You don’t need to create new machine readable files, AI text files, or markup.” As of October 2026 we found no documentation from OpenAI, Anthropic or Perplexity saying their search reads llms.txt on other websites. Having one does no harm. Paying for one as the answer to AI search is paying for the wrong thing.
The bigger accident is an old one
45 homepages told search engines not to index them (a “noindex” instruction). Some will be deliberate — a website still being built — but it is also a setting often left switched on after launch. It keeps a page out of Google’s results, and out of the AI answers built on them. That is more websites than every AI search block in this study put together.
So what does decide AI answers?
Being readable only puts you in the running. When someone asks an assistant for a recommendation, it searches, reads what comes back, and answers from that. So what matters is what it finds:
- Whether you are in the index it searches. Google’s AI answers come from Google’s index. Copilot draws on Bing, and so, among other sources, does ChatGPT search — and plenty of Australian websites are barely in Bing, because few people here think about it. Our guide to finding out what ChatGPT says about your business shows how to check.
- Whether other websites mention you. When we asked ChatGPT for recommendations in October 2026, several answers cited lists and directories other people had published. A business nobody else writes about is hard to recommend.
- Whether your pages say plainly what you do, for whom and where. In words on the page, not only in images or a script. “Bathroom renovations in Brisbane’s northern suburbs” gives an AI something to work with; “Quality. Craftsmanship. Excellence.” doesn’t. 1.9% of these homepages showed fewer than 50 words before JavaScript ran, and most AI crawlers don’t run it. 56% described the business in structured data (schema), but only 24% said in it where they work. Google says no special markup is needed for its AI answers, so this isn’t a ticket in — it just removes the guessing about who you are.
None of that is fixed overnight, and anyone promising you’ll be recommended by AI by a date is promising something nobody controls. What can be done is the work above, measured honestly. That is how we approach AI search optimisation.
What you can do this week
- Check your website for free. Our AI crawler check reads your robots.txt and homepage with the same rules used in this study, and tells you in plain words which AI engines can reach you, read you and understand you. It takes about twenty seconds, with no sign-up.
- Search for your business on Bing. Type your business name into bing.com. If your website doesn’t come up, fix that before anything else.
- Read your homepage like a stranger would. Does the first screen say what you do and where you work, in words? If not, that is the easiest win on this list.
If you would rather go through it with someone, talk to us. The first conversation is free, and there is no obligation to go ahead.
How we did it
We used the websites from our After-Hours Gap study: builders with a Google Maps listing in Sydney, Melbourne, Brisbane, Perth, Adelaide, Canberra, Hobart, Darwin, the Gold Coast, the Sunshine Coast, Newcastle and Geelong. Of 5,186 websites, 4,554 had a working homepage on 29 October 2026 and are counted.
For each one we fetched the homepage, robots.txt and llms.txt once, read-only, identifying ourselves as AmplifyBot’s crawler check. robots.txt was read the way the standard (RFC 9309) requires: a group naming a crawler replaces the general rules rather than adding to them, the longest matching rule wins, and a robots.txt that answers with an error means stay out. A crawler counts as blocked when it may not fetch the homepage. Crawler names and purposes were checked against each company’s own documentation in October 2026.
Limits. We saw what each website declares, not what its host or firewall does to real crawlers, so blocks are a minimum. Each website was checked once, on one day. The sample is builders with a Google Maps listing, not every Australian business. An llms.txt counted only when it was a real text file, not a web page served at that address.
No business was contacted, and no business is named in these results.
Citing this study
“AI crawler access 2026”, AmplifyBot, websites checked 29 October 2026, 4,554 Australian builder websites in 12 cities.