Your next customer might not be the one reading your website.
More of the looking around is being handed to software. Find me an air conditioning company in Sydney that does ducted repairs. Check who covers my suburb. Tell me which ones answer after hours.
The person still decides who to ring. The reading is done by an assistant, and an assistant does not look at your website the way you do. It asks your server for the page, takes whatever comes back in that one reply, and works from that.
So we went and looked at what it gets.
On 23 September 2026 we pulled live Google Australia results for 20 trade searches across Perth, Sydney, Melbourne, Brisbane and Adelaide. We took every ordinary business that ranked in the top ten and ran the same checks against all of them. 131 websites, one afternoon, every answer written down.
The result we did not expect was at the front door.
Not one site in the sample uses its robots file, the small text file that tells crawlers what they are allowed to read, to keep AI crawlers out. Eight of them go out of their way to invite them in. And 30 of the 123 sites that answered an ordinary browser refused the identical request when it arrived saying it was ChatGPT's crawler.
What Agentic AI Actually Means
Agentic AI is software that can carry out a task for you, not only write something for you.
There is a ladder here, and most of the confusion comes from people standing on different rungs of it and using the same word.
- A chatbot is a brain in a box. You type, it answers, and that is the end of it. It cannot go and fetch anything.
- An assistant is the same brain plugged into the tools a business already runs on, so it can look things up in your own systems.
- An agent is a brain with hands. It has tools. It can read a file and write one, fetch a web page, call another piece of software, and work through a sequence of steps instead of producing a single answer.
Generative AI writes the email. Agentic AI finds the three air conditioning companies, reads their pages, and drafts the email with their numbers already in it.
That second one is why your website now has a second kind of reader.
Agentic AI vs Generative AI
Generative AI produces content. You ask, it writes.
Agentic AI produces outcomes. It plans the steps, uses tools to carry them out, checks whether the result is any good, and has another go if it is not.
The generative part is still in there as the engine. What gets added is the hands, a memory of what it has already done, and a set of limits on what it is allowed to touch.
For a service business the practical difference is small and specific. A customer using a generative tool still visits your website. A customer using an agent might never open it.
What We Tested
| Element | What we did |
|---|---|
| Date | 23 September 2026, single day snapshot |
| How the sample was picked | 20 trade searches, 14 trades, 5 capital cities, live Google Australia results, top 20 deep |
| Sample | Every ordinary business site that ranked in the top ten. Directories, marketplaces, social platforms, government and education sites were removed, and so were our own builds |
| Sites | 131 Australian trade and service websites. 123 answered an ordinary browser request and 126 answered our audit client, so both counts appear below and each result says which one it is measured against |
| Checks | Whether the page content arrives without a browser, whether there is a phone number an agent can act on, what the site says about itself in structured data, whether a missing page returns a real error, whether robots.txt exists and what it permits, and whether the site serves llms.txt or plain markdown |
| Identity | Every request said who it was. We also repeated the homepage request as an ordinary browser, and again as four named AI agents, and compared the answers |
The Honest Limit
When we requested a homepage as an AI agent, we sent that agent's name in the user agent string. That is the short line of text every request uses to say what software it is. We sent it from an ordinary connection.
The real crawlers publish their internet addresses, and a site running proper bot checking can tell the difference. So a site that refused us might still let the genuine article through.
It is also true that the assistant working on behalf of an actual customer is usually not on anybody's verified list either. It gets treated exactly the way we were.
What We Found
- 1No site in the sample blocks an AI crawler in robots.txt. Not one of the 126 that answered.
- 2Eight sites name AI crawlers in robots.txt, and all eight allow them. One of them, a Perth carpenter, has written comments in the file explaining the decision.
- 330 of the 123 sites that served a browser refused the same request from ChatGPT's crawler, with the server itself returning an error rather than the page.
- 427 refused Anthropic's crawler. 37 sites, 30 percent of the sample, refused at least one of the four AI agents we tried.
- 5The live fetchers mostly got through. Only 4 sites refused PerplexityBot and only 4 refused ChatGPT-User, the agents that go and read a page because someone just asked a question.
- 6114 of 126 deliver their page content without needing a browser to run any code, with a median of 1,052 words of readable text on the homepage.
- 7103 of 126 have a phone number an agent can act on, written as a dialable link rather than as a picture or plain text.
- 885 of 126 describe the business in structured data, the machine readable summary in the page code. Only 30 use the LocalBusiness description, and only 24 publish their opening hours that way.
- 9107 of 126 return a real error for a page that does not exist. Seven return a normal page, which tells an agent that every address it guesses is real.
- 1034 of 126 already serve an llms.txt, a plain text summary of the site written for AI tools. At least ten of those were generated by an SEO plugin, and their first line says so.
- 11One site out of 126 will hand over a clean text version of a page when an agent asks for one. Not five. One.
A Quarter Of Them Shut The Door Without Meaning To
This is the finding worth acting on, because almost nobody it applies to knows it is happening.
What We Did
We asked each homepage for the same thing four times over, changing only the name the request gave for itself.
As an ordinary browser, 123 of the 131 sites answered. As ChatGPT's crawler, 30 of those 123 refused: the server sent back an error code instead of the page. Anthropic's crawler got the same treatment on 27 of them.
We then repeated three of those refusals by hand, to be sure they were real and not a wobble on the day.
Nothing In The Robots File Says So
In every one of those cases the site's own published rules say a crawler may read the page, and the server then refuses to hand it over.
The block is being applied somewhere else. A security plugin or a firewall setting, shipped with a list of known AI agents already switched off, usually installed by somebody trying to stop scrapers and never mentioned to the owner.
What It Costs
The crawler that gets refused is the one that builds the index.
The live fetcher, the one that goes and reads a page because a customer just asked a question, mostly still gets through. But it only ever fetches a page that something already suggested, and the thing that suggests it is the index.
Blocking the crawler that builds the index while allowing the one that reads on request means you keep the doorbell and take out the street sign.
Checking it takes one line in a terminal, and the fix is a setting in whatever plugin or firewall is doing it. The awkward part is knowing to look, because your website will keep loading perfectly in your own browser the entire time.
What The Pages Themselves Look Like To An Agent
The good news is that the basics are mostly in place, and they are the same basics that have always mattered.
The Words Arrive
114 of the 126 sites that answered send their words down in the first reply. An agent can read the page without running the site the way a browser would.
Twelve cannot be read that way. On those, everything the business says about itself is assembled after the page arrives, and an agent reading the reply on its own sees close to nothing.
The Phone Number Is Usually Dialable
103 of 126 write the phone number as a dialable link. That sounds like a detail until you watch what an agent does with it.
A number typed into a picture of a header is a number it cannot pass back to the customer. 58 of 126 publish an email address the same way.
Structured Data Has The Most Room Left
101 of 126 include some structured data, and 85 use it to say what the business is.
Only 30 use the LocalBusiness description. That is the one carrying the address, the service area and the phone number in a form nothing has to guess at. Only 24 publish opening hours in it.
Opening hours matter more than they used to, because after hours is exactly the question people ask an assistant.
Seven Sites Say Yes To Everything
Seven sites return a normal looking page for an address that does not exist. An agent takes that as confirmation the page is real, and will happily tell a customer about a service you do not offer at a URL you never made.
Six Things That Make A Site Readable To An Agent
None of these are exotic. Five of the six are ordinary web housekeeping that also happens to be what an agent needs.
1. The Page Arrives Complete
Everything the business says about itself should be in the first reply from the server, not assembled afterwards by code running in a browser.
Test it by loading your homepage with JavaScript turned off. If the page empties out, an agent sees what you just saw.
2. A Phone Number Written As A Link
A dialable link is the difference between an agent telling a customer your number and an agent telling a customer it could not find one. The same goes for the email address and the street address.
3. A LocalBusiness Description In The Page Code
Structured data is a short machine readable summary of the business that sits in the page and is never shown to a person.
The LocalBusiness type carries the name, address, phone number, service area and opening hours. It removes the guessing, and guessing is where an assistant invents a suburb you do not cover.
4. A Real Error For A Page That Does Not Exist
A missing address should return a proper not found response, not a normal page. Otherwise every address an agent invents looks real to it, including the ones it then reads out to a customer.
5. An llms.txt, If It Says Something True
llms.txt is a plain text file at the root of the site that summarises what the business does and lists the pages worth reading. A quarter of the sample already has one.
A fair number are plugin boilerplate that starts by naming the plugin, which is a wasted opportunity rather than a problem.
Written properly it is the one place you get to say, in your own words and without any design in the way, what you do and where you do it.
6. A Plain Text Version Of Each Page
One site in 126 will hand over a clean markdown copy of a page when an agent asks for one. It asks by setting a header saying that is the format it wants.
Serving it costs nothing at the point of writing. It removes every menu and cookie banner from what the agent has to read, and it means the version an agent gets and the version a person reads can never drift apart.
Why We Build For Agents On Our Own Sites
We did not run this study out of curiosity. We run all six of those things on the sites we build, and we put them in before there was a study to point at.
The reasoning is short.
The Index Decides Whether You Are A Candidate
When somebody asks an assistant for a plumber in their suburb, the assistant does not go and crawl the internet. It works from a list of candidates it already has, then reads a few of them properly.
Everything a business does to be findable feeds the first step. Almost nothing most businesses do helps with the second.
A site that arrives complete, says what it is in a form nothing has to guess at, and can hand over a clean copy of any page on request is a site that survives the second step intact. One that needs a browser to assemble itself does not.
The Cheapest Version Of Being Quoted Correctly
An assistant repeating your business to a customer is going to repeat something. The only question is whether it repeats what you wrote or what it inferred.
Every one of the six is a way of removing an inference. The phone number is stated, so it is not read off an image. The service area is stated, so it is not guessed from a suburb name in a heading. The opening hours are stated, so after hours is answered rather than assumed.
We Test It On Ourselves First
Every page on our own site answers the same requests we ran across those 131 sites, and it has done since before we measured anybody else.
Ask any of our page addresses with .md on the end and you get the text of that page. There is an llms.txt at the root. An agent that wants to work with the site properly can ask it what pages exist and read any of them, without scraping and without guessing.
That is not a clever trick. It is the same six things, applied to us.
Where AI Websites Go Wrong
There is a lot of enthusiasm about AI websites at the moment and most of it is aimed at the wrong layer.
The visible layer is easy now. A tool will write the copy, generate the pictures and lay out the page in an afternoon, and the result can look completely professional.
Everything that decides whether an agent can use the site sits underneath that, and none of it is visible from the front.
The Four We Kept Finding
| What it looks like | What is actually happening | What it costs |
|---|---|---|
| The site loads perfectly in your browser | A security plugin is refusing the crawler that builds the AI index | You are not in the list of candidates, so you are never read at all |
| The page is full of text | The text is assembled by code after the page arrives | An agent reading the reply sees an almost empty page |
| There is an llms.txt, so the box is ticked | It is plugin boilerplate that opens by naming the plugin | The one place you could describe the business in your own words says nothing |
| Every link works | A page that does not exist still returns a normal page | An agent believes an address it invented, and tells a customer about it |
The Tell Is Always The Same
In all four, the site looks right to the person who owns it. That is what makes them survive.
A broken image gets noticed within a day because somebody sees it. A firewall rule that turns away an AI crawler is never seen by anyone with a browser, so it can sit there for a year.
None of those four is a hard problem. Each is a setting, a build step or twenty lines of code. What they need is somebody who knows to check, and who checks the thing that serves the page rather than the thing that builds it.
We Have Shipped This Fault Ourselves
It is worth being specific rather than pointing at other people's work.
While running these checks we pointed them at our own builds too, and found one of ours handing agents a clean, readable, professional copy of a pricing page with every price silently removed. Twenty three pages on that site had the same fault.
One line in the code that produces the plain text copy dropped every table, and tables were where all the numbers lived. The site was perfect in a browser. The build was green. Nothing anywhere reported a problem.
We only found it because we were checking a hundred other people's sites and thought we should point the same instrument at our own.
That is the actual difference, and it is not talent. It is having an instrument, pointing it at yourself, and believing the result when it is unflattering.
The Part Worth Not Automating
There is a version of this that goes badly, and it goes badly in one specific way.
Whoever Sends It, Owns It
Work an agent can do most of, it should do most of. The last step, the one where something leaves your business with your name on it, is the one to keep.
Once you forward what a machine wrote, it is yours. A customer is not going to accept that the software said it, and neither is a court.
Nobody Wants To Be Rung By A Robot
That holds for the friendly parts of the job too. Nobody enjoys explaining a burst pipe to an automated voice that keeps asking them to rephrase.
A customer who talks to you about their job, and hears you ask a question they were not expecting, walks away thinking about the review they are going to leave.
That conversation is where the referral comes from. It is the highest value minute in the whole business, and it is the one most likely to be handed to a machine, because it is also the one that takes the longest.
Where The Line Sits
The split that works is machines for the gathering, the drafting and the admin, and people for the moment where trust gets built and the moment where something gets signed.
An agent that books the job and a person who turns up to it is a good business. An agent that talks to the customer instead of you is a worse one, however well it talks.
Who Signs It
The other half of this is boring, and it is the half that bites.
What The Bigger Companies Are Finding
In the State of AI Agent Security 2026 report, a survey of more than 900 executives and technical staff, 88 percent of organisations said they had confirmed or suspected a security incident involving an AI agent in the past year.
In the same survey, 82 percent of executives said they were confident their existing policies covered it, and only 14.4 percent said their agents went live with full security approval.
The Version For A Business With Three People And A Van
The principle is identical and the scale is smaller.
- Give an agent the information it needs for the job in front of it and nothing else. Never hand it your whole customer list for a single job.
- Decide in advance which actions it may take on its own and which ones need somebody to say yes.
- Write down who approves what, so that when something goes out wrong there is a name against it rather than an argument.
Brief it, watch it, check it, sign it. The last one is the step people skip, and it is the one carrying the liability.
Why We Keep Sharpening Our Own Tools
Most of what is written about AI and websites is written from a dashboard. Somebody reads a score, repeats it, and nobody opens the thing being scored.
We would rather own the instrument.
Checks We Wrote, Pointed At The Live Site
Every number above came from a request made to a real website on a real afternoon, not from a report about those websites. The difference matters more than it sounds.
A report tells you a site has an llms.txt. A request tells you the file is a redirect to the homepage, which is what we found on several of them before we tightened the check.
The first version of this study told us a site had no pages at all, because we had pointed it at a sitemap that turned out to be an index of other sitemaps. It returned a clean, confident, completely wrong zero.
An instrument that has never been wrong in front of you is an instrument you have not finished testing.
Forty Years Of Being Wrong In Useful Ways
There are more than forty years of search, development and AI research between the four of us, and almost none of its value is in what we know.
It is in what we have already broken.
Brad has been doing SEO since 2009, and scaled a start up into Australia’s largest online kitchenware retailer before most agencies had heard of local search.
Levi has been at it since 2015 and took Best SEO Blog in Australia at the SEMrush Search Engine Awards.
Miguel came out of enterprise ecommerce, where he ran Portugal’s largest WooCommerce migration without losing the traffic that paid for it. He now runs Nortex Labs in Porto and designs agentic systems for a living.
Every one of those years produced a check that did not exist the day before, because something went wrong and somebody wrote down how to catch it next time.
The Part That Cannot Be Caught Up On
The six things in this study are easy to list. You have the list, free, above. That is deliberate.
The hard part was never knowing them. It is that all four of the faults we kept finding look completely correct from a browser, and you cannot go looking for a problem you do not know exists.
A firewall turning away an AI crawler does not show up on your screen. A plain text copy that quietly drops every price reads beautifully. A page that assembles itself after loading looks better than one that does not.
None of those get found by someone checking their own website carefully. They get found by an instrument built by somebody who has been caught by that exact thing before.
You cannot buy that, and you cannot read it in an afternoon. It accumulates, one mistake at a time, and it only accumulates for people who keep checking after the site already looks finished.
What That Buys A Client
- Problems get found on our sites before they are found on yours. The pricing page fault above was ours, and it was fixed because we look.
- A claim we make about your site is a claim we measured. If we say your pages are readable by an agent, it is because we asked your server and read what came back.
- When the ground moves, we already have the instrument. AI crawlers turning up in firewall default lists is nine months old. Finding out it had happened to 30 businesses took an afternoon, because the tooling to ask already existed.
- Every site we build starts where this study finished. The six things are not a project we quote for. They are the floor.
What To Do In The Next 30 Days
- 1Check whether your own server is refusing AI agents. Ask your developer to request your homepage with an AI crawler's user agent string and compare the response to a normal browser. If it comes back as an error, the setting doing it is in a security plugin or a firewall rule, not in your robots file.
- 2Load your homepage with JavaScript turned off. If the page goes blank, that is roughly what an agent reads. Fix it at the template, not with a patch.
- 3Make the phone number a dialable link, on every page, in the header and in the footer. Same for the email address.
- 4Add a LocalBusiness description to the page code, with the address, the service area, the phone number and your real opening hours. If you are a mobile business with no shopfront, say so in it properly rather than inventing an address.
- 5Visit an address on your site that does not exist and confirm you get a not found page and not a normal one.
- 6Write an llms.txt in your own words. What you do, where you do it, what you do not do, and the pages worth reading. Twenty lines is plenty.
- 7Decide the one step you are never going to automate, and tell whoever works with you what it is.
Frequently Asked Questions
Agentic AI is software that carries out a task rather than only answering a question. It has tools, it can fetch and read a web page, call other software, work through several steps and check its own result.
Generative AI writes you a paragraph. An agent goes and does the errand.
An AI agent is a model with tools and a job. The model supplies the language and the reasoning, the tools let it act, and the job is whatever you briefed it with.
The part that matters for a business is the acting: an agent can read your website, take your phone number off it and hand it to a customer, which a chatbot cannot do.
Generative AI produces content when you ask for it. Agentic AI produces an outcome: it plans the steps, uses tools to carry them out, checks whether it worked and tries again if it did not.
Generative AI is the engine inside an agent. The agent is the engine plus the hands and the limits on what those hands may touch.
No, and the numbers in this study are the reason. An assistant that wants to answer a question about plumbers in Perth still has to get its candidates from somewhere, and that somewhere is an index built by crawling.
Being findable and being readable are the same job they have always been. What changes is that a second kind of reader now has to be able to finish the job on your page.
For a local service business, usually not. Blocking the crawler that builds the index removes you from the pool an assistant chooses from, and the customer never learns you exist.
If your concern is your work being used to train a model, that is a narrower choice: you can refuse the training crawlers and still allow the search ones. Make it deliberately, in your robots file, rather than leaving it to a plugin's defaults.
A plain text file at the root of a website, at /llms.txt, that summarises what the business does and lists the pages worth reading.
It is not an official standard and no search engine promises to use it. It costs twenty minutes, and it is the one place you describe your business in your own words with nothing else in the way.
That is a separate question from this one and the answer is often no. A chat widget answers people who are already on your website.
The work in this study is about being readable to the software that decides whether they ever get there. Fix the reading first.
Yes, and the first two take about five minutes. Turn JavaScript off and load your homepage, and visit an address on your own site that does not exist.
The crawler question needs somebody who can make a request with a chosen user agent string, which is one line for a developer and is worth asking for by name.
Where That Leaves You
Nothing in this study asks you to rebuild anything.
The sites that came out well were not the clever ones. They were the ones that send their words down in the first reply, write the phone number as a link, say what the business is in a form nothing has to guess at, and return a proper error for a page that is not there.
The one genuinely new thing is the front door, and it is the one most likely to be shut on you by software you did not install and cannot see from your own browser.
A quarter of the businesses in this sample are invisible to the crawler that decides whether an assistant has ever heard of them, and every one of their robots files says they are open.
So there are two honest ways forward from here.
Take the list and do the work. It is all above, it is free, and nothing in it is a secret. If you have somebody technical who will check the thing that serves the page rather than the thing that builds it, you can be in the better half of that sample inside a month.
Or start from a site that was built this way and has been measured since. That is the one we sell, and the reason it is worth buying is not the list. It is the twenty years of finding out which checks were wrong.
Either way, go and check yours. Turn JavaScript off and load your homepage. It takes a minute and it will tell you which conversation you are having.
Ready For More Quality Leads?
Apply for the Lead Launch Program and start getting exclusive, high-intent enquiries sent straight to you, with no shared leads and no bidding wars.