Is this tool free?
Yes, and there is no limit on this one — it costs us nothing but a few HTTP requests. A Citex account is for keeping the result, re-running it on a schedule and seeing what moved.
AI Crawler Access Checker
Check what your robots.txt says to AI search crawlers and data-use controls. This is a policy check, not proof of indexing.
What you get
- Which listed crawler and data-use tokens your robots.txt allows or blocks
- The exact rule responsible for each block
- Whether a blanket Disallow is catching them by accident
- Who each crawler belongs to and what it feeds
How it works
Add your site
Paste the domain or page you want checked. Nothing to install, no tracking code.
We read your robots.txt
We fetch the file and parse it the way a crawler does — group by group, because a rule for GPTBot says nothing about the group for *.
Read the result
The finding, why it matters and what to change — on the page, straight away. No account, nothing to install.
Frequently asked questions
How accurate is the result?
It is exact, not a sample. This tool reads your own files and reports what they say, with no model in the loop — run it twice a minute apart and you get the same answer unless you changed something.
Related free tools
Five of these read your site and answer instantly. Four ask a real AI platform. All ten are free.
AI Visibility Checker
Run one live query on Google AI Mode and see whether it names you, and who it names instead.
Open the tool
AI Share of Voice Calculator
Your mentions are only meaningful next to everyone else’s. This puts a percentage on it.
Open the tool
AI Brand Sentiment Checker
Being mentioned is half the story. This shows you how you are described.
Open the tool
Turn one check into tracking
A single run is a snapshot. Citex runs your prompts on a schedule and shows what changed across five AI platforms.
Start 7-day free trial ->