Crawlability Checker
Check whether AI crawlers like GPTBot and ClaudeBot can access your pages. Understand robots.txt, live fetch results, and what blocks AI.
Updated
For AI to use information from your website, it needs to be able to access the content. Access restrictions can make it harder for AI systems to retrieve your pages.
"Crawlability Checker" helps you investigate whether different AI bots are allowed to visit a specific page and whether Klarvu can retrieve it using the bot's name.
How to run an audit
You'll find the tool under "Improve" → "Audit Tools" → "Crawlability Checker".
The page is titled "Crawlability Checker — [brand]" and includes the description:
"Checks whether AI crawlers can fetch a given URL — robots.txt rules and a live fetch, per bot."
Enter the page you want to check
Enter the full URL in the field with the placeholder "https://example.com/page".
The field is prefilled with your brand's website. You can replace it with another URL, including a page on a different domain.
You can test one URL at a time.
Start the audit
Click "Run Audit".
While the check is running, the button displays "Running audit...". If you've already checked a URL, you can use "Re-run Audit" to test it again.
Follow the progress
Klarvu updates the page automatically while the audit is running. You'll see "refreshing automatically...".
Under "Audit status", the status may appear as "queued" or "running".
Review the results
Once the audit is complete, you'll find the results under "Server access check".
There's a card for each bot showing its name, owner, and two status labels: one for robots.txt permissions and one for whether Klarvu could retrieve the page.
Before your first check, you'll see "No audit yet". If an audit fails, the status changes to "failed" and displays an error message.
The page shows the latest audit result for the URL you're checking.
Understand the results
Under "Server access check", every bot has two status labels. These are either green ("Allow") or red ("Block"). The first label shows the robots.txt result, and the second shows the live fetch.
They check different things:
| Label | What does it check? |
|---|---|
| robots.txt (first label) | Whether your website's robots.txt rules allow the bot to access that specific URL path. If the bot doesn't have its own rule, the general (*) rule applies. |
| Live fetch (second label) | Whether Klarvu can retrieve the page using the bot's name as its user agent. "Allow" means the server returned an HTTP status in the 200 range. "Block" displays an HTTP status code or an error type if the page couldn't be fetched. |
robots.txt is a file websites use to specify which pages certain bots are allowed to visit.
Live fetch checks how the server responds to an actual request for the page.
The two results don't always match.
What do the different combinations mean?
| Situation | What it means | What you can do |
|---|---|---|
| robots.txt: "Allow" + Live fetch: "Allow" | The bot is permitted to access the page, and Klarvu successfully retrieved it. | No access restriction was detected in this test. |
| robots.txt: "Block" | Your website has a rule instructing the bot not to access the page. | Check whether this restriction is intentional. |
| robots.txt: "Allow" + Live fetch: "Block" | robots.txt permits access, but Klarvu couldn't retrieve the page using the bot's name. | Investigate whether a firewall, CDN, or bot protection is blocking the request. |
The last situation is particularly worth investigating.
If robots.txt allows a bot but live fetch returns something like "Block · HTTP 403", your website may have additional access restrictions.
That doesn't automatically mean something needs to be changed. But it's useful to investigate if you expect that bot to be able to visit the page.
Which AI bots are checked?
Klarvu checks 15 bot names and rules associated with different AI companies and services.
| Owner | Bots and rules |
|---|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot |
| Perplexity | PerplexityBot, Perplexity-User |
| Google-Extended, GoogleOther | |
| Apple | Applebot-Extended |
| Meta | Meta-ExternalAgent |
| ByteDance | Bytespider |
| Common Crawl | CCBot |
| xAI | GrokBot |
Which pages should you check?
It's tempting to test only your homepage, but different pages on your website may have different access rules.
If Klarvu identifies a restriction, you can use the result as a starting point for investigating your website's access settings.
You may also encounter accessibility-related suggestions under the "Crawlability" category on the Recommendations page.
The Content Checker page covers reviewing page content, and an upcoming Agent traffic guide explains visits from AI crawlers. You can also use the Sources page to see which pages AI actually cites.
Frequently asked questions
Should I allow every AI bot?
That depends on how you want your website content to be accessed and used.
"Crawlability Checker" shows your current access rules and test results. Whether to allow or block a particular bot is your decision.
Why does it say "Block · HTTP 403"?
It means the server rejected Klarvu's attempt to retrieve the page using the selected bot name.
This may be caused by a firewall, CDN, or bot protection. Review your access settings to investigate the specific reason.
I've changed my robots.txt file. When will I see the result?
Click "Re-run Audit" to start a new check.
Klarvu will check the rules and attempt to retrieve the page again.
Does "Allow" mean AI will cite my page?
No. "Allow" indicates that access is permitted or that the test successfully retrieved the page.
It doesn't guarantee that AI will use or cite the page in its answers. You can investigate actual citations separately under "Sources".