Is my website blocking AI?

In today's digital age, where artificial intelligence (AI) plays a critical role in finding information and analyzing content, it's crucial to make sure your website is accessible to language model bots (LLMs). These bots, such as those from OpenAI, Google, and others, are responsible for crawling and analyzing web pages to deliver relevant results in search engines.

Similar to how bots operate in the SEO process, they first find content, crawl, scan, classify it, and store it in their memory. In the event that it is of quality, it will be eligible to appear in their answers.

From there, what we must ask ourselves is whether we are interested in our company, its products or services appearing in the AI results. If your answer is yes, read on. If not, you can leave it here or take advantage of this content to do the opposite.

But let's assume that we are interested in AI tracking our content. And the question you may be asking yourself is: is my website blocking access to these tools? Well, it is possible that it does, even if you don't know it and you don't want it.

How to check if AI can access my website?

1.     Using the Robots Exclusion Checker extension

This extension is wonderful for detecting URLs that cannot be indexed for various reasons. But in this case, it will also check if we are blocking the 14 most common AI bots using robots.

2.     With LLMPulse

This GEO tool, which we have mentioned on other occasions, comes with a small tool called "Robots.txt checker", which will give us light. This one is much more complete, as it analyzes 20 bots and gives us details on whether the block is total or partial, and where to locate it. This example is of a real customer, and it's a drama (easy to solve).

3.     With AEO Checker

This is a GEO-free tool that's pretty cool. It gives us a lot of additional information in addition to what we see here, but this one gives you in an exhaustive way. It measures 23 bots and gives all the details.

4. Additional Methods

We've mentioned these 3 because they're simple, accessible, and very intuitive. Now, if the majority of crashes in AI come from robots.txt configurations, this is not always the case. Sometimes, the block can come from a higher entity, usually a server with many layers of security or attack prevention, or a CDN wary of certain queries (especially Cloudflare).

In these cases we recommend using Techincal SEO's AI Bots Testing Tools. In this case, you don't just have to check if the robots.txt is blocking you. It's going to emulate being an AI agent, and it's going to see our server's response and access to the content. It may give some false positives, but we have not yet seen a false negative. Paradoxically, making this post we have detected that Claude could not access:

Additionally, you can do the following:

• Review your CDN settings
• Check with your hosting service if they are applying any restrictive rules that may affect
• Perform empirical checks: Ask your assistant to access a URL on your website and process it. If he tells you he can't, we're in a bad way
• Emulate being an AI bot, using a browser or tools such as Screaming Frog, to test content
• If you have time, knowledge and access permissions, you can access the server logs, cross the AI bots that interest you, and you will know: which ones visit you how often and if we are blocking access to some of them.

Access is necessary but not sufficient

As we mentioned at the beginning, the path to GEO begins with AI bots like ChatGPT being able to access your website. This step is essential, but you will need much more to reach the desired citation or mention. 

But access to AI bots is only the first step. Once you've allowed this access, it's essential to work on optimizing your content, site structure, and other technical aspects to improve ranking in AI results. Basically, the engine of GEO is to do SEO, but good SEO and with an LLMS-oriented approach.

If you've already opened the door to AI, you've taken a very important first step. We will address in other posts what comes after ;)