AI crawler

A bot that fetches web pages for AI training or live retrieval, such as GPTBot or PerplexityBot.

AI crawlers are how engines read your site, whether to train on it or to retrieve it at answer time. If your important pages are blocked or slow, you make yourself harder to cite.

Some tools track these visits as a signal of AI interest in your content. At minimum, make sure your robots rules do not accidentally shut the door on engines you want to reach.

See it in your own data

Elmo is an open-source AI visibility platform. Self-host it for free and track how AI answer engines mention and cite your brand.