Our crawler

Our crawler identifies itself as LLMsTxtGenerator/1.0. It only runs when a person enters a website on llmstxgenerator.com.

• It reads robots.txt, the sitemap, the home page and up to 29 more public pages, 6 at a time.
• It skips login, cart, tag and category pages and any page marked noindex.
• It does not store page content. Only the finished llms.txt is shown and emailed to the person who asked.

Questions or problems: anthony@supercanic.com.