ai.robots.txt is a curated list of AI crawlers and robots to block. It ships ready-to-use robots.txt, .htaccess, Nginx, Caddy and HAProxy snippets so site owners can keep AI scrapers out.
4248ai,crawlers,crawlingai.robots.txt is a curated list of AI crawlers and robots to block. It ships ready-to-use robots.txt, .htaccess, Nginx, Caddy and HAProxy snippets so site owners can keep AI scrapers out.