Shield · AI crawlers
robots.txt is a request, not a rule. Shield sits in front of the site, on your server, and decides who gets in.
Community is free on one server, with every protection.
A crawler that wants to respect robots.txt reads it; one that doesn't ignores it. Nothing on the server enforces it.
And every page fetched is a page your server has to build: a crawler that goes too fast slows the site down for people who are buying.
Google, Bing, Apple, Yandex, Baidu, Qwant and Petal are verified and never stopped. Your SEO is untouched.
In full protection a fake Googlebot is refused, and so are scripts and automated browsers. In observe mode they are recorded.
By default they get in, slowed down if they go too fast. One setting refuses them all; verified engines and link previews stay in.
Shield does not pick crawlers one by one: robots.txt does that, for those who respect it. Shield takes care of the rest.
Google uses the same Googlebot for search and for some AI features, and Shield always lets the verified Googlebot in. To keep your content out of Google's AI, use the Google-Extended line in robots.txt.
No. Verified search engines are never stopped, whatever you choose for other crawlers.
Shield judges them by how they move through the site. During a wave, new visitors go through a short check in the browser.
Yes. Previews are recognised and let through.
No. Improvements come with Shield updates.
Shield installs in minutes on your Linux server and starts in observe mode: you see what it would stop before it stops anything.
Install Shield → See the plans →