Shield · AI crawlers

Keep out the crawlers you don't want. Keep Google in.

robots.txt is a request, not a rule. Shield sits in front of the site, on your server, and decides who gets in.

Community is free on one server, with every protection.

§01The problem

robots.txt alone is not enough

A crawler that wants to respect robots.txt reads it; one that doesn't ignores it. Nothing on the server enforces it.

And every page fetched is a page your server has to build: a crawler that goes too fast slows the site down for people who are buying.

§02What Shield does

Three rules

01

Real search engines always get in

Google, Bing, Apple, Yandex, Baidu, Qwant and Petal are verified and never stopped. Your SEO is untouched.

02

Impostors are out

In full protection a fake Googlebot is refused, and so are scripts and automated browsers. In observe mode they are recorded.

03

You decide on declared crawlers

By default they get in, slowed down if they go too fast. One setting refuses them all; verified engines and link previews stay in.

§03Good to know

What Shield does not do

Shield does not pick crawlers one by one: robots.txt does that, for those who respect it. Shield takes care of the rest.

Google uses the same Googlebot for search and for some AI features, and Shield always lets the verified Googlebot in. To keep your content out of Google's AI, use the Google-Extended line in robots.txt.

FAQQuestions

Questions about AI crawlers

Will blocking AI crawlers cost me Google rankings?

No. Verified search engines are never stopped, whatever you choose for other crawlers.

What about scrapers posing as a browser?

Shield judges them by how they move through the site. During a wave, new visitors go through a short check in the browser.

Do link previews on WhatsApp or Slack still work?

Yes. Previews are recognised and let through.

Do I have to keep a list of crawlers up to date?

No. Improvements come with Shield updates.

→Read next

Decide who reads your site

Shield installs in minutes on your Linux server and starts in observe mode: you see what it would stop before it stops anything.

Install Shield → See the plans →