robots (dot) txt & other security complaince

This guide has templates for robots txt & meta tags for blocking or allowing search engine crawlers & indexers.

The txt is regularly updated.

DNS configuration for email security is also laid down.

Other notable policy documents to implement for your sites:

https://iabtechlab.com/ads-txt/

https://securitytxt.org/

https://help.shopify.com/en/manual/online-sales-channels/google/requirements

https://www.eff.org/deeplinks/2014/04/warrant-canary-faq