Robots.txt Guidelines

What is a Robots.txt file ? 

The Robots.txt file tells search engines which pages they can explore or not. This file is placed at the root of the site at

This file can, if misused, cause some of your pages to be indexed in search engine results, even if you don't want them to.

How to check the Robots.txt file ?

  • Make sure robots.txt doesn't return an HTTP 5XX status code
  • Keep robots.txt smaller than 500 KiB. Search engines may stop processing robots.txt midway through if the file is larger than 500 KiB.
  • Fix any format errors 
  • Provide an absolute URL for sitemap ( instead of /sitemap.xml)