What is robots.txt file?


#1

Can anyone elaborate what actually robots.txt file of a blog website?


#2

Simply a file using which you can tell search engine bots what to index and what not .
Like, If I want that Google does not index a category like /users then I can simply add it in my Robots.txt as

Disallow:- /users


#3

The robots.txt file is used to set the crawling instruction for respecful bots like GoogleBot. This file is recommended to place at the root public html directory.

https://example.com/robots.txt

For example, in WordPress we can have …

Sitemap: https://www.example.com/sitemap_index.xml
User-agent: *
Disallow: /?s=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-admin/images/
Allow: /wp-admin/css/
Allow: /wp-admin/js/

Line by line Explanation

  1. Sitemap declared
  2. For all user-agents
  3. Blocked Search Results
  4. Blocked administrative path
  5. Allowed ajax theme elements
  6. Allowed wp-admin images path for Embed content
  7. Allowed wp-admin CSS
  8. Allowed wp-admin JS

[The robots.txt rule is written as per Google Mobile Friendly guidelines which says, do not block CSS and JS path.]


#4

Apart from this, do you have any related specific question? I would be happy to answer.


#5

Should I press Enter Button of the keyboard to put another command in next line?
Also my admin dashboard link is different than wp-admin
Should I keep “Disallow: /wp-admin/” or change it?


#6
  • Yes, all rule should be written per line
  • No need to add changed wp-admin path