Google webmasters message on robots.txt

HI
what is the meaning of this?? is anything we need to modify in settings of blog

Below rule is no longer valid.

# This is not going to work any more.
User-agent: *
Noindex: /

The sole purpose of robots.txt is to allow and block crawling and never it was meant for indexing or noindexing purpose. To make things clear, probably Google is officially going to discontinue support for the Noindex directive for the robots.txt file.

Webmaster should never disallow path which they want to index or noindex. It is no way helpful.
Instead just use noindex tag in the HTML head section via meta tag of X-robots tag via HTTP header. That’s sufficient.

am not having that much idea about these terms …indexing , no indexing, allowing crawling, or disallow…like these terms. wanted to know now anything i need to change in wordpress or i can neglect it

You can ignore it.

To display a link in the Search Result

To hide a link in the Search Result

Allowing bot like Googlebot to crawl (see) the web page

Disallow means hinting bot to do not crawl certain page. This doesn’t mean noindex.

1 Like

thanks for clear explanation

You’re welcome!

Can we still use Disallow/Allow?

Yes.

1 Like