what is the meaning of this?? is anything we need to modify in settings of blog
Below rule is no longer valid.
# This is not going to work any more. User-agent: * Noindex: /
The sole purpose of robots.txt is to allow and block crawling and never it was meant for indexing or noindexing purpose. To make things clear, probably Google is officially going to discontinue support for the Noindex directive for the robots.txt file.
Webmaster should never disallow path which they want to index or noindex. It is no way helpful.
Instead just use noindex tag in the HTML head section via meta tag of X-robots tag via HTTP header. That’s sufficient.
am not having that much idea about these terms …indexing , no indexing, allowing crawling, or disallow…like these terms. wanted to know now anything i need to change in wordpress or i can neglect it
You can ignore it.
To display a link in the Search Result
To hide a link in the Search Result
Allowing bot like Googlebot to crawl (see) the web page
Disallow means hinting bot to do not crawl certain page. This doesn’t mean noindex.
thanks for clear explanation
Can we still use Disallow/Allow?