How to Noindex OneSignal WordPress Plugin Directory?


(Amit Tiwari) #21

What will be the benefit of blocking the search result page from crawler,?


(Ultra Noob) #22

To prevent unnecessary crawling, spam links. Ref: labnol


#24

@GulshanKumar Is there some issue with this code? It is not adding the “X-Robots-Tag” header. I added using code snippets. Thanks!


(Ultra Noob) #25

Try this method


#26

Thanks for the help!

It is working! I have added the following in my Nginx configuration.

location ~ .*/(?:onesignal-free-web-push-notifications)/.* {
      add_header X-Robots-Tag "noindex, follow" always;  }

Observation:
“X-Robots-Tag” header is not showing for this URL. I believe as we have already blocked the main folder, so that should not be an issue!

https://www.example.com/wp-content/plugins/onesignal-free-web-push-notifications/views/css/icons.css

“X-Robots-Tag” header is showing for following URLs,
https://www.example.com/wp-content/plugins/onesignal-free-web-push-notifications/
https://www.example.com/wp-content/plugins/onesignal-free-web-push-notifications/views/
https://www.example.com/wp-content/plugins/onesignal-free-web-push-notifications/views/css
https://www.example.com/wp-content/plugins/onesignal-free-web-push-notifications/readme.txt


(Ultra Noob) #27

Have you purged cache? (If using any CDN)


#28

Well, I have not integrated any CDN as of now, will be adding it soon. Though, I am using Cloudflare with HTTP Proxy, with the suggestions that Bhanu gave by adding example.com/* with page rule “cache level: bypass”, “disable performance” etc. so that I get an advantage of hiding my cloud VPS IP, DDoS, firewall for blocking visitors IP etc.

Now, the interesting thing is, when I am using http://www.webconfs.com/http-header-check.php , it does not show “X-Robots-Tag” header.

Though in another incognito tab, when I use Developer tools, it shows “X-Robots-Tag” header. LOL! :yum: I think a few hours back, I checked using Developer tools also. “X-Robots-Tag” header was not showing! Though as you mentioned about CDN, I just did “Purge everything” in Cloudflare!