Robots.txt keeps changing

The issue started from the beginning of this month. I don’t know what was the issue but my robots.txt keeps changing. I tried everything from clearing cache from cloudflare, creating exceptional rules and even changed my server. It keeps changing when I see it in incognito and google has blocked my several hundred posts due to this.

There is a robots.txt file in the root which shouldn’t be overwritten but it keeps changing automatically in the browser and thus in the eyes of Google although it is intact in the website. Also, I have tried changing theme and disabling all the plugins. It doesn’t gets fixed. Please can anybody help me?

May be any SEO plugin writing its rules.

Welcome back to HGK forum :slightly_smiling_face:

Just add a robots.txt file via SFTP/File Manager.

Already did but it gets cached. I don’t know why. Sometimes I think it is due to cloudflare. Yesterday the robots.txt url was not redirecting to https from http till I changed the setting in cloudflare although it was redirecting a few days ago. Now the non www is not redirecting to www although every page is redirecting properly

Would be possible to share site URL so I can check once what’s possibly could be issue?
I would be interesting taking a look at Header Response.

xtremegaminerd[dot]com

I got total three version of robots.txt for https non-www, https-www, https-www with query string.

Your robots.txt file has following header

cache-control: max-age=31536000
cf-cache-status: HIT

It is doing browser & edge cache for 1 year which is abnormal.

Possible fixes


  • Ideally, you should keep ‘Respect Existing Header’ value for Browser Cache TTL

  • Try setting once robots.txt at Yoast > File Editor > Robots.txt field. Or, keep it blank. Save.

  • Login to Ezoic & Cloudflare both. Check your DNS for www and non-www records. Please make sure it is pointed to your current hosting Public IP address.

1 Like

I do not have A record in the Cloudflare but have it in the Ezoic. I also checked the Browser Cache TTL in Cloudflare and it is set to 1 year just like you said. I am gonna send this info to my web developer but I think you have got a good catch on the possible problem. I will update soon.

Also my other site has Browser Cache TTL set to 4 hours.

Everything is fixed now. We had to delete wp rocket, Amp etc. which were in conflict with Ezoic and also decreased the cache timing to 4 hours in cloudflare. Googlebot also has updated the robots.txt.

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.