Although there could be many reasons for a particular server experiencing Internal 5xx Server errors, when it comes to WordPress, new Google Search Console Index Coverage report is picking up URL’s which are part of normal WordPress operations, such as /wp-includes/ and also /wp-content/themes/ folders, and, whatever template files which are required for WordPress site to load. If you are also seeing these types of Index Coverage Server Errors, then, here’s how you can fix them.
How to Video for Fixing Index Coverage Server Error 5xx for WordPress
Simplified Version for Fixing Server Errors (5xx)
Remember that 5xx server error could be temporary or even to do with your web hosting service provider and not just due to Google crawlers trying to access WordPress template files.
There is many occasions where Googlebot is trying to crawl core WordPress files, example URL’s like the sample ones below
- /wp-includes/class-wp-customize-section.php
- /wp-includes/class-walker-page-dropdown.php
- /wp-content/themes/Jupiter/sample.php file
Since these files shouldn’t be crawled or accessed by search engines most web servers with WordPress installation often throws 500 Internal Server Error to Googlebot.
That means, the best way to remedy these errors is first make sure you are using XML sitemap. And then, block Google through robots.txt file so that Search Console does NOT show these in reports in the future.
Remember, this is if the index coverage issue is for CORE WordPress Files only. In other cases, you have to identify why the error is occurring through debugging.
To block Google accessing CORE WP Files
You can insert this code in .htaccess file (its usually on your web server public_html > .htaccess (right click edit))
<IfModule mod_autoindex.c>
Options -Indexes
</IfModule>
This will disable directory browsing, thus, will also tell Googlebot “hey, wp-content/themes/rankyatheme/header.php etc. are not part of the general public view of my website, so ignore it when crawling my website”
Next step is to double check your WordPress robots.txt file. Simply visit your own website and request
https://www.exampleyoursite.com/robots.txt
If there is code in that file similar to this
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/*
Disallow: /wp-includes/themes/YourThemeName
User-agent: Googlebot
Disallow: /wp-includes/*
Disallow: /wp-includes/themes/YourThemeName
Remove the robots.txt directives (even delete robots.txt file completely till the index coverage Server Error 5xx is validated. This file is located on your web server where you installed WordPress. Do keep in mind that some plugins auto-generate this file.
You can then validate fix in Google Search Console Index Coverage Report. Wait couple of day or even a week. Because Google Search Console will email you of the outcome. If the validation fails, then you need to go deeper in troubleshooting and identify PHP errors.
Hi, my name is Alister, I am facing the same problem with my website please help me out I already disallow the link and remove it from the search console as well. Link (removed by admin)
Waiting for your Reply
That URL throws internal 500 error message. For such matters, quickest option is to make sure that you submit the correct XML sitemap for your clients site. Thus, if you got parent sitemap in search console, first work out site structure, then, ONLY submit XML for URL’s you want your client to rank for in Google.
For example: wanna rank blog posts only? Then submit post_sitemap XML. Wanna rank pages only? Then submit page_sitemap XML. Wanna rank products only? Then submit product_sitemap XML. If you’ve submitted PARENT sitemaps to Google Webmaster Tools, then delete that and RE-SUBMIT correct one. This will most likely remove the index coverage issue server 500 internal error from Search Console. If after you follow these steps, then, its also wise to use robots.txt to block crawling that particular WordPress Theme folder (Google shouldn’t request that directly anyway), therefore, error will not show up again Google Search Console (your clients Google rankings will NOT be affected when you follow these insights).
Hello Alister, then, all you need to do is simply wait for Google to update its database. Although you could remove the XML sitemap you submitted to Google Search Console and then re-submit the new updated XML sitemap, I encourage you to NOT do this, but rather allow Googlebot to crawl your website and it will remove the 5xx Error from Search Console reports (unless of course the issue is to do with something else)
Hello, good morning, I had a site that had a good ranking, but a month ago, a lot of fake links started to be created, which gave 404 when you clicked on them.
And to solve this problem, I bought the themeForrest template, but those pages are still visited (according to Google Analytics)
And about 1900 of these pages give a 5xx error. How to fix this error in Google Webmaster?
Can you advise what we can do?
Hello Hyka, this could perfectly be okay (it could be either a bot causing this, OR, a competitor spamming your site through referrer spam). UNLESS you are losing rankings, simply ignoring the matter is good enough. IF you are losing rankings, then it sounds like a negative SEO. Basically, you control your own website, thus, 5xx issues are your problem that requires fixing.
How to remedy? Make sure the XML sitemap has URL’s that you want to rank in Google
Fix Theme issues. Then use robots.txt to block Googlebot
You are welcomed to hire us
i really found interest in your website, thanks for the articls.
You’re welcome. And yes, this “article” is useful for eliminating Google Search Console index coverage issues for WordPress
hello google refuse to index my homepage, and also display a 5xx error
It is most likely that google refuse to index your homepage, because of 5xx error (fix the Server 5xx Error) and then Google should index your home page (as long as your home page has unique content)
hello, I have php website, I am seeing this error on my website. I am wandering to get the solution but i couldn’t find it yet, can you please help me with this Server Error
#RankYa has created many videos on https://www.youtube.com/c/rankya about fixing how to remedy Index Coverage Issue Server Error 5xx particularly for WordPress built websites
Dear sir, you have nicely explained about the Server Error 5xx WordPress.
On my google search console, Coverage>Server error (5xx) my original URL is effected due to this that URLs are not indexed in google
what should I do?
for example (URL removed)
Please reply
There could be many reasons for 5XX server errors, could even be temporary, if search console is not indexing, then make sure canonical URL’s are set correctly, and then, explore the Search Console features, including this new version of GSC https://support.google.com/webmasters/answer/7440203 and here https://support.google.com/webmasters/answer/9689846 one way or another these errors must be fixed
hello sir…
i try to fix this but it comes more then before, now i and understood what is this? and what to do plz help me to solve this permanently
Usually a incorrectly coded WordPress Theme causes this, OR, it could be plugin conflict or error. You will need to either contact the Server Support Team, Theme support or provide more information
My site has received server error 5xx issues many times and i solved all those but this time i received server issue for the tags which were deleted one.Actually it should show 404 error because the tags were deleted but it shows as server error 5xx
Yes, that’s correct Steve, it should show 404. Have you fixed it? Could be a temporary issue. I would clear cache and try again
Hi, I am currently getting 2 errors, they are for pages that no longer exist on my site.
I recently switched the format of links on my site.
From the date and then title format to just the plain title format.
This is a valid link.
URL removed
how would I go about removing these from the search console?
URL removed
I am no newbie with websites or wordpress, i just have no idea where to find this info in the dashboard or if I have to go edit a ini file.
Thanks!
https://www.rankya.com/wp-content/uploads/2021/05/sample-PHP-condition-for-setting-noindex-in-WordPress-header-php.png
You would need to add that in header.php within HTML
portion. What will it do? it will blurt out noindex directives for parts of WordPress such as Date Archives (they should not be in Google (that is the case for 99% of WordPress setups)) furthermore: it will do the same for Tags (you can remove or add more to it). Also, I think with your website, there is somehow a problem with your setup (could be .htaccess rules). Let us know the outcome. Importantly, when submitting sitemaps to Google, then, make sure you submit ONLY the XML sitemaps YOU WANT Google to rank / thus / index the rest will be waste of time (such as date or tag archives)Am also having similar issue..i tried it but still not working…Am currently using a VOICE wordpress themes….(removed) is my Domain URL
Contact Theme support section
shouldnt that ROBOTS.TXT code be:
wp-content/themes ???
is there a themes folder in wp-includes???
You can put any directives within robots.txt file, however, keep in mind that robots.txt file directives are NOT to do with controlling how #Google search engine indexes a site. Robots.txt file is for crawling. And to answer your question: no you should NOT block Googlebot with wp-content/themes (also, there is no themes folder in wp-includes)
Fixing WordPress 5xx Server Error is to do with how WordPress Themes work. Some themes require certain files, so thus when accessed directly (as such in Googlebot’s case) web server will throw an error, therefore, that’s what Google Search Console shows in the report. Nothing to be too concerned about because its more than likely the theme will be updated with the next update release. You should definitely contact the theme support about their theme having issues. Jupiter Theme that I use always, had a similar issue in version 5, but now the WordPress theme developers have fixed it.
After few days of search i finally resolve google webmaster coverage issues, thank you RankYa 🙂
this post need to top when some searches for wp-include 5xx error.
I got an email from google search console “New Index coverage issues detected for my website. I feared what is this then i search thing problem on google and find solution at your blog. The very genuine solution you shared. Thank you very much.
Great to hear that Search Console index coverage solutions are here at RankYa Raghwendra