Although there could be many reasons for a particular server experiencing Internal 5xx Server errors, when it comes to WordPress, new Google Search Console Index Coverage report is picking up URL’s which are part of normal WordPress operations, such as /wp-includes/ and also /wp-content/themes/ folders, and, whatever template files which are required for WordPress site to load. If you are also seeing these types of Index Coverage Server Errors, then, here’s how you can fix them.
How to Video for Fixing Index Coverage Server Error 5xx for WordPress
Simplified Version for Fixing Server Errors (5xx)
Remember that 5xx server error could be temporary or even to do with your web hosting service provider and not just due to Google crawlers trying to access WordPress template files.
- /wp-content/themes/Jupiter/sample.php file
Since these files shouldn’t be crawled or accessed by search engines most web servers with WordPress installation often throws 500 Internal Server Error to Googlebot.
That means, the best way to remedy these errors is first make sure you are using XML sitemap. And then, block Google through robots.txt file so that Search Console does NOT show these in reports in the future.
Remember, this is if the index coverage issue is for CORE WordPress Files only. In other cases, you have to identify why the error is occurring through debugging.
To block Google accessing CORE WP Files
You can insert this code in .htaccess file (its usually on your web server public_html > .htaccess (right click edit))
This will disable directory browsing, thus, will also tell Googlebot “hey, wp-content/themes/rankyatheme/header.php etc. are not part of the general public view of my website, so ignore it when crawling my website”
Next step is to double check your WordPress robots.txt file. Simply visit your own website and request
If there is code in that file similar to this
Remove the robots.txt directives (even delete robots.txt file completely till the index coverage Server Error 5xx is validated. This file is located on your web server where you installed WordPress. Do keep in mind that some plugins auto-generate this file.
You can then validate fix in Google Search Console Index Coverage Report. Wait couple of day or even a week. Because Google Search Console will email you of the outcome. If the validation fails, then you need to go deeper in troubleshooting and identify PHP errors.