Index Coverage Issue

Google Search Console Index Coverage Report UI

Been getting messages due to the latest Google Search Console Update showing Index Coverage Issue Errors and Warnings? You are not the only website owner facing this problem (particularly if you are using the popular Yoast SEO Plugin and its sitemap functionality). So let’s get to work and find out how to fix index coverage issues.

What is Index Coverage Issue?

Basically (and usually) this is Google telling you

Hey, you told us to index your website by submitting a XML sitemap, through which we analyze your website URL structure, but when we access that URL you told us about in your XML sitemap. We are having issues seeing the contents of that URL = INDEX COVERAGE ISSUE

How to Fix Indexation Coverage Issues

Easy, just make sure that your submitting XML sitemap does NOT contain URLs which are either

Sending noindex directive (usually through using meta tags) which look like this:

<meta name="robots" content="noindex">

Or are URL’s blocked by robots.txt file but those URL’s are in the XML Sitemap you’ve submitted to Google Search Console

Fixing Index Coverage Issues Due to URL’s blocked by robots.txt

All you need to do is modify your robots.txt file and simply delete the line which is blocking Googlebot to see what is on that page.

For example: let’s imagine that Google Search Console Index Coverage error was due to Googlebot not being able to access
https :// www.rankya.com/ samplepage

And my robots.txt file had directive like the one below:

User-agent: *
 Disallow: /cgi-bin/
 Disallow: /wp-admin/$
 Disallow: /wp-content/cache*
 Disallow: */trackback/$
 Disallow: /samplepage
 Disallow: /comments/feed*
 Disallow: /wp-login.php?*
 Allow: /*.js*
 Allow: /*.css*
 Allow: /wp-admin/admin-ajax.php
 Allow: /wp-admin/admin-ajax.php?action=*
 Disallow: /2016/
 Disallow: /*?wordfence_logHuman=*
 Allow: /wp-content/themes/rankya/*.css

All I would need to do is just delete the line Disallow: /samplepage and re save the robots.txt file on my server (if using Yoast, you can update your robots.txt file through WordPress Dashboard > Yoast SEO Plugin > Tools > file editor

And the index coverage issue due to robots.txt blockage would go away, as in be fixed. You could follow the same procedure for other URL’s index issues due to robots.txt file

Fixing Index Coverage Issues Due to noindex tags

All you have to do in a case where Googlebot is able to access the URL but is seeing

<meta name="robots" content="noindex">

In this scenario, all you will need to do is one of two things

  1. Remove the noindex meta tag
  2. Remove the URL from the XML sitemap

How to Remove the noindex meta tag When Using Yoast SEO PLugin?

This will be dependant on the way in which your website is blurting out the noindex meta tags. If you are using popular Content Management System like WordPress and SEO plugins like Yoast. Then, the way to remove the noindex is through its settings called Search Appearance

replacewithyourdomainnameandcopypasteintoaddressbarofyourbrowser.com/wp-admin/admin.php?page=wpseo_titles#top#post-types

When you are there, anything you set to Yes is shown in the XML sitemap automatically. That means, when you set anything to NO it will remove the entire part of WordPress (posts/pages) from shown in the sitemap.

Warning: do not use these settings to remove important section of your WordPress site (posts pages custom post types which you want to rank in Google for, products for WooCommerce) all of which MUST BE indexed by Google, or else you will not get any website traffic through Google.

Instead, you can go to individual post/page/custom-post-type/categories/tags and press on the gear icon and select

WordPress Post Settings for noindex
Shows user interface and option for Yoast SEO Plugin noindex option for individual WordPress blog post

This will fix index coverage issues due to noindex meta tag because Yoast SEO Plugin will REMOVE that URL from showing in the XML sitemap.

At the end of the day, fixing Google Search Console index coverage issue isn’t that hard at all. Basically, what you need to do is triple check the URL’s which you submitted to Google Search Console XML sitemap and make sure that those URL’s are either not serving noindex directives, OR, those URL’s are NOT blocked by robots.txt file.

Video About Google Search Console > Index Coverage > Error > Submitted URL has Crawl Issue

Keep in mind that crawl issues usually occur due to pages that are no longer on your site. And also, if there are crawl errors for pages that never existed on your site. Then, keep in mind that it is more than likely that some external website has a backlink to your website which Googlebot is following (those external site could also be a web scraper or spammer which is causing those 404 errors shown in Search Console Webmaster Tools). For such cases, you don’t need to worry or do anything about it. If you want, you can add this to your Apache .htaccess file or even block the ip:

### Referrer spam https://support.google.com/analytics/answer/1034842
RewriteCond %{HTTP_REFERER} ^(www\.)?ilovevitaly\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?sertacyildiz\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?sharebutton\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?aliexpress\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?Get-Free-Traffic-Now\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?trafficmonetize\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?buy-cheap-online\.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?successfulworkonline\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?elrincondetolito\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?tworzeniestrongorzow\.pl [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?blackhatworld\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?vk\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?jetradar\.co\.th [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?avgoogle\.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?xxxfo\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?100dollars-seo.com\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?best-seo-offer\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?buttons-for-your-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?todaperfeita\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?(.*).free-social-buttons\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?site\.ru [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?james\.blog1989\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?xtrafficplus\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?your-website-sucks\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^(www\.)?crypto-code\.xyz [NC]
RewriteRule ^(.*)$ - [F]

##hotlinking remove the next 3 lines hashtags is you want to use the below directives which will prevent hotlinking (which often causes 404 page not found errors due to others linking to your images, but doing so using incorrect URL structure

#RewriteCond "%{HTTP_REFERER}" "!^$"
#RewriteCond "%{HTTP_REFERER}" "!www.rankya.com" [NC]
#RewriteRule "\.(gif|jpg|png)$" "-" [F,NC]

# To block IP addresses, see example format below. Add to list as needed, to find more analyze your web server log files for suspicious URLs requesting parts of your site:
deny from 91.109.201.210
deny from 144.48.37.206
How to Fix 404 Errors in Google Search Console

The above video mainly focuses on creating redirection 301 Moved Permanently. Keep in mind that you should only use the 301 redirection IF the content you are redirecting to, has similar content.

Fixing Search Console Index Coverage 500 Internal Server Error for WordPress Themes

The video above shows the typical WordPress Theme URL and Googlebot complaining about Server Errors 5xx response code. The best way to remedy this error is to serve 404 page not found because there is nothing in that URL that Google should index for WordPress setup.

Hire RankYa Hire Experience and Have a Better Website

If all these are confusing to understand (noindex, meta tags, robots directives) and you are using WordPress, check out the latest services

Author: RankYa

RankYa: Online Entrepreneur, Qualified Web Developer, Google AdWords and Google Analytics Certified Professional. Specialist in: SEO, Website Optimization, WordPress, Structured Data, JSON-LD, Microdata, Microformats, RDF, Schema.org Vocabulary, HTML5, Advanced Image Optimization, Google Search Console, Google Webmaster Guidelines, Social Media Marketing, Facebook marketing and YouTube video ranking.

Thank you for sharing this blog post. Description: 'Google Search Console Index Coverage Issue can be confusing to understand and fix, RankYa explains what the index coverage issue is and how to fix it'