WordPress robots.txt File

WordPress

Using robots.txt file on your WordPress built site? If not, you should.

How to create it? Easy, just open a Microsoft Notepad and create a text file and name it robots.txt

Then upload this to your WordPress installation.

Login to cPanel > File Manager > public_html (path may be different for your website setup) > upload robots.txt here.

When you need to provide directives so that you can better control (or even block) crawling of search engines like Google, here’s all you need to know about using robots.txt file for WordPress Content Management System.

It is absolutely important that you actually understand what these directive do, or else, you may see many Google Search Console Errors (such as URL’s blocked by robots.txt).

Google search engine before it indexes your website, it CRAWLS IT using a user-agent called Googlebot.

Other user-agents = search crawlers, web crawlers all mean the same thing. They basically request URL to see what’s on that URL

WordPress built site owners can use the robots.txt file to give further instructions about their website to Googlebot. This is called The Robots Exclusion Protocol. Below are 2 additional readings to expand your knowledge about Google and Website optimization.

How to Create & Use robots.txt on WordPress (Video by RankYa)

WordPress robots.txt File Location

By default, when you install WordPress, there is NO robots.txt file on your installation. Although that is the case, WordPress will automatically create a virtual robots.txt file. Hence the confusion when you try to look for such file on your web server. Usually you would look here:

File Manager > public_html > robots.txt file

And then you don’t see anything because there is NO physical robots.txt file. That means, you need to create one like this:

WordPress robots.txt File Example 1

User-agent: * Disallow: Simply create a text file and call it robots.txt and then insert the above code and upload to your web server where your WordPress is installed. Above sample code says

Hey Googlebot you CAN access my entire website, I disallow you NOT

WordPress robots.txt File Example 2:

User-agent: Googlebot Disallow: /cgi-bin/ Disallow: /wp-admin/$ Disallow: */trackback/$ Disallow: /comments/feed* Disallow: /wp-login.php?* Allow: /*.js* Allow: /*.css* Allow: /wp-admin/admin-ajax.php Allow: /wp-admin/admin-ajax.php?action=* Allow: /wp-content/uploads/* User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/$ Disallow: */trackback/$ Disallow: /comments/feed* Disallow: /wp-login.php?* Allow: /*.js* Allow: /*.css* Allow: /wp-admin/admin-ajax.php Allow: /wp-admin/admin-ajax.php?action=* Allow: /wp-content/uploads/* #Sitemap: https://CHANGE/page-sitemap.xml #Sitemap: https://CHANGE/post-sitemap.xml #Sitemap: https://CHANGE/product-sitemap.xml

In the above robots.txt file example, all you need to change is the location of your sitemaps, and then remove the #tag as it is used as a comment. Just create a text file and call it robots.txt and then place the above code and upload to your web server where your WordPress is installed (once you change sitemap location). You can safely use the above code as it will help your WordPress site avoid many Google Search Console errors such as Server 5xx Errors.

Download robots.txt File Example

robots.txt (.zip format, simply unzip it)

WordPress robots.txt Not Updating?

Its usually because you are seeing the cached version of your website. You need to clear the cache of your internet browser.

WordPress robots.txt Yoast SEO Plugin?

Yoast SEO plugin for WordPress does create a robots.txt file for you and allows you to modify it from your WordPress dashboard. Simply use the sample codes above and Save changes to robots.txt file.

WordPress robots.txt Plugin?

Do NOT use plugins for using robots.txt file because creating one is simple as detailed above.

URL Blocked by robots.txt WordPress

What this means is that when Google tries to crawl your website / web pages, it sees something in the robots.txt file that says “Hey Google you are NOT allowed (hence DISallow) to crawl that URL” causing URL blocked by robots.txt file errors in Google search console.

How to Unblock URL blocked by robots.txt in WordPress

Simply use the instructions shown in WordPress robots.txt File Example 1 above as it will unblock any directives in your robots.txt file

How to Unblock URL blocked by robots.txt in WordPress

Search Engine Crawl vs Index process To avoid most issues with Google Search Console robots.txt blocking certain URL’s from your own WordPress built web site, understand the fact that Google’s CRAWLING and INDEXING Process is 2 completely different processes.

That means, just because you are using robots.txt file does NOT mean you are telling Google to NOT index certain parts of your website. To be able to control Google’s indexing certain parts, you need to use meta tags with noindex option. Example only for WordPress CMS <?php if (is_front_page()) : ?> <meta name="Googlebot" content="index"> <meta name="robots" content="index"> <!--this is front--> <?php elseif (is_search()) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <!--searchqueries now part of the main conditional logic--> <?php elseif (is_page('samplePageNameYouWantToNOINDEX')) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <?php elseif (is_singple('sampleBlogPostNameYouWantToNOINDEX')) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <?php endif; ?> Path to that file is Login to Web Hosting > File Manager > public_html > wp-content > themes > YourThemeName > header.php

Basically, locate header.php file and find < head > portion and modify above PHP code and then copy paste save, to control indexing of Google for certain parts of your WordPress site.

RankYa WordPress SEO Tip

  • do NOT index tags so use is_tag()
  • do NOT index attachment URL’s is_attachment (unless your website is about photography or photo sharing site)
  • do NOT index paginated content is_single() && is_paged()

Don't let Google's latest Page Experience update derank your business website in search, instead, master Google rankings and website conversions with this Master Class Course by RankYa (Free Lessons Included)

Learn how press here

By RankYa

RankYa is a passionate digital marketer, website optimizer, content creator, and a fully qualified web developer helping businesses of all sizes (big or small) to achieve better results online. We are a reputable and registered Australian business serving valued clients worldwide since 2008, built on hard work, honesty and providing exceptional personalized service

We love sharing our passion through freely available how to videos and courses related to business website marketing and optimization, Google (Search Console, Ads, Analytics, YouTube), SEO, HTML5, Structured Data and WordPress. Thank you for visiting our website. Rest assured that it will serve you well as well.

6 comments

  1. Also I would appreciate if you walk me through the last 3 bullet points, SEO tips. How and where do we incorporate the codes?

  2. I don’t know how I came across your youtube channel but I am glad I did. I am going to install the WP ex 2 accordingly. Hope to hear from me soon.

  3. Ever since last year, I’ve already subscribed to your youtube channel and learned much about robot.txt things.
    And it really helps me a lot.
    Now I am still facing problems to submit my sitemap to Google. I’ll leave comments if I’ve encountered any further problems.

    1. Great to hear that RankYa Digital Marketing how to videos has contributed to your knowledge. Furthermore, because you are using Google Maps on your site (I think you need to triple check it as its throwing an error) you will need to use Google Maps API

Leave a comment

Your email address will not be published. Required fields are marked *