Robots.txt file is just a text document you can create using Windows programs and name it as robots with a file extension .txt however, when you place rules within this document and place it on web server root folder, then, you can better control crawling of search engine bots such as Googlebot. Important to Note… Continue reading robots.txt File
Tag: Googlebot
Block Search Indexing with NO-INDEX
Caution should be given to the reader about using WordPress noindex nofollow settings as these settings guide Google about what part of your website, and which links on your WordPress site that it can index and follow. Because getting this right can benefit your web site, but getting this wrong can make your site (or… Continue reading Block Search Indexing with NO-INDEX
How to NOINDEX in WordPress
It is smart to begin with a caution before you learn how to NO-Index certain parts of your WordPress built website. Because getting these noindex directives for user-agents (search engines) like Googlebot wrong, can have unexpected results for websites built on WordPress. The easiest way to noindex parts of a WordPress site is to use… Continue reading How to NOINDEX in WordPress
Google Search Console Crawl Stats
Google’s ability to fetch and crawl web pages is incredibly fast and efficient, the crawl stats provides insights for Pages Crawled Per Day Kilobytes downloaded per day This option will show you how much Kilobytes (thousand bytes) has been downloaded while crawling your web pages Time spent downloading a page (in milliseconds) Time in milliseconds… Continue reading Google Search Console Crawl Stats
How Google Web Crawler Works
Have you ever wanted to know how Google’s web crawlers sees your web pages? Curious about what happens when Google requests a web page? If so, let’s learn about the Crawling process of Google search engine. Since Google is a search engine for many different media type, it also has different crawlers for different purposes.… Continue reading How Google Web Crawler Works
How Does Google See Your Site
Knowing how Google sees your web site can help you to better optimize your web pages for higher Google ranking results. So what happens each time Google visits your website? Googlebot: is the name given to Google’s web crawling user agent (also known as web spiders that access your website) Google Crawl Process Simplified Googlebot… Continue reading How Does Google See Your Site