Changes to Google News – Google News Bot Retired

By August 25, 2011 Google News No Comments

Google has announced that they are changing the way they crawl News content. Previously, Google used a robot which they called GoogleBot News to index news content, so if you wanted to protect that particular content, you needed to no-index for that Robot.

Starting now, the normal GoogleBot which indexes all the normal web content will also be crawling News content instead of the GoogleBot News.

This is how Google News started out – with the one Bot indexing everything – but then it changed to two, so that publishers could easily block certain content from either web or news.

Now they are going back to their original way – which is fine, because you can still do the either/or blocking, BUT, it does mean that some of the more detailed reporting in WebMasterTools will be lost.

What does this mean for news providers?

Your content can still be protected from being indexed by Google News by disallowing Googlebot-News in robots.txt (or using meta robots tag), Google will respect that robots.txt if it is more restrictive than the robots.txt entry for Googlebot.

For news sites which require login/payment/registration, Google will of course only be able to index that information which is available to the normal public.

This will not result in any changes for your news sitemap, which will be indexed as normal.

SEO implications?

None, really. But this is a good time to remind news sites to;

  • Implement a Google News Sitemap
  • Remember that content hidden behind subscriber walls is of no value for SEO.


One New Idea To Help Achieve Your Sales Goals REQUEST YOUR FREE SPARK