Why might Googlebot get errors when trying to access my robots.txt file?

Matt's answer:

MATT CUTTS: We have a fun question today from Palo Alto, California. Yang asks “I’m getting errors from Google Webmaster Tools about the Googlebot crawler being unable to fetch my robots.txt file 50% of the time, but I can fetch it with 100% success rate from various other hosts.” And then he goes on to say, I’m on a plain old server. He has an mit.edu host, so it should have pretty good uptime. OK, so let me just tell you a story. And I’m not saying you’re doing this, but some people try to cloak, and they end up making a mistake, and they end up reverse cloaking. And so when a regular browser visits, they serve the content. And then when Google comes and visits, they will serve empty or completely zero length content. So every so often, we see that where in trying to cloak people make a mistake and actually shoot themselves in the foot and don’t show any content at all to Google. But one thing that you might not know and most people don’t know, we just confirmed it ourselves, is you can use the free Fetch as Googlebot feature of Google Webmaster Tools on robots.txt. So if you’re having failures 50% of the time, then give that a try and see whether you can fetch it. Maybe you are load balancing between two servers, and one server has got some strange configuration, for example. But not everybody realizes that so let me just reiterate. You can fetch your robots.txt file. And sometimes there’s weird stuff going on with that by using the Fetch as Googlebot functionality that’s built into Webmaster Tools.

by Matt Cutts - Google's Head of Search Quality Team

Original video: