First of all if you just don’t know what the robots.txt is then you can check it out here and find some more info on it:- Link
It is used to allow or disallow robots from crawling the pages on your site.
If you don’t have one it might not hurt much in rankings on the search engine but if you have one then you should surely check if it is allowing search engines to crawl your site or not.
Now you would say who would do that kind of foolish thing and block the search engines completely and try to Optimize the site for the search engines. But here is what happened with one of my friend recently.
I was just visiting him and as usual we started his PC and he showed me some of his sites which were not performing well enough. He said he is doing really good internet marketing and still he is not seeing any good effects for his new forum.
I said well lets check even if it is indexed or not. I did checked Google’s index to see if there was a listing for his site or not. I didn’t found one. Not a single page indexed out of 1500 pages. Thought this is not good. Maybe he is banned due to excess of keywords on his pages.
Then I said lets check your awstats. He gave me his cpanel pass and to my surprise google bot not even visited his site once in last 8 months. I thought something is really wrong here. I was not hoping he would block google bot from his robots.txt an then ask me this kind of questions but still I thought lets check that too.
Well I did found that google bot was completely blocked from his site. I asked him why, what made him do this. He said he haven’t even touched the robots.txt file. This brought us to one conclusion that he was using one open source script for his web site and that script did supplied robots.txt file with it. So he might have uploaded it with installation of that script.
We downloaded the same script from the website and checked the robots.txt and yes it was mentioned right over there block google bot. Now I don’t know why but this was a good lesson and a thing to remember for me in future also.
If you have a blank robots.txt then scrapers and other link harvesting programs might create problems for you. But if you know nothing about robots.txt and there are no areas on your site that you don’t want the bots to see. Then just leave it blank. This tip is for new webmasters if you are an advanced webmaster then you might already be knowing what to block and what not to.