It happened one summer in 2006.

I had just started my new job as SEO campaign manager for a well-known agency in a well known U.S. city.  One of my tasks was to deal the the problem of a particular Web site’s lack of organic search performance.  This site had done well for the agency because it had helped bring on board a very large paid search account which had resulted in a great deal of revenue. This site had about 100 articles and was completely search engine friendly in terms of on-site SEO and running a propriety CMS that seemed stable.  Word around the office was that the site was becoming very important for the company’s organic search initiatives.

I was flummoxed. I’d done everything I knew how to make the site rank for the keywords. So, I set out to do some serious link building and link building I did. Still, no movement in the SERPS.

Then, one day as I plowed through the Google SERPs I began to notice that the actual title tags from the site were not appearing. Instead, I saw “Untitled” or simply the domain name.

Something spurred me to look in this new Google Webmaster Tools product that had been launched a while back. I did this and noticed a message stating the site files had been blocked from crawling via robots.txt.

So, immediately I checked the Robots.txt file and saw:

# go away User-agent: * Disallow: /

How could I be so dumb? Immediately I contacted IT to remove this. Apparenlty they had pushed the robots.txt file from the deve server to the live server and the robots.txt file had been included with it.

Thanks Google!