QUOTE(jackrough @ Jul 16 2013, 08:02 AM)
Its not robo.txt its robot.txt. its an html file uploaded to a website to prevent search engine crawlers to crawl some of your pages. pages mentioned in this file could not be crawled by search engine spiders.
I think this answers it all.