QUOTE(Ridz Ria @ Feb 6 2013, 04:05 PM)
Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.
Robots.txt also known as robots exclusion protocol, Robots.txt is a text file stored in a site's root directory that tells a search engine Crawler which site pages and sub-folders should not be included in the search engine Index. However, there is no guarantee that a Crawler will comply with this request. Robots.txt is an alternative to a Meta Robots Tag or password protection.