DreamTeam Money Making ForumRegister
Contact Us Rules Link To Us Blog Directory Advertise on DTMHelpLog In
Welcome Guest ( Log In | Register )
Hi Friend,Please can tell me about something about robots.txt file?
It tells search engine bot what to index and what to not.
Hi FriendRobots.txt File is a on page activity, if we don't want to indexing of any page so use Robots txt file.
Robots.txt file allows or dis-allows certain pages or contents of a website to crawl or not. It is basically use to protect our website content from robots.
It is used to not crawl a webpage that contain duplicate information of your website. If you site contains two pages of same information, one page for reading by viewers and another for printing then, you can stop indexing the printable page, otherwise the information will be treated as duplicate.
The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
Featured Sites:
Sponsor Ads:
Advertisement
Follow Us:
Related Topics
