How Much Important Is To Have Robots.txt File?

Money Making Forum - e-gold autosurf hyip mlm discussion | monitor. A place where leaders meet and grow! This is a positive community. Post your opportunity here. More investment Module coming soon.

Place your 468x60 banner below for as low as $87 for 5 days

-Paid Advertisement- DTM does not endorse any of the program advertised

DreamTeam Money Making Forum Register

Contact Us Rules Link To Us Directory Advertise on DTM Help Log In

Welcome Guest ( Log In | Register )

SecureAssets

Place your Rotating 728x90 banner below for as low as $176 for 5 days or Non Rotating banner for $1130 for 5 days

Advertisement

-Paid Advertisement- DTM does not endorse any of the program advertised

DreamTeamMoney Forum > Internet Marketing, Affiliate Marketing, SEO, Marketplace, Webmaster & Social Media Lounge > Internet Marketing, Affiliate Marketing, SEO, Marketplace & Webmaster Lounge > Search Engine Optimization (SEO) > SEO Discussion

How Much Important Is To Have Robots.txt File?

post

Jun 24 2010, 12:30 PM

Post #1

Active Money Maker

Group: Members
Posts: 101
Dream Points: 460
Experience Points: 151

Joined: 22-June 10
Member No.: 100,348

Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?

--------------------

Online trading account | Mutual funds | Financial advisor |
Financial planning

User is offline

Go to the top of the page

Quote Post

2 Pages

1 2 >

Reply to this topic

Start new topic

Replies(1 - 9)

musicworld1	Jul 15 2010, 07:53 AM Post #2
New Money Maker Group: Members Posts: 23 Dream Points: 320 Experience Points: 89 Joined: 15-June 10 Member No.: 99,556	QUOTE(jiten702 @ Jun 24 2010, 12:30 PM) Hi all...... As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO? Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file. If your Web site doesn't have a robots.txt file yet, read on to learn how to create one. If you already have a robots.txt file, read our tips to make sure that it doesn't contain errors. -------------------- Top Ten Lists \| Top 10 Rappers

fixaprob	Jul 23 2010, 07:49 AM Post #3
Serious Money Maker Group: Members Posts: 1,252 Dream Points: 10,473 Experience Points: 1,322 Joined: 24-September 09 Member No.: 73,146	Do you really need a robots.txt even if you're not excluding any robots? It's a good idea. Why? First and foremost, it's an invite to the search engines. In addition, some of the good bots may step away from your website if you do not have a robots.txt created in the top level of your website. Sometimes you may want to exclude some pages from the search engine's eye. What type of pages? 1. Pages that are still under construction 2. Directories that you would prefer not to have indexed 3. Or you may want to exclude those search engines whose sole purpose is to collect email addresses or who you do not what to have your website appear in. -------------------- Profit is just a click away - packages available now >>> FREE TRIAL FOR TWO DAYS COMMODITY TIPS

alevin46	Aug 21 2010, 08:06 AM Post #4
New Money Maker Group: Members Posts: 19 Dream Points: 240 Experience Points: 69 Joined: 19-August 10 From: UK Member No.: 107,372	Hello Friends....... Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results. Thanks -------------------- best windows hosting

Asadd	Sep 25 2010, 09:40 AM Post #5
Junior Money Maker Group: Members Posts: 66 Dream Points: 710 Experience Points: 118 Joined: 3-September 10 Member No.: 109,331	Here is a good article to read. businessseek.biz/article-directory/article-59.html -------------------- Technology Articles \| Automated Testing

louis2609	Nov 7 2010, 06:57 AM Post #6
New Money Maker Group: Members Posts: 8 Dream Points: 104 Experience Points: 60 Joined: 7-November 10 Member No.: 118,334	QUOTE(jiten702 @ Jun 24 2010, 12:30 PM) Hi all...... As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO? In my opinion, i think robots.txt file tells search engines whether they can access and therefore crawl parts of your site. -------------------- Marketing Online \| SEO Solutions \| SEO \| Japanese Games \| Famicom

Debbie Johnson	Nov 7 2010, 08:29 AM Post #7
Legendary Money Maker Group: Lifetime Supporters Posts: 16,723 Dream Points: 56,863 Experience Points: 17,322 Joined: 15-April 07 Member No.: 15,677	What Is A Robots.txt File & Do You Need One? Rand Fiskin, Jim Boykin, Danny Sullivan (on Daggle), John Andrews, XMCP, Michael Martinez, Marty Weintraub were notable for the lack of a detectable Robots.txt file, so no, it appears it’s certainly not necessary to include a Robots.txt in your website, and I survived without one for long enough. A Robots file can, simply, tell search engines certain pages of your site not to return in search results crawl. It can also save valuable bandwidth. -------------------- Money Forum Lens ~~Don't click here $~~ Follow me on Twitter

jerryton	Nov 7 2010, 11:25 PM Post #8
Senior Money Maker Group: Members Posts: 632 Dream Points: 4,762 Experience Points: 687 Joined: 14-March 10 Member No.: 88,726	Its useful if you have pages that are duplicates or are no use being indexed, such pages include registration pages, checkout pages etc.. if indexed they can take away some pagerank from your more important pages. -------------------- credit \| loan \| loans \| prepaid finance

cheapseoservices	Nov 9 2010, 12:39 PM Post #9
Junior Money Maker Group: Members Posts: 29 Dream Points: 340 Experience Points: 79 Joined: 1-October 09 Member No.: 73,723	It is important to have robots.txt and instead of it you can use more convenient html: <meta name="robots" content="index, follow" /> <meta name="robots" content="all" /> <meta name="revisit-after" content="7 days" /> <meta name="revisit" content="7 days" /> -------------------- Affordable SEO services - SEO packages, Offsite Onsite SEO, Directory submission, SEO Content creation, Wordpress SEO, Joomla SEO, Adwords management.

kitty15	Nov 22 2010, 09:44 AM Post #10
New Money Maker Group: Members Posts: 21 Dream Points: 260 Experience Points: 71 Joined: 18-November 10 From: UK Member No.: 120,222	Hello Friends........... When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. Thanks -------------------- asp.net based cms

« Next Oldest · SEO Discussion · Next Newest »

Sponsor Ads:

Advertisement

Place your 728x90 banner below for as low as $87 for 5 days

-Paid Advertisement- DTM does not endorse any of the program advertised

Related Topics

Related Topics

Topic Title	Replies	Topic Starter	Views	Last Action
I Have $5000, How Can I Invest It?	6	andreybyldakov	137	Yesterday, 02:36 PM Last post by: mirza_987
Have Wu, Need Libertyreserve Or Mb (skrill)	4	dumisache	220	Yesterday, 01:02 PM Last post by: Nanko
02hosting - Have You Known Them?	2	Gathering	25	Yesterday, 12:47 PM Last post by: Nanko
Send As Much As 200,000 Emails Per Day	0	cindy42	27	Yesterday, 12:14 PM Last post by: cindy42
Forum Posting Is Important????	3	royalsmith	30	Yesterday, 11:50 AM Last post by: petersmith5544

2 Pages

1 2 >

Reply to this topic

Start new topic

1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)

0 Members:

Display Mode: Switch to: Standard · Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum

-Paid Advertisement- DTM does not endorse any of the program advertised

Lo-Fi Version

Advertise on DTM | Rules | Disclaimer | Privacy Policies

Time is now: 23rd May 2013 - 06:27 AM

© 2005 - 2012 All Rights Reserved
www.dreamteammoney.com Money Making Forum