Format Of Robots.txt

Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic

   How do you setup a robots.txt to correctly spider only the productcart files necessary for good page ranks but to ignore the other asp pages?

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
cart Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 08-March-2007 Status: Offline Points: 0	Post Options Post Reply Quote cart Report Post Thanks(0) Quote Reply Topic: Format Of Robots.txt Posted: 08-March-2007 at 7:39am
	How do you setup a robots.txt to correctly spider only the productcart files necessary for good page ranks but to ignore the other asp pages?
	ct

NWilliams Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 19-November-2006 Location: United States Status: Offline Points: 0	Post Options Post Reply Quote NWilliams Report Post Thanks(0) Quote Reply Posted: 10-March-2007 at 7:09pm
	The Web Robots Page Noone's going to do it for you, so the best solution is learn how it works. You'll find syntax information on the "Robots Exclusion" page.

watercrazed Members Profile Send Private Message Find Members Posts Add to Buddy List Groupie Joined: 31-December-2005 Location: United States Status: Offline Points: 0	Post Options Post Reply Quote watercrazed Report Post Thanks(0) Quote Reply Posted: 01-April-2007 at 8:05pm
	This is what I use, what about the rest of you? anything I am missing? Disallow: /ProductCart/calendar/ Disallow: /ProductCart/cartdata/ Disallow: /ProductCart/htmleditor/ Disallow: /ProductCart/pcadmin/ Disallow: /ProductCart/UPSLicense/ Disallow: /ProductCart/database/ Disallow: /ProductCart/setup/ Disallow: /ProductCart/pc/New Disallow: /ProductCart/pc/catalog/ Disallow: /ProductCart/pc/advSrca.asp Disallow: /ProductCart/pc/Affiliate Disallow: /ProductCart/pc/advSearch_h.asp? Disallow: /ProductCart/pc/allreviews.asp? Disallow: /ProductCart/pc/checkout.asp Disallow: /ProductCart/pc/custPref.asp Disallow: /ProductCart/pc/custva.asp Disallow: /ProductCart/pc/Custva.asp? Disallow: /ProductCart/pc/custwl.asp Disallow: /ProductCart/pc/default.asp Disallow: /ProductCart/pc/tellafriend.asp? Disallow: /ProductCart/pc/ViewCart.asp Disallow: /productcart/pc/viewCat_h.asp?ProdSort
	John massagers, heat therapy, buckwheat pillows and more

Stuck Members Profile Send Private Message Find Members Posts Add to Buddy List Groupie Joined: 09-March-2007 Location: United States Status: Offline Points: 0	Post Options Post Reply Quote Stuck Report Post Thanks(0) Quote Reply Posted: 18-April-2007 at 9:54pm
	Thanks for sharing this John! I am just now creating our robots.txt file and this was a big time saver.

carstone Members Profile Send Private Message Find Members Posts Add to Buddy List Groupie Joined: 12-July-2006 Status: Offline Points: 0	Post Options Post Reply Quote carstone Report Post Thanks(0) Quote Reply Posted: 14-March-2009 at 7:04pm
	I would like to revive this question again, because I think there has been folder structure changes since this was last discussed. What is a good robots.txt file for PC. Also, should we NOT list some folders, like our ADMIN folder, in the robots.txt file because it shows a savvy surfer where the sensitive folders are??? I can go to anyone's site and type www.domain.com/robots.txt and learn a lot about their websites structure. Is that a real problem?

ProductCart Members Profile Send Private Message Find Members Posts Add to Buddy List Admin Group ProductCart Team Joined: 01-October-2003 Status: Offline Points: 135	Post Options Post Reply Quote ProductCart Report Post Thanks(0) Quote Reply Posted: 15-March-2009 at 7:14am
	We do not believe this is necessary. A search engine spider will only locate and index pages that you (or the shopping cart itself) are linking to, and those are only the storefront pages. We don't see any reason why some of the storefront pages should be excluded from the index. For example, why would you want to prevent the search pages from being indexed? A hardcoded link to a price-range search, for instance, could certainly be indexed. And why preventing product reviews from being found and indexed? Product reviews, which are text-heavy, typically rank well in search engines. We honestly don't see any reason for doing this. If there are some that we are not seeing, certainly discuss them in this thread. Even if you decide to use robots.txt, the renamed "pcadmin" folder should absolutely not be included in the robots.txt file: if you do so, you are making its path known to everyone. Hiding the path to the ProductCart Control Panel is a simple and strong security measure (beyond the Control Panel's built-in authentication system), and it should be implemented on all ProductCart-powered stores.
	The ProductCart Team Home of ProductCart shopping cart software