Format Of Robots.txt
Printed From: ProductCart E-Commerce Solutions
Category: ProductCart
Forum Name: Search Engine Optimization
Forum Description: Talk about ways to optimize your ProductCart store for search engines
URL: https://forum.productcart.com/forum_posts.asp?TID=688
Printed Date: 10-May-2024 at 8:09am
Software Version: Web Wiz Forums 12.04 - http://www.webwizforums.com
Topic: Format Of Robots.txt
Posted By: cart
Subject: Format Of Robots.txt
Date Posted: 08-March-2007 at 7:39am
|
How do you setup a robots.txt to correctly spider only the productcart files necessary for good page ranks but to ignore the other asp pages? ------------- ct |
Replies:
Posted By: NWilliams
Date Posted: 10-March-2007 at 7:09pm
|
http://www.robotstxt.org/wc/robots.html - The Web Robots Page Noone's going to do it for you, so the best solution is learn how it works. You'll find syntax information on the "Robots Exclusion" page. |
Posted By: watercrazed
Date Posted: 01-April-2007 at 8:05pm
|
This is what I use, what about the rest of you? anything I am missing? Disallow: /ProductCart/calendar/ Disallow: /ProductCart/cartdata/ Disallow: /ProductCart/htmleditor/ Disallow: /ProductCart/pcadmin/ Disallow: /ProductCart/UPSLicense/ Disallow: /ProductCart/database/ Disallow: /ProductCart/setup/ Disallow: /ProductCart/pc/New Disallow: /ProductCart/pc/catalog/ Disallow: /ProductCart/pc/advSrca.asp Disallow: /ProductCart/pc/Affiliate Disallow: /ProductCart/pc/advSearch_h.asp? Disallow: /ProductCart/pc/allreviews.asp? Disallow: /ProductCart/pc/checkout.asp Disallow: /ProductCart/pc/custPref.asp Disallow: /ProductCart/pc/custva.asp Disallow: /ProductCart/pc/Custva.asp? Disallow: /ProductCart/pc/custwl.asp Disallow: /ProductCart/pc/default.asp Disallow: /ProductCart/pc/tellafriend.asp? Disallow: /ProductCart/pc/ViewCart.asp Disallow: /productcart/pc/viewCat_h.asp?ProdSort ------------- John http://www.ultimatewatermassage.com - massagers, heat therapy, buckwheat pillows and more |
Posted By: Stuck
Date Posted: 18-April-2007 at 9:54pm
|
Thanks for sharing this John! I am just now creating our robots.txt file and this was a big time saver. |
Posted By: carstone
Date Posted: 14-March-2009 at 7:04pm
| I would like to revive this question again, because I think there has been folder structure changes since this was last discussed. What is a good robots.txt file for PC. Also, should we NOT list some folders, like our ADMIN folder, in the robots.txt file because it shows a savvy surfer where the sensitive folders are??? I can go to anyone's site and type www.domain.com/robots.txt and learn a lot about their websites structure. Is that a real problem? |
Posted By: ProductCart
Date Posted: 15-March-2009 at 7:14am
|
We do not believe this is necessary. A search engine spider will only locate and index pages that you (or the shopping cart itself) are linking to, and those are only the storefront pages. We don't see any reason why some of the storefront pages should be excluded from the index. For example, why would you want to prevent the search pages from being indexed? A hardcoded link to a price-range search, for instance, could certainly be indexed. And why preventing product reviews from being found and indexed? Product reviews, which are text-heavy, typically rank well in search engines. We honestly don't see any reason for doing this. If there are some that we are not seeing, certainly discuss them in this thread. Even if you decide to use robots.txt, the renamed "pcadmin" folder should absolutely not be included in the robots.txt file: if you do so, you are making its path known to everyone. Hiding the path to the ProductCart Control Panel is a simple and strong security measure (beyond the Control Panel's built-in authentication system), and it should be implemented on all ProductCart-powered stores. ------------- The ProductCart Team Home of ProductCart http://www.productcart.com" rel="nofollow - shopping cart software |