ProductCart E-Commerce Solutions Homepage
Forum Home Forum Home > ProductCart > Search Engine Optimization
  New Posts New Posts RSS Feed - Format Of Robots.txt
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Format Of Robots.txt

 Post Reply Post Reply
Author
Message
cart View Drop Down
Newbie
Newbie
Avatar

Joined: 08-March-2007
Status: Offline
Points: 0
Post Options Post Options   Thanks (0) Thanks(0)   Quote cart Quote  Post ReplyReply Direct Link To This Post Topic: Format Of Robots.txt
    Posted: 08-March-2007 at 7:39am

How do you setup a robots.txt to correctly spider only the productcart files necessary for good page ranks but to ignore the other asp pages?

ct
Back to Top
NWilliams View Drop Down
Newbie
Newbie


Joined: 19-November-2006
Location: United States
Status: Offline
Points: 0
Post Options Post Options   Thanks (0) Thanks(0)   Quote NWilliams Quote  Post ReplyReply Direct Link To This Post Posted: 10-March-2007 at 7:09pm
The Web Robots Page

Noone's going to do it for you, so the best solution is learn how it works.  You'll find syntax information on the "Robots Exclusion" page.
Back to Top
watercrazed View Drop Down
Groupie
Groupie
Avatar

Joined: 31-December-2005
Location: United States
Status: Offline
Points: 0
Post Options Post Options   Thanks (0) Thanks(0)   Quote watercrazed Quote  Post ReplyReply Direct Link To This Post Posted: 01-April-2007 at 8:05pm

This is what I use, what about the rest of you?

anything I am missing?


Disallow: /ProductCart/calendar/
Disallow: /ProductCart/cartdata/
Disallow: /ProductCart/htmleditor/
Disallow: /ProductCart/pcadmin/   
Disallow: /ProductCart/UPSLicense/
Disallow: /ProductCart/database/
Disallow: /ProductCart/setup/
Disallow: /ProductCart/pc/New
Disallow: /ProductCart/pc/catalog/
Disallow: /ProductCart/pc/advSrca.asp
Disallow: /ProductCart/pc/Affiliate
Disallow: /ProductCart/pc/advSearch_h.asp?
Disallow: /ProductCart/pc/allreviews.asp?
Disallow: /ProductCart/pc/checkout.asp
Disallow: /ProductCart/pc/custPref.asp
Disallow: /ProductCart/pc/custva.asp
Disallow: /ProductCart/pc/Custva.asp?
Disallow: /ProductCart/pc/custwl.asp
Disallow: /ProductCart/pc/default.asp
Disallow: /ProductCart/pc/tellafriend.asp?
Disallow: /ProductCart/pc/ViewCart.asp
Disallow: /productcart/pc/viewCat_h.asp?ProdSort

Back to Top
Stuck View Drop Down
Groupie
Groupie
Avatar

Joined: 09-March-2007
Location: United States
Status: Offline
Points: 0
Post Options Post Options   Thanks (0) Thanks(0)   Quote Stuck Quote  Post ReplyReply Direct Link To This Post Posted: 18-April-2007 at 9:54pm

Thanks for sharing this John!

I am just now creating our robots.txt file and this was a big time saver.

Back to Top
carstone View Drop Down
Groupie
Groupie


Joined: 12-July-2006
Status: Offline
Points: 0
Post Options Post Options   Thanks (0) Thanks(0)   Quote carstone Quote  Post ReplyReply Direct Link To This Post Posted: 14-March-2009 at 7:04pm
I would like to revive this question again, because I think there has been folder structure changes since this was last discussed. What is a good robots.txt file for PC. Also, should we NOT list some folders, like our ADMIN folder, in the robots.txt file because it shows a savvy surfer where the sensitive folders are??? I can go to anyone's site and type www.domain.com/robots.txt and learn a lot about their websites structure. Is that a real problem?
Back to Top
ProductCart View Drop Down
Admin Group
Admin Group

ProductCart Team

Joined: 01-October-2003
Status: Offline
Points: 135
Post Options Post Options   Thanks (0) Thanks(0)   Quote ProductCart Quote  Post ReplyReply Direct Link To This Post Posted: 15-March-2009 at 7:14am
We do not believe this is necessary. A search engine spider will only locate and index pages that you (or the shopping cart itself) are linking to, and those are only the storefront pages.

We don't see any reason why some of the storefront pages should be excluded from the index. For example, why would you want to prevent the search pages from being indexed? A hardcoded link to a price-range search, for instance, could certainly be indexed. And why preventing product reviews from being found and indexed? Product reviews, which are text-heavy, typically rank well in search engines.

We honestly don't see any reason for doing this. If there are some that we are not seeing, certainly discuss them in this thread.

Even if you decide to use robots.txt, the renamed "pcadmin" folder should absolutely not be included in the robots.txt file: if you do so, you are making its path known to everyone. Hiding the path to the ProductCart Control Panel is a simple and strong security measure (beyond the Control Panel's built-in authentication system), and it should be implemented on all ProductCart-powered stores.
The ProductCart Team

Home of ProductCart shopping cart software
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.04
Copyright ©2001-2021 Web Wiz Ltd.

This page was generated in 0.047 seconds.