php script directory webmaster resources
php scripts
Best Rated		 asp scripts
New Scripts		 java scripts
Popular Scripts		 perl scripts
Advanced Search
   
Home About Us FAQ
Add New Script Edit Script Contact Us
PHP :: Web Search :: Robots.txt parser
Robots.txt parser    Hits: 19


Description:
Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).


 Resource Specification
Platform(s): linux, windows, freebsd, osx, sun
Date Added: July 31, 2006
Script ID: 1314



Reviews by Other Users

Average Rating This Year (0.00 out of 5) from 0 reviews

Average Rating Last Year (0.00 out of 5) from 0 reviews

Read Reviews

Post Review


 


Home | About Us | FAQ | Add a Listing | Login | Contact Us | Partners | Getting Rated

php script directory | php email scripts | webmaster resources | php scripts | free web site templates


email marketing | online store software | web file manager | customer support software | knowledge base software
email forms | cheap web hosting | viral marketing software | affiliate software
email hosting | applicant tracking software | php article software
Copyright © 2004 ClickitScripts.com. All rights reserved.