![]() |
![]() |
|
|||||||||||||
|
|||||||||||||||
| PHP :: Web Search :: Robots.txt parser |
| Robots.txt parser | Hits: 19 |
|
| Description:
Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details). |
| Resource Specification | |
| Platform(s): | linux, windows, freebsd, osx, sun |
| Date Added: | July 31, 2006 |
| Script ID: | 1314 |
|
Reviews by Other Users
Average Rating This Year (0.00 out of 5) from 0 reviews Average Rating Last Year (0.00 out of 5) from 0 reviews |
|
|
| Copyright © 2004 ClickitScripts.com. All rights reserved. |