PHP Web Spider (Robot)

phpspider is a web spider written in PHP. With phpspider you can download entire web sites or simply crawl them looking for broken links. The code can be executed from the command line or from a web server. Configuration consists of pointing the code at a web site, setting some configuration options, and starting the crawl.

System Requirements:

PHP 5.2+
PEAR Log package
PEAR HTTP_Request package
phpquery http://code.google.com/p/phpquery/

At present the system has several known issues, but works on most basic web sites. The biggest missing feature at present is a report at the end of the crawl summarizing the activity, including a list of broken links similar to that produced by linklint (a link checker written in Perl).