Authors
---------------------
Bryce Boe (bboe _at_ cs.ucsb.edu)
Christo Wilson (bowlin _at_ cs.ucsb.edu)

About
---------------------
CRAWL-E was designed to crawl the web fast fast as possible with as little
development time as possible. It is only a framework, and requires the
development of a Handler module in order to function properly.

The CRAWL-E developers are very familiar with how TCP and HTTP works and using
that knowledge have written a web crawler intended to maximize TCP throughput.
This benefit is realized when crawling web servers that utilize persistent HTTP
connections as numerous requests will be made over a single TCP connection thus
increasing the throughput.

Other features of CRAWL-E are multiple HTTP request method support, the most
basic being GET, POST, PUT, DELETE, HEAD.

Installation
---------------------
Requirements: python >= python2.5
Run: python setup.py install
Note: You will probably need to be a root user to install the package.

Running
---------------------
Check out the page downloader in the examples folder. The script run_it.sh
should be sufficient to get you started.

The heart of CRAWL-E relies on extending the Crawle.Handler, in which one
must implement a process function. This function is where all the magic can
happen. We say can happen because it's entirely up to you what to do in this
function. Some possibilities are parsing links to add to the queue, simply
saving the contents of the page, or both!