| Sign in

no summary

Project Home		Downloads		Wiki		Issues		Source

Search

Updated Jun 11, 2008 by stefan.petrea

full_list_v08

list of features crawler

Full list of features for Crawler v0.8

crawls links from the forum
if closed from interface remembers where it left off
is able to crawl multiple links at the same time
the user is able to customize the work time and the pause time
is able to use proxies
for each link provided by the user stores data in a differnt "data.txt"
has an easy to use interface
the user is able to download all the data.txt files from the web interface
features a 3-time retry on a link if failure occurs,and if those don't work starts using proxies
the user can delete easily from the interface all data associated with any link that was crawled

Sign in to add a comment

Hosted by