Super Web Crawler

Goal

• Implement a simple web crawler in python.
• Capable to serve as a vertical search backend crawler. (100k pages?)
• Store page text and meta data at Mysql db.

Working Environment

• Python 2.6.1
• Mysql 5.1.30
• GVIM 7.2 for windows
• MysqlDB for python.
• SVN

NOTE: Refer to the project links at right side.

Members

• Yunqiao Zhang
• Zhijian He