Standalone application that starts with a URL and proceeds to visually graph all links recursively.

A simple Web crawler that searches web pages for key words.

TCSS 442 - UW - Autumn 2009 - Project 01 - Webcrawler

a web crawler for the friend recommender system

CS421 Programming Assignment 1

=OsoFramework= Web robot framework, works with .NET 3.5 and Mono 2.3. Example code: {{{ // using read string goog1 = Read(new HttpSettings { Query = "http://www.google.com" }); // using get XDocument for url // pre parsing is for manually parsing non conforming XHTML var goog2 = Read...

The *ldspider* project aims to build a web crawling framework for the linked data web. Requirements and challenges for crawling the linked data web are different from regular web crawling, thus this projects offer a web crawler adapted to traverse and harvest sources and instances from the linked ...

SLBCrawler is a highly scalable distributed web crawler written in Erlang.

An open source implementation of distributed web crawler that uses hadoop and map reduce

This project propose a method to discover the schema of a website, to implement the numerical analysis on web pages. The final goal is to implement the Unsupervised Learning on Web Information Extraction. 综合目前对信息抽取技术的分析，本文借鉴在搜索引擎中，将文档集转换成向量空间的思路上，提出将网页布局数值化的方法，用以实现网页信息抽取的自动化。在本文构建的向量空间中，DOM树中的每一个...