Introduction

NetSpider is a .NET library which provides search engine functionality.

The library provides a simple API which enables simple content retrieval facilities. Content is retrieved according to various built-in rules which are sufficient for many purposes.

If you wish, it is very easy to extend NetSpider by creating classes that implement various core NetSpider interfaces. You can also replace various NetSpider factory classes with custom implementations.

Please take a look at the core library architecture for more information.

Features

Due to the current early stage of development some of these features are not yet implemented.

• Extensible architecture.
• A set of core retrieval policies which cover various aspects of content retrieval.
• Respectful of a websites robots policy.
• Adherance to standard robot etiquette / practices (rate limitting and preventing unnecessary retrieval).
• Various page parsing and content retrieval stategies.
• Support for multiple transport methods (or URI schemes).