Overview

The GeoWeb is inspired by the Web itself and particularly the rich set of tools built for Weblogs. Publishing, distributing and consuming maps and spatial data is as simple as publishing a weblog and running an RSS aggregator.

However there is a crucial component missing in the GeoWeb: Ping servers, exemplified by Weblogs.com and blo.gs, which provide a simple central point to receive updates of new content, and distribute those updates to interested consumers.

geoping hopes to serve the commons, so that all parts of the GeoWeb have equal access to updates.

Technical Overview

This overview is intended to give a high level sketch of the technical solution. There are four components to the service:

• a JSON/XML RPC and RESTful API;
• a web form for submitting updates;
• publishing of latest updates on the website;
• an archive of updates.

Implementation

geoping is built with Erlang, a functional programming language, which is ideally suited for concurrent, reliable applications. Erlang achieves this by using lightweight processes, and message passing between these processes. Further it has no shared state threading, thus allowing for lock-free concurrency and keeping the entire system more robust.

Overview of different processes

Here is a brief overview of the different Erlang processes inside the Erlang virtual machine:

Update notifications will be stored in a queue. A spam checking process will check the frequency of update of each resource, rejecting those occurring too often. And as a crude change detection, it will make a HEAD request to see if the size has changed since last update. If the resource has been updated, information on that resource will be updated in the database. A separate process, the update checker, will periodically query for the latest changes from the database, publish changes in html and xml, and archive the published list.

Finaly, a Webserver together with the database and file storage engine, will respond to REST or RPC pings and also serve the static webpages.

Initial prototype

To benchmark the Erlang server, a Merb server will be quickly prototyped. Merb, which is a simple and extremely fast Ruby framework, is also ideal for a ping server. If the content-type of the RPC calls are the ones for xml or json, Merb automatically converts the call to a Ruby params hash-table, which can then be processed and a resulting hash-table can be coerced into a xml or json response, without the need for templates which would slow the service down.

Further, Merb comes with a background worker, which can execute some action after the request dispatch has finished. Such a worker, could then be used as a spam checker and also to query the latest updates periodically.

Site mock-ups

The initial site mock-ups are on this wiki page: SiteMockups

Proposed Schedule (TODO)

Week 1

Initial prototype and benchmark:

• Erlang prototype with a basic ping api;
• Merb prototype with similar api;
• Benchmarking on the same machine under similar loads.

Week 3

Initial work on the spam checking worker task/process:

• Need to find good heuristics for ping spams;
• Implementation and testing;
• Integration into the main service.

Week 5

Initial work on the database query task/process:

• Create a model for storage of the urls;
• Create a periodic worker task that queries the latest urls;
• Implement the witting of the changes.xml file;
• Implement the logic for archiving and retrieving the urls.

Week 7

Basic REST api and webforms

• Also the static webpages and info for using the API.

Week 8

Add the Geo component to the service.