HowItWorks - geodatastore - Project Hosting on Google Code

Updated May 26, 2008 by pamela.fox

HowItWorks

Model
  name = a string
  description = a string, can hold HTML
  type = a string, point, line, or poly
  dateModified = the date when last modified
  coordinates = a list of GeoPts, which contain lat and lng
  bound = bounding box, a list containing east, west, north, south floats
  timeStamp = the when first added
  altitudes = A list of altitudes as float
  userId = if user logged in, the userid that created it
  tags = A string list of tags

Here's what that looks like in code:

class Geometry(db.Model):
  name = db.StringProperty()
  description = db.StringProperty(multiline=True)
  type = db.StringProperty()
  dateModified = db.DateProperty(auto_now=True)
  coordinates = db.ListProperty(db.GeoPt, default=None)
  bound = db.ListProperty(float, default=None)
  timeStamp = db.DateProperty(auto_now_add=True)
  altitudes = db.ListProperty(float, default=None)
  userId = db.StringProperty(default=None)
  tags = db.ListProperty(unicode,default=None)

To query the datastore, we only have to call the gql method of Geometry: geometries = Geometry.gql(qryString). That returns an iterator over a list of Geometry entities which you can then process. For instance:

    geometries = Geometry.gql(qryString)
    outputAction = {'json': jsonOutput(geometries,'get'),'kml': kmlOutput(geometries)}

WebApp RequestHandler

GeoDataStore accepts both Get and Post requests, and essentially treats them the same. Post, of course, allows sending larger amounts of data and for shorter URLs, and is more secure. We decided to treat them the same, that is to allow adding data using Get, to allow for easy user testing of the datastore without using the Maps interface. These are the steps involved:

All operations go through the same RequestHandler: Request.

Request receives the request, which should include an operations parameter, and sends it to operationPicker.
Based on the operations parameter, operationPicker calls the appropriate CRUD method, addGeometries, getGeometries, editGeometries, or deleteGeometries.
Each operation conducts its business, and then returns a content and a contentType variable which specifies the MIME type to be used. That way the operationPicker method doesn't need to know anything about each operation.
It then writes out the response using self.response.out.write, after adding in the Content-Type header.

class Request(webapp.RequestHandler):
  def post(self):
    self.operationPicker()

  def get(self):
    self.operationPicker()

  def operationPicker(self):
      operation = self.request.get('operation')
      out,contentType = '',''
      if operation == 'add':
        out,contentType = self.addGeometries()
      elif operation == 'edit':
        out,contentType = self.editGeometries()
      elif operation == 'delete':
        out,contentType = self.deleteGeometries()
      else:
        out,contentType = self.getGeometries()
      self.response.headers.add_header('Content-Type', contentType)
      self.response.out.write(out)

At this point, the edit operation is dense. You can't edit a geometry without passing everything you know about the geometry back to GeoDataStore. This mirrors the Google Data Protocols (link), and the Atom Publishing Protocol (link) they implement, which similarly require full edits.

When GeoDataStore receives a request, it checks for the BBOX parameter. If it gets one, it splits the BBOX parameter on the comma, and passes it to the bboxWest,bboxSouth,bboxEast, and bboxNorth variables. It then passes these variables on to kmlOutput().

kmlOutput() iterates through the Geometry entites creating

<placemark>

elements. For each entity, it checks to see if there is a bboxWest, and if there is it makes sure that the appropriate properties of the entity fall within the bounding box:

    createPlace = True
    if bboxWest != None:
      if geometry.bboxWest > bboxWest and  geometry.bboxEast < bboxEast and geometry.bboxNorth < bboxNorth and geometry.bboxSouth > bboxSouth:
        createPlace = True
      else:
        createPlace = False
    if createPlace == True:

If it does, then the entity is used to create a

<placemark>

, otherwise it skips on to the next entity.

GeoPts and altitude

Latitude and Longitude values are stored separately as GeoPts for each geometry. This was done to demonstrate usage of GeoPt, which is a native App Engine type for coordinate data. However, GeoPt does not store altitude. This isn't a problem for 2D data, such as is used in the Maps API, but it is a problem for 3D data, such as data represented by KML. So, GeoStore stores the list of altitudes separately.

Output Currently, GeoDataStore provides two output formats, JSON and KML. Here's an example of the JSON output:

{operation: 'get', status: 'success', result:{geometry:{records:[{key: 'ag5nZW9zZXJ2LW1tYXJrc3IOCxIIR2VvbWV0cnkYGQw', name: 'The Silk Road Turkish Mediterranean Restaurant', type: 'point', description: '8-10 Arcade Road', timeStamp: '2008-03-29', coordinates: [{lat: 50.808741, lng: -0.540634}], altitudes: [0.0]},

As you can see, each action returns a JSON object that identifies the operation and the status of that operation ('success') and has a result object. Currently, the result object has only one child object, geometry, but it is built with the possibility that other types of objects might be included. The geometry object has one child, records, an array of geometry entities, with the following name/value pairs: key (uniquely identifies the entity in the datastore), name, type (point, line, poly), description, timeStamp (when it was added to the datastore), coordinates (an array of lat and lng values) and altitudes, an array of altitude values. If the operation fails, it returns an error message.

The KML document looks very different:

Each entity is a Placemark element with a Point, LineString, or Polygon in it. The name and description properties map exactly to the name and description elements. GeoDataStore creates the appropriate Geometry elements for each of the entities.

Front end

Retrieving data

As described earlier, the backend provides both JSON and KML output format. Since JSON is native JavaScript data format, that output is optimal for reading into the JS frontend. For loading it, we can take advantage of the API function GDownloadURL. GDownloadURL is a wrapper for the XMLHttpRequest that's used to request a file (not necessarily XML) from the server where the HTML page resides. The first parameter to GDownloadURL is the path to your file (in this case, the URL that invokes the Python script), and the second parameter to GDownloadURL is the function that's called when the file is returned to the JavaScript.

var url = url_base + 'request?operation=get&output=json'
GDownloadUrl(url, function(data, responseCode) { me.handleDataResponse_(me, data, responseCode); });

In the callback function, you first use eval() to store the JSON output into a local variable. Then, after verifying that the status of the operation was 'success',' iterate through the records array and call

createGeometry

on each record entry. That function is shown below:

geoserver.adminPanel.prototype.handleDataResponse_ = function(me, data, responseCode) {
  if (responseCode == 200) {
    var json_data = eval('(' + data + ')');
    if (json_data.status != 'success') return;
    switch (json_data.operation) {
      case 'get':
        var geometries = json_data.result.geometries;
        var bounds = new GLatLngBounds();
        for (var i = 0; i < geometries.records.length; i++) {
          var record = geometries.records[i];
          var geometry = me.createGeometry_(record);
        }
    }
  }
};

Modifying data

The user interface allows the user to modify the geometry loaded in, provided that it was created by them originally. To make changes in the data store, we once again use GDownloadUrl. To tell the server what type of operation we're doing, we specify it in the URL that gets sent as the first argument to GDownloadUrl ("operation=add" vs. "operation="edit"). Then we transform the geometry's JSON data into form-encoded parameters ("&name=Bla&description=BlaBla"), and send the data as the third argument to GDownloadUrl. This tells the API to turn the request into a POST (more secure than a GET), and to send that data as the body of the POST. That line of code is shown below:

GDownloadUrl(url, this.handleDataResponse_, url_params.join('&'));

Creating geometry

The GeoDataStore interface displays three types of geometry: markers (single points), polylines (arrays of coordinates), and polygons (the same as polylines, but filled). Thankfully, there are corresponding objects in the Maps API for each of those types of geometry: GMarker, GPolyline, and GPolygon, respectively. There are two situations where we must create geometry - when the map loads in the initial data, and when the user uses the EditControl and clicks the map to specify they'd like a new geometry in that location. To avoid code redundancy, we handle both of those situations with a generic createGeometry function. That function begins by creating either the Maps API object (one of the three above), then creates the corresponding div in the sidebar, and finally assigns various event listeners to the new geometry object. The beginning of that function is shown below:

geoserver.adminPanel.prototype.createGeometry_ = function(data, is_editable) {
  var me = this;
  if (data.type == 'point') {
    var geometry = new GMarker(new GLatLng(data.coordinates[0].lat, data.coordinates[0].lng), {draggable: true, icon: me.icons_.unchanged});
  } else if (data.type == 'line' || data.type == 'poly') {
    var latlngs = [];
    for (var i = 0; i < data.coordinates.length; i++) {
      latlngs.push(new GLatLng(data.coordinates[i].lat, data.coordinates[i].lng));
    }
    var geometry = (data.type == 'line') ? new GPolyline(latlngs, '#0000ff', 0,
    0.1) : new
    GPolygon(latlngs, '#0000ff', 0, 0.1, '#0000ff', 0.01);
    }
  // ...

Comment by dmc...@richmond.ca, Aug 10, 2008

Bravo and very well done!

Comment by chrisjmurray3, Aug 23, 2008

you've shown me the way, thanks.

Comment by animesh, Oct 11, 2008

great work! BTW, does not render well in Chrome. Wonder whether the bug is in Chrome or geodatastore.

Comment by chrisjmurray3, Oct 27, 2008

Now I want to make it work on django / app engine - any ideas for porting it pamela?

Comment by colombia, Nov 13, 2008

any other option for "advanced" spatial querys?

Comment by rlspelich, Dec 18, 2008

This is awesome. And to think I racked my brain trying to write the locator module on my own. Thank you.

Comment by guilherme.fernandes, Jan 27, 2009

Hi,

First of all, congratulations for the initiative.

I understand you're using the geohash number to find near locations, correct?

Early tonight, as I was trying to do the same thing for my app, I learned that this is one of geohash's limitations http://en.wikipedia.org/wiki/Geohash#Limitations

I still haven't figured out a way to find near locations in GAE, since it doesn't support multiple inequality filters: http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Restrictions_on_Queries

Do you guys have any comment to share on this subject? (search for near locations)

Thanks

Comment by lucio.piccoli, Nov 15, 2009

the getLocationsNear below queries the entire Geometry table.

will this work if there are more than a 1000 results items, as this is a Big table limitation?

snippet

def getLocationsNear(self, lat, lon, count, less_than=True):
hash = str(geohash.Geohash((lat, lon))) query = Geometry.all() query.filter('type = ', 'point') if less_than:
query.filter('geohash < ', hash) query.order('-geohash')

-lp