VMwareTest

taking the vmware image for a quick spin

Updated Mar 23, 2011 by jeremy.c...@gmail.com

Download the vmware player

Note the below image install functions are for a desktop-PC based 'player' version, while the latest Xenia image is ESXi 4.0 server-based. To convert to a player-based version of the image, see instructions at http://blog.laksha.net/2009/10/how-to-create-virtual-machine-using.html or use the vmware vCenter convertor standalone application(should be available for free from vmware website).

Windows

Goto vmware player download page at http://www.vmware.com/download/player/download.html and download, install windows version(2.5.1 tested)

Linux

Goto vmware player download page http://www.vmware.com/download/player/download.html and download the Linux version 2.5.1(the 32-bit, not 64-bit) .bundle file

Used the instructions at https://help.ubuntu.com/community/VMware/Player for guidance, only had to run the following line

gksudo bash ./VMware-Player-2.5.1-126130.i386.bundle

Macintosh

see http://www.vmware.com/products/fusion

Download the vmware image instance

The latest vmware image download instructions are at VMwareDownload

This vmware image was developed on top of the FOSS geospatial vmware image(geostatistics package which includes 'R') available at http://gisvm.com (appliance description) - thanks to Ricardo Pinho.

While the Xenia scripts do not directly utilize/interact those included applications in the gisvm image at this time, I thought they represent the best existing collection of FOSS geospatial software for future development.

Start the vmware image on the vmware player by opening the associated .vmx file

Startup

The image being played may report in the player title the gisvm version 'gisvm20081101en_r' which can be ignored. I have not determined how to set this title properly yet.

For the 'virtual' server on the vmplayer menu at screen top, changed the 'Devices->Network Adapter' setting to 'Bridged' which establishes network communication between the virtual server and host.

The image should boot to a command line user login. Login as user xeniaprod with password xeniaprod99

I have the GUI display (gdm) disabled by default. To start a GUI session enter the following command and wait for the GUI to display.

sudo gdm start

If you wish to re-enable the GUI default behaviour see instructions at http://ubuntuforums.org/showthread.php?t=664199

You may also need to set the radio button 'Wired Network' from the top menu icon(circled in red) in the following image

At this point your image instance should have internet connectivity and you should be able to start a firefox browser(browser shortcut at top-of-screen menu) which will default to the website at http://gisvm.com Typing in the URL http://localhost/xenia should display the same as the above screenshot.

While leaving the xenia vmware instance running, it is by default using the xeniaprod crontab to once an hour(at the top of the hour) fetch 3 sample organization hourly data feed files formatted as ObsKML (see ObsKML simple sample below) to populate a local sqlite database and produce several products.

The main file which creates the data process flow can be invoked manually by the following command (also automatically executed hourly at the top of the hour by the xeniaprod crontab)

bash /home/xeniaprod/cron/xeniaflow.sh

After the xeniaflow.sh script has executed, most of the products listed at VMwareProductsLocal should be populated locally to the vmware image instance(start a firefox browser image within the instance and goto http://localhost/xenia which should display the same as the above screenshot). A few of the graph products,etc require two or more runs before displaying properly so you may have to wait a few hours/passes or so to start seeing proper results.

Adding a new ObsKML feed

note - ObsKML is an XML feed oriented way of supplying new data to the Xenia database, but writing SQL INSERT statements to import/populate or ASCII/CSV export data are valid approaches also - in this case the import/export scripts at http://code.google.com/p/xenia/source/browse/#svn/trunk/postgresql/import_export (in particular http://code.google.com/p/xenia/source/browse/trunk/postgresql/import_export/obskml_to_xenia_postgresql.pl ) may be helpful in understanding the structure of the data as stored in the relational database

How to add additional data feeds(obskml formatted) beyond those already included (usgs, nerrs, ndbc, nws).

Note the following currently available ObsKML feeds at http://carocoops.org/obskml/feeds/

ndbc (National Data Buoy Center) included with test install
nos (National Ocean Service - Tides and Currents)
nws (National Weather Service) included with test install
usf (University of South Florida)
nccoos (North Carolina Coastal Ocean Observing System)
cormp (UNCS Coastal Ocean Research and Monitoring Program)
carocoops (Carolinas Coastal Ocean Observing and Prediction System)
wq (this is Southeast USGS coastal stations) included with test install
nerrs (National Estuarine Research Reserve System) included with test install
vos (NOAA Volutary Observing Ships)
can add more ObsKML feeds to an online catalog if community interest

The following example demonstrates how the hourly ObsKML data feed was added for NWS as available from

http://carocoops.org/obskml/feeds/nws/nws_latest_obskml.zip

Changes

All the following steps are done as user xeniaprod

Create necessary organization folder

mkdir /home/xeniaprod/feeds/nws

Edit /home/xeniaprod/cron/getObskml.sh

add line to retrieve obskml files(one or several as .kmz,.zip) to local folder

wget http://carocoops.org/obskml/feeds/nws/nws_latest_obskml.zip -O /home/xeniaprod/feeds/nws/nws_metadata_latest.kmz

Edit /home/xeniaprod/cron/mk_xenia_all_latest.sh

add line to unzip one or several input obskml files(.kmz,.zip) to common temporary folder for common zip/processing

unzip -q "../nws/nws_metadata_latest.kmz" >> /home/xeniaprod/tmp/cron.log 2>&1

Add organization name to be included in status graphs, email notification

file /home/xeniaprod/config/config.xml

  <Organizations>
    <org>
      <name>nws</name>
      <count>1</count>  <!-- lowest expected count/email alert threshold -->
    </org>
...

utilized by file(as $XMLConfigFile, no changes) /home/xeniaprod/scripts/sqlite/flow_monitor/check_status.pl

my ($dbname,$email_flag,$hour_offset,$status_file,$status_file_latest,$image_file,$title,$range_top,$XMLConfigFile) = @ARGV;

optional

add organization to styled KMZ

file /home/xeniaprod/cron/styleLatest.sh

add org name to array list

my @org_array = qw(usgs nerrs ndbc nws);

mkdir /home/xeniaprod/feeds/nws/archive

Edit xeniaprod crontab to nightly cleanup archive files

As user xeniaprod run the following command to edit xeniaprod crontab in 'nano' editor crontab -e add line

2 0 * * * find /home/xeniaprod/feeds/ndbc/archive/* -maxdepth 1 -cmin +1400 -exec rm -f {} \;

ObsKML simple example

The below sample kml file shows an example of two observations listed at a platform. The xml schema is available here

ObsKML is not a community standard at this time, but an in-house developed convention for staging/ingesting imported data to a common XML format using KML as a wrapper(GeoRSS,Atom/JSON could also possibly utilize this data content schema also). Further ObsKML documentation is available here. Specific instrumentation examples are available at InstrumentationExamples

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.1"
     xmlns:obskml="http://carocoops.org/obskml/1.0.0/obskml_simple.xsd">
<Document>
  <name>Obskml sample</name>
  <open>1</open>
  <Placemark id="carocoops.CAP1.buoy">
    <name>carocoops.CAP1.buoy</name>
    <description>An html table derived from the obs kml metadata tags would be displayed here</description>
    <Point>
      <coordinates>-79.68,32.86,0</coordinates>
    </Point>
    <TimeStamp><when>2007-01-15T14:00:00</when></TimeStamp>
 
    <Metadata>
    <obsList>
    <obs>
      <obsType>air_temperature</obsType>
      <uomType>celsius</uomType>
      <value>21</value>
      <elev>3</elev>
    </obs>

    <obs>
      <obsType>water_temperature</obsType>
      <uomType>celsius</uomType>
      <value>16</value>
      <elev>-1</elev>
    </obs>
    </obsList>
    </Metadata>

  </Placemark>
</Document>
</kml>

The above schema represents two observations from a single platform.

The 'elev' tag in the above schema corresponds to elevation. Positive is meters above sea level and negative is meters below sea level. Missing or unknown elevation values can leave an empty elev tag like <elev />

A recommended but not required convention is to use the Placemark id attribute to help uniquely identify observations between data providers. The recommended convention is the concatenation of <organization>.<platform>.<package> like carocoops.CAP1.buoy or nws.KCAE.met

Observation listing order is preferred to be from highest elevation/altitude to lowest elevation/depth.

Redundant sensors/observations would be listed in their order of importance (primary, secondary, etc).

Adding new measurement types (m_type_id)

The existing database supported data dictionary of measurement types(m_type) which is the combination of observation type(obsType) and unit of measurement type(uom_type) is listed here

If you would like to add additional measurement types see documentatation here and database schema diagram here

Example scripts for generating ObsKML

Example scripts detailing how to create ObsKML from several data source types are documented at

http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/ObsKMLGenerate

Note that the ObsKML schema does not currently utilize an 'obskml:' namespace convention but may in a future version.

If you have created an ObsKML feed that you would like to share please email me (jeremy.cothran@gmail.com) and I will add it to an online community catalog.

► Sign in to add a comment