My favorites ▼ | Sign in
Project Home Downloads Wiki Issues Source
Search
for
AggregateReleaseNotes  
release notes for ODK Aggregate
Updated May 14, 2012 by mitchellsundt@gmail.com

Aggregate v1.0.6 Production

  1. fix  issue 542  - Add a Refresh Credentials button on publishers. This button appears when the publishing to an external service (Google Spreadsheet or Fusion Tables) fails with a 401 (Unauthorized) error. Clicking the button will walk through the authorization screens for publishing to that service, yielding a new credential for that publisher.
  2. fix  issue 512  - Add a Purge Submission Data button on the Form Management / Submission Admin tab. This button enables you to delete submission data up to a specified date. Incomplete submissions are not deleted.
  3. fix  issue 422  - digest auth failing in 1.0.5 Release Candidate.
  4. Includes all functionality from the 1.0.5 release candidate.

Upgrading

See v1.0.4 Production Upgrading instructions.

Aggregate v1.0.5 Release Candidate

  1. Reduce memory requirements for KML and CSV generation to enable generation of larger datasets.
  2. Restore bookmark navigation after login challenges - v1.0.4 did not navigate to a requested bookmark page if the user was challenged to log onto the site.
  3. Add support for Oauth 1.0 publishing from Build into Appspot-hosted Aggregate instances. Oauth 1.0 tokens do not work on Tomcat servers.
  4. Add support for Oauth 2.0 access to Aggregate. Aggregate will accept any Google token granting access to https://www.googleapis.com/auth/userinfo.email with access rights equivalent to the indicated Google user.

Upgrading

This is a Release Candidate and should be considered less reliable than a Production release; please test it thoroughly before deploying.

The primary changes in this release are in the areas of authentication, permissions and page navigation. All the typical interactions have been tested, but failures may produce unexpected errors.

No special upgrade steps are required when upgrading from v1.0.4; see v1.0.4 Production Upgrading instructions when upgrading from versions prior to v1.0.4.

Aggregate v1.0.4 Production Release

  1.  issue 483  - unable to log into MySQL or PostgreSQL deployments with openId.
  2.  issue 484  - streaming does not function.
  3.  issue 486  - odk:length not respected on Google AppEngine
  4. Fix layout issue with HelpBook and misc. popups.
  5. Update all security jars and misc. others to latest releases.
  6. Update to Google AppEngine 1.6.1 SDK

Upgrading

NOTE: Google AppEngine requires special upgrade steps if you utilize odk:length.

Upgrading Google AppEngine

See v1.0 Production Upgrading instructions.

If none of your forms use odk:length to alter the maximum string length, then the upgrade is complete.

Otherwise, for ONLY those forms that use odk:length:

  1. Download the forms that use odk:length using the ODK Briefcase application.
  2. Delete the forms on Google AppEngine (this may take several minutes or hours if you have many submissions).
  3. Upload the forms from ODK Briefcase back onto your Google AppEngine instance.

Upgrading MySQL or PostgreSQL

See v1.0 Production Upgrading instructions.

odk:length has always been respected on MySQL and PostgreSQL; there are no additional steps to be performed on those systems.

Aggregate v1.0.3 Production Release

NOTE: MySQL and PostgreSQL deployments of 1.0.3 suffer from  issue 483 . As a work-around, if you can live without OpenID logins for google accounts, you can install 1.0.2, configure an ODK Aggregate username with password and Site Admin privileges, then upgrade to 1.0.3 and use the ODK Aggregate username to administer your site. Otherwise, continue to use 1.0.2; the fix for this issue will be in 1.0.4.

  1. updates to make ODK Aggregate OpenRosa 1.0 compliant provided the version is an integer value.
  2.  issue 416  - avoid full scan of submissions on the Submission Admin sub-tab.
  3.  issue 416  - improvements to submission processing and publishing to minimize quota usage.
  4.  issue 416  - change attachment handling to use lazy fetches, reducing quota. Eliminate immediate refetch upon change.
  5.  issue 468  - form delete consuming significant quota.
  6.  issue 478  - excessive quota during submission processing.
  7.  issue 478  - improvements to submission processing to regulate the number of background tasks launched. Fixes to limit rate of Watchdog firings.
  8.  issue 478  - move upload-submission background task to run on the foreground instance.
  9.  issue 474  - case-sensitive hostname compare fails.
  10. update libraries to the latest Apache commons and others
  11. plumb missing quota exceptions up through services API
  12. improve logging during background task activities
  13. add logging and improve tracking of watchdog executions

Upgrading

See v1.0 Production Upgrading instructions.

Downgrading

See v1.0.2 Production Downgrading instructions if downgrading to a version prior to v1.0.2.

Aggregate v1.0.2 Production Release

  1. fix  issue 458  - incorrectly presenting ⁄ for slashes in form ids.
  2. fix  issue 457  - forms containing slashes or characters that should be escaped could not be deleted.
  3. fix  issue 426  - publishing of dates and date-time values to Google Spreadsheet and Fusion Tables -- now recognized as dates.
  4. fix bug in publishing that was using wrong date during retries. Could cause some data to not be published to Google Spreadsheets and/or Fusion Tables.
  5. fix bug in image thumbnails that improperly set the size of the thumbnail
  6. fix bug in kml export that didn't properly escape strings
  7. fix bug in Google Spreadsheet publishing that delayed publishing start
  8. log application version during start-up.
  9. adjust refresh rates of the UI pages to lessen database accesses.  issue 416 
  10. add over-quota error pages.
  11. plumb up to the UI all over-quota exceptions -- as distinct from generic datastore failures.
  12. optimize all queries for GAE operations.  issue 416 
  13. move background process (/gae/watchdog) to run in backend instance.
  14. move background activities (watchdog, publishing, csv/kml export, purge submissions and delete form) to backend instance.
  15. watchdog is now run every 40 minutes, but only when activity is occuring on the webserver. If there is no work pending, the watchdog is not run.  issue 416 
  16. enable multi-threaded webserver operations (should reduce instance-hour consumption on heavily-used sites).
  17. improve low- and out-of-memory performance within GAE QueryImpl class. Lower the pre-fetch size and reduce the fetch limit to avoid timeouts on large dataset reads.  issue 416 
  18. change publishing to use longer timeout values to Fusion Tables and to dynamically adjust for slower (or faster) external service connections.
  19. reduce the batch size during publishing so as to minimize unnecessary datastore reads.  issue 416 
  20. improve error logging during publishing and form deletion.
  21. move Form creation and cache-management into FormFactory. Reduce cache lifetime to 3 seconds.  issue 416 
  22. better encapsulate and hide FormDefinition object within Form object.
  23. track and emit datastore query, query-read, get, put and delete usage counts (does not yet flush to log during webserver shutdown).  issue 416 

Upgrading

See v1.0 Production Upgrading instructions.

Downgrading

On Google AppEngine, should you need to roll back to an earlier release (e.g., v1.0.1 Production), you will need to manually delete the "background" backend. This can be done through the AppEngine dashboard, via the Backends link.

On MySQL and PostgreSQL, no special action is required.

Aggregate v1.0.1 Production Release

  1. add 30-second cache for Form definition  issue 416 .
  2. change GAE queue configuration to limit retries on failure  issue 416 .
  3. add logging on datastore accesses  issue 416 . Disabled for performance.
  4. double watchdog interval to 6 minutes  issue 416 .
  5. double webpage background-refresh interval to 10 seconds  issue 416 .
  6. change the default number of records to display to 100 (from 500)  issue 416 .
  7. fix  issue 410  publishing does not flush all data to external services.
  8. fix spurious error message on Form Management tab when Aggregate 1.0 has no form loaded (noted on  issue 412 ).
  9. fix integration test breakage.
  10. fix datetime string parse error that was causing rounding errors in the millisecond digit. Impacts all data display, extraction, publishing and Briefcase activities.

Upgrading

See v1.0 Production Upgrading instructions.

Aggregate v1.0 Production Release

  1. add version information on Site Admin / Preferences tab
  2. fix date/time parsing problems that were affecting Briefcase data transfers.

Upgrading

If you are running RC4, RC3, RC2, or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 4 (RC4)

  1. update javarosa library
  2. allow null URI in resume cursor
  3. fix  issue 387  ( issue 380 ) - Briefcase download generates invalid Submission XML
  4. NOTE: date/time data may not be properly parsed during Briefcase transfers.

Upgrading

If you are running RC3, RC2, or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 3 (RC3)

  1. fix  issue 345  - unable to change user passwords on Tomcat deployments
  2. fix  issue 317  - bookmarked pages weren't working because the hashes and query parameters were being stripped during authentication.
  3. fix  issue 375  - csv downloads (and kml downloads) only emited the first 500 records, not the entire dataset.
  4. filters could not be defined and/or were not working on Tomcat (MySQL, PostgreSQL) deployments.
  5. KML file exports now honor the active filter and emit only what that allows (plus the image, title and location fields).
  6. conform to OpenRosa Form Discovery API by handling the verbose and formID parameters.
  7. include a specially-built javax imageIO library that is redistributable so that Tomcat instances show thumbnail images out-of-the-box.
  8. integration tests are now working in the Maven build, using seliniumhq 2.8.0. Coverage reports are not yet available and test suite only verifies the app starts, the superUser can log in, and a form can be uploaded.

Upgrading

If you are running RC2 or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 2 (RC2)

  1. improved help content, including links to youtube videos.
  2. show only "complete" submissions on the Submissions tab.
  3. row filters now properly implement hide logic
  4. now paginates results on Submissions tab.
  5. fix for Google Spreadsheets publishing issue.
  6. new "Submission Admin" tab under "Form Management" supports marking as complete an incomplete submission (incomplete submissions are those lacking one or more of their audio, image or video captures).
  7. only submissions that are marked as complete are published.
  8. media file count on Forms List tab is now clickable and displays the filename, file type and file size of each uploaded media file associated with a form.
  9. fix bug causing csv and kml downloads to not automatically open Excel and Google Earth, respectively.
  10. fix bug causing Change Password to report failures (now waits up to 15 seconds for a server response before failing this operation).
  11. fix bug causing Permissions page to only display 15 usernames.
  12. update to gdata 1.46.0 libraries

Upgrading

If you are running RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 1 (RC1)

  1. further improvements to filters
  2. csv file export now applies the display filter active at the moment the user presses the Export button.
  3. restructure external services to make it easier to add new publishers
  4. restartable cursors for paging through large sets of submissions (implemented at the datastore layer, but not yet up into the UI).
  5. allow aggregate username to be designated as a superUser (not exposed in Installer; partial support in UI).
  6. updated to GWT 2.4 and AppEngine SDK 1.5.4
  7. remove Briefcase applet (functionality will be replaced by the Briefcase application).
  8. add large dataset (310,000 record) tests for restartable cursor.

Beginning with Release Candidate 1, there will be a data migration path across all further upgrades; we don't expect any further incompatible schema changes. However, in the event that there is an incompatible schema change, you will be able to migrate your data to a newer ODK Aggregate 1.0 instance via a Briefcase application that can download submissions from ODK Aggregate 1.0 RC1 and onward and upload those submissions into future ODK Aggregate releases.

An initial release of the Briefcase application will be available soon.

Upgrading

To upgrade from Beta rev.4, follow the instructions below.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance before deploying the RC1. If you are using MySQL or PostgreSQL, you should start with an empty database.

On GAE

You don't need to do anything. It should just work.

On MySQL or PostgreSQL

The persistent results tables that hold the exported csv and kml files have changed (adding support for emitting csv files filtered by the active filter).

  1. Stop Tomcat
  2. In your database's administration tool, connect to the database and:
    • drop table _persistent_results;
    • drop table _persistent_result_file_bin;
    • drop table _persistent_result_file_ref;
    • drop table _persistent_result_file_blb;
  3. copy the new WAR to the /webapps directory.
  4. Start Tomcat

Aggregate v1.0 Beta rev.4

  1. split management tab into 2 tabs (Management and Site Admin).
  2. hide tabs that the user doesn't have permission to access.
  3. add help tooltips and documentation pop-ups.
  4. single-click hide/show of metadata in UI.
  5. delete individual submissions through the UI.
  6. make explicit that string fields are truncated to 255 characters.
  7. allow upload of forms that do not have data types defined for all fields (with suitable warnings).
  8. add odktables features.
  9. added a new metadata field, _MARKED_AS_COMPLETE_DATE that tracks the date the submission was last marked as complete.
  10. fix  issue 311  that causes infinite redraw loop if a form definition is corrupted.
  11. fixed several bugs related to incomplete uploaded submissions -- submissions with only some of their media attachments present.
  12. gain a global lock during instance start-up to ensure that only one instance might be changing the configuration during that time.
  13. improve GAE query and task locking logic; introduce datastore settle-time logic, remove read-after-write attempts and inserted retries in key places to improve real-world performance.
  14. extend timeouts for FusionTables to minimize publishing failures.
  15. refactor MiscTasks, PersistentResults and FormInfo to no longer themselves be form and submission structures.
  16. rework treatment of metadata so that it is more like other data.
  17. clear session cookies more completely when logging out.

Upgrading

Changes to the database schema necessitate either using a clean appspot instance or deleting ALL data in your appspot instance before deploying the beta 4. If you are using MySQL or PostgreSQL, you should start with an empty database.

Aggregate v1.0 Beta rev.3 utf-8

This fixes 4 issues:

  1. utf-8 characters are not displaying or downloading properly from a Google AppEngine server. ( Issue 286 )
  2. filtering on metadata fields does not work. ( Issue 299 )
  3. filtering does not work with date fields or select1 fields. ( Issue 299 )
  4. insert a 3 second delay in order for the datastore to settle before streaming newly-submitted data records to external services. TBD: This may not be enough time when GAE datastore is having performance issues.

This update also exposes the "meta-is-complete" metadata on a submission. If a submission with a media attachment (e.g., image, audio recording or video capture) is uploaded but the attachment is not, the submission will be marked as not complete (false). Only completed submissions are published to external services.

Upgrading

If you are running Beta rev.3, you should just be able to deploy this on top of that installation. If UTF-8 characters do not display properly in your form definition, you will need to delete the form (and submitted data) then upload the form anew.

Otherwise, if you are not running Beta rev.3, then follow the steps required for upgrading to Beta rev.3 prior to moving to Beta rev.3 utf-8.

Aggregate v1.0 Beta rev.3

Upgrading

To upgrade from earlier Alpha or Beta releases, you must...

On GAE

  1. If running builds prior to Beta 2 on Google AppEngine cloud services, any forms with Decimal data will need to be deleted and reloaded. Decimal data had been represented as strings and is now represented properly as double-precision numbers in Google AppEngine. Note: while Briefcase can be used to download data, there is currently no way to upload your downloaded data into the beta 3.
  2. Disable writes to the datastore (via Application Settings / Disable Datastore Writes )
  3. Deploy to GAE.
  4. Delete all the records in these kinds (using Datastore Viewer):
    • opendatakit._granted_authority_hierarchy
    • opendatakit._registered_users
    • opendatakit._user_granted_authority
  5. Disable your application (via Application Settings / Disable Application )
  6. Enable writes,
  7. Enable your application,

On MySQL or PostgreSQL

  1. Stop Tomcat
  2. In your database's administration tool, connect to the database and:
    • drop table _granted_authority_hierarchy;
    • drop table _registered_users;
    • drop table _user_granted_authority;
  3. copy the new WAR to the /webapps directory.
  4. Start Tomcat

Upgrade Impacts

The primary impact is the loss of all registered users and their privileges (what the manual steps above do). Beta 3 changes the user configuration. Registered users are now either gmail accounts or ODK Aggregate usernames. ODK Aggregate now supports only anonymous access and/or registered users.

Known Issues

  •  issue 265  -- IE 8 doesn't refresh the submissions page on adding or deleting forms.
  • issue 261 -- logins can time out while sitting on permissions page, causing unsaved changes to be lost.
  •  issue 231  -- large submission sets may cause UI to time out.

Changelist

  • recognize OpenRosa meta/instanceID tag and use it if present. Generalize to use the first meta/instanceID as the unique ID for a submission (don't require a namespace).
  • change-password now uses https when available (JSOP).
  • digest auth (ODK Aggregate logins) now have nonce timeout set to 30 minutes.
  • better logging of submissions and reasons for failures (for GAE diagnosis, since ODK Collect doesn't necessarily report this)
  • disable or hide UI functionality if user does not have required permissions for that functionality.
  • update to AppEngine SDK 1.5.1
  • update installer screens; https automatic on GAE deployment.
  • installer now deletes its temp directory
  • update security model (extensive changes)
  • hide password when deploying to Appspot ( issue 222 )
  • fix launching of scripts and display of README on windows ( issue 223 )
  • show the number of media files associated with a form ( issue 224 )
  • fix crash on MacOS when some fonts are present ( issue 236 )
  • fix lockouts when changing super-user ( issue 242 )
  • hide forms that are queued for deletion from the UI ( issue 255 )
  • remove the old UI (servlet) pages.
  • remove old applets
  • fix operations behind proxies ( issue 212 )
  • fix lingering UTF-8 character display issues ( issue 218 , 215)
  • fix new form and upload submissions links ( issue 243 ; 257) and IE page display and upload submissions bug.
  • fix PostgreSQL create-database scripting errors. ( issue 256 )
  • changes to improve client-side javascript operations, especially the refresh timer operations.
  • changes to poll for configuration changes so that instances running in multi-server farms all eventually synchronize their security config and forms list.
  • changes to get working (depressingly...) minimal unit tests for gae, mysql, postgresql under maven.

Aggregate v1.0 Beta rev.2

  • fixes for Tomcat operations (missing images, styles)
  • updates to tabs, stylesheets and layout of new UI
  • configuration wizard improvements ( Issue 222 , 223)
  • Postgresql install works ( issue 227 )
  • updated to AppEngine SDK 1.5.0.1
  • add Tomcat image preview bean
  • simplified permissions tab
  • change Google AppEngine decimal value storage to use double.

Limitations:

  • will only display 1000 records
  • will only export 1000 records to csv and kml files
  • publishing broke as of Oct 2011.

To work around the publishing failures and the 1000-record limitation, use the Briefcase applet to download the data to a local csv file. Access the Briefcase applet through the URL: 

https://your-application-id-here.appspot.com/Briefcase

Aggregate v1.0 Beta

Useful URLs during the install process:

Google AppEngine

Apache Tomcat 6

PostgreSQL 9.0

MySQL 5.1 Community Edition

Installer Special Instructions

You must have the Java runtime environment (JRE) installed. To download:

Installer Known Issues

  • Can only select install folder; can't create it.

Aggregate 1.0 Beta Known Issues

Aggregate 1.0 Beta Feature Requests

  • Need indication of whether a form has associated media files

Determining Aggregate 1.0 Version

n the 1.0 Production release, the Version appears on the Site Admin Tab, under the Preferences sub-tab.

Publishing to Google is broken prior to RC2 (due to a change in Google infrastructure over the summer).

Here's my cheat sheet:

  • Beta 1 and earlier: permissions tab looks nothing like the tab in the Production release.
  • Beta 2: permissions tab looks like the Production release; Forms tab does not show the number of media files associated with a form.
  • Beta 3: display the number of media files associated with a form; publishing to Google is broken.
  • Beta 4: there are "Form Management" and "Site Admin tabs; publishing to Google is broken.
  • RC1: csv exports apply the active Filter to the exported columns; publishing to Google is broken.
  • RC2: submissions tab is paginated (shows 500 records per page by default); publishing to Google works.
  • RC3: kml exports apply the active Filter to the exported data. exports all data -- no longer truncates to the first 500 records.
  • RC4: submissions for forms with non-repeating groups are now properly reconstructed into Briefcase. date fields are broken.
  • Production: Version info is shown on the Site Admin / Preferences sub-tab; date fields work with Briefcase.

► Sign in to add a comment
Powered by Google Project Hosting