What is AppRocket?

AppRocket replicates AppEngine datastore to a MySQL database and vice versa. There are several things you can do with it, such as:

• Backup and restore AppEngine datastore
• Manipulate data using a regular SQL
• Process data outside the tight AppEngine request limits

AppRocket was orignally developed as an utility for the upcoming music discovery service mixize.com to address various functional challenges common to most AppEngine applications:

• It's hard to move data around
• It's hard to analyze, aggregate and manipulate data
• It's hard to refactor data
• AppEngine doesn't directly support asynchronous / batch processing
• AppEngine request constrains are very limiting

Now we are glad to share our efforts.

How does it work?

AppRocket performs a replication in two cycles:

• Receive - new and updated AppEngine entity instances are replicated to MySql database.
• Send - new and updated MySQL records are replicated to AppEngine datastore.

The replication is performed by two components:

• AppEngine application extension rocket.py - running inside AppEngine application,
• Replication daemon station.py - running on your server (such as VPS or physical machine).

Each AppEngine entity instance is replicated to it's own table in MySQL and vice versa. Each AppEngine list property is replicated to and from a separate table in MySQL. Each list value maps to a single record in this table and each list record has a key referencing to it's owner entity.

During receive cycle, MySQL database table structures are initialized and updated automatically to match the entities replicated from AppEngine.

Entity instances in AppEngine are looked up based on a timestamp field, that should be defined as

timestamp = db.DateTimeProperty(auto_now=True)

This property is required for every entity that needs to be replicated. Similiarly each MySQL table that needs to be replicated to AppEngine must have a column with type TIMESTAMP.

Getting started

Getting AppRocket running for you application is fairly simple:

1. Download latest AppRocket release
2. Add "rocket" package from it to your AppEngine application's root
3. Add the following configuration to your app.yaml file, right after the line handlers:

- url: /rocket/.*
  script: rocket/rocket.py

4. Ensure entities that you want to replicate have a timestamp property as described above.
• If you already have a timestamp property, but it is named differently, you can map it using TIMESTAMP_FIELD entity configuration option.
• If you just added a new timestamp property, AppRocket will only see entities which has been created or updated after this property is added. If you want to "unhide" older entities, you need to run a script that walks through all the entities and simply invokes put() on them.
5. Edit rocket/config.py file:
• Edit ROCKET_URL (should not be neccessary if you are testing it locally)
• Edit MySQL database connection parameters
• Define each entity you want to replicate under ROCKET_ENTITIES section, providing additional configuration options for each entity if neccessary. Different options are described in the config.py file. Here's an example entity configuration:

   ROCKET_ENTITIES = {
   "Address": {TIMESTAP_FIELD: 'modified_datetime'},
   "Order": {},
   }

6. Run station.py script in your application's rocket subdirectory
7. If all is well you should now have your entities replicated to a MySQL database

Have questions or suggestions?

Please use AppRocket discussion group.