If you haven't read the Concepts yet, please do so first.

• Create Your Entity Classes
• Registering Your Classes
• Basic Operations: Get, Put, Delete
• Querying
• Cursors
• Asynchronous Calls
• Asynchronous Queries
• Asynchronous get()/put()/delete()
• Considerations of Asynchronous Requests
• Optimizing Storage
• @Indexed and @Unindexed
• Partial Indexes
• @NotSaved
• Polymorphism
• Implementation Considerations
• Relationships
• Parent Relationship
• Single-Value Relationship
• Multi-Value Relationship
• Transactions
• Lifecycle Callbacks
• Migrating Schemas
• Adding Or Removing Fields
• Renaming A Field
• Transforming Data
• Changing Enums
• Moving Fields
• @Embedded
• Embedded Classes
• Embedded Collections and Arrays
• Indexing Embedded Classes
• Querying By Embedded Fields
• Entity Representation
• Schema Migration
• @Serialized
• Caching
• Session Cache
• Global Cache
• Example

This will explain how to use Objectify to get, put, delete, and query data. You may find it helpful to open the Objectify javadocs while reading. These examples omit getter and setter methods for clarity.

Create Your Entity Classes

The first step is to define your entity class(es). Here is an example of a Car:

public class Car
{
    @Id Long id;
    String vin;
    int color;
    @Transient String doNotPersist;

    private Car() {}
    
    public Car(String vin, int color)
    {
        this.vin = vin;
        this.color = color;
    }
}

Things to note:

• Objectify persists fields and fields only. It does not arbitrarily map fields to the datastore; if you want to change the way a field is stored... rename the field.

• Objectify will not persist static fields, final fields, or fields annotated with javax.persistence.Transient (it will persist fields with the transient keyword).

• One field must be annotated with javax.persistence.Id. It can be of type Long, long, or String. If you use Long and put() an object with a null id, a value will be generated for you. If you use String or the primitive long type, values will never be autogenerated.

• You can persist any of the core value types, Collections (ie Lists and Sets) of the core value types, or arrays of the core value types. You can also persist properties of type Key.

• There must be a no-arg constructor (or no constructors - Java creates a default no-arg constructor). The no-arg constructor can have any protection level (private, public, etc).

• If you are converting entities from a JDO project, note that Objectify uses JPA annotations (javax.persistence) and not JDO annotations (javax.jdo.annotations). Of course, Objectify adds several annotations of its own.

• String fields which store more than 500 characters (the GAE limit) are automatically converted to Text internally. Text fields, like Blob fields, are never indexed.

• byte[] fields are automatically converted to Blob internally. However, Byte[] is persisted "normally" as an array of (potentially indexed) Byte objects. Note that GAE internally stores all integral values as a 64-bit long.

More information can be found in the AnnotationReference.

Registering Your Classes

Before you perform any datastore operations, you must register all your entity classes with the ObjectifyService.

ObjectifyService.register(Car.class);
ObjectifyService.register(Motorcycle.class);

Objectify does not scan your classpath for @Entity classes. There are good reasons for and against this - see the discussion in BestPractices. If you are using Spring, see the objectify-appengine-spring project.

Basic Operations: Get, Put, Delete

You can obtain an Objectify interface from the ObjectifyService:

Objectify ofy = ObjectifyService.begin();

// Simple create
Car porsche = new Car("2FAST", "red");
ofy.put(porsche);
assert porsche.id != null;    // id was autogenerated

// Get it back
Car fetched1 = ofy.get(new Key<Car>(Car.class, porsche.id));
Car fetched2 = ofy.get(Car.class, porsche.id);    // equivalent, more convenient
assert areEqual(porsche, fetched1, fetched2);

// Change some data and write it
porsche.color = "blue";
ofy.put(porsche);

// Delete it
ofy.delete(porsche);

The interface supports batch operations:

Objectify ofy = ObjectifyService.begin();

// Create
Car porsche = new Car("2FAST", "red");
Car unimog = new Car("2SLOW", "green");
Car tesla = new Car("2NEW", "blue");
ofy.put(tesla, unimog, porsche);    //varargs; Car[] and Iterable<Car> also work

// Get the data back
List<Key<Car>> carKeys = new ArrayList<Key<Car>>();
carKeys.add(new Key<Car>(Car.class, porsche.id));
carKeys.add(new Key<Car>(Car.class, unimog.id));
carKeys.add(new Key<Car>(Car.class, tesla.id)));
Map<Key<Car>, Car> fetched1 = ofy.get(carKeys);

// More convenient shorthand, note the return type
Map<Long, Car> fetched2 = ofy.get(Car.class, new Long[] { porsche.id, unimog.id, tesla.id });

// This works too
Map<Long, Car> fetched3 = ofy.get(Car.class, Arrays.asList(porsche.id, unimog.id, tesla.id));

// Batch operations need not be homogenous:
List<Key<? extends Vehicle>> vehKeys = new ArrayList<Key<? extends Vehicle>>();
vehKeys.add(new Key<Car>(Car.class, porsche.id));
vehKeys.add(new Key<Motorcycle>(Motorcycle.class, ktm.id));
Map<Key<Vehicle>, Vehicle> fetched4 = ofy.get(vehKeys);

// Delete the data
ofy.delete(fetched1.values());

// You can delete by key without loading the objects
ofy.delete(
    new Key<Car>(Car.class, porsche.id),
    new Key<Car>(Car.class, unimog.id),
    new Key<Car>(Car.class, tesla.id));

Querying

Here are some examples of using queries. Objectify's Query mimics the human-friendly Query class from GAE/Python rather than the machine-friendly GAE/Java version.

Objectify ofy = ObjectifyService.begin();

Car car = ofy.query(Car.class).filter("vin", "123456789").get();

// The Query itself is Iterable
Query<Car> q = ofy.query(Car.class).filter("vin >", "123456789");
for (Car car: q) {
    System.out.println(car.toString());
}

// You can query for just keys, which will return Key objects much more efficiently than fetching whole objects
Iterable<Key<Car>> allKeys = ofy.query(Car.class).fetchKeys();

// Useful for deleting items
ofy.delete(allKeys);

Note that queries are closely related to indexes. See the appengine documentation for indexes for detail about what you can and cannot filter by.

Cursors

Cursors let you take a "checkpoint" in a query result set, store the checkpoint elsewhere, and then resume from where you left off later. This is often used in combination with the Task Queue API to iterate through large datasets that cannot be processed in the 30s limit of a single request. The algorithm for this is roughly:

1. Create a query, using an existing cursor if you have one.
2. Iterate through the results, processing as you go.
3. If you near the 30s timeout:
1. Get the cursor
2. Create a new processing task with the cursor
3. Break out of the loop

Cursor Example

The Iterables provided by Objectify (including the Query object) are actually QueryResultIterable. This will produce a QueryResultIterator, which allows you to obtain a Cursor.

This is an example of a servlet that will iterate through all the Car entities:

public static final long LIMIT_MILLIS = 1000 * 25; // provide a little leeway

@Override
protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    long startTime = System.currentTimeMillis();

    Objectify ofy = ObjectifyService.begin();
    Query<Car> query = ofy.query(Car.class);

    String cursorStr = request.getParameter("cursor");
    if (cursorStr != null)
        query.startCursor(Cursor.fromWebSafeString(cursorStr));

    QueryResultIterator<Car> iterator = query.iterator();
    while (iterator.hasNext()) {
        Car car = iterator.next();

        ... // process car

        if (System.currentTimeMillis() - startTime > LIMIT_MILLIS) {
            Cursor cursor = iterator.getStartCursor();
            Queue queue = QueueFactory.getDefaultQueue();
            queue.add(url("/pathToThisServlet").param("cursor", cursor.toWebSafeString()));
            break;
        }
    }
}

Asynchronous Calls

The GAE's low-level datastore API supports parallel asynchronous operations. GAE's model of asynchrony does not follow Javascript's "pass in a callback function" model; rather, when you make an asynchronous call, you get back a reference to the pending operation. You can create multiple references which will execute in parallel, however, any request to fetch a concrete result will block until the result is available.

This is better explained by example.

Asynchronous Queries

All queries are now asynchronous by default. The "reference" to a query is the Iterator object. For example, these two queries are executed in parallel:

Iterator<Fruit> fruitIt = ofy.query(Fruit.class).filter("color", "red").iterator();
Iterator<Animal> animalIt = ofy.query(Animal.class).filter("hair", "short").iterator();

// both queries are executing in the backend

while (fruitIt.hasNext()) {    // hasNext() blocks until query results are available
    ... process fruits
}

Create multiple Iterators, then execute over the iterators.

Asynchronous get()/put()/delete()

NOTE: This requires Objectify v3.x

NOTE: If you use Objectify's global memcache with asynchronous operations, you MUST install the com.googlecode.objectify.cache.AsyncCacheFilter. If you do not, your cache will not properly synchronize with the datastore. This is a workaround for a limitation of the GAE SDK; please star this issue.

Queries require no special interface to parallelize requests because the Iterator interface acts as a convenience reference to a pending operation. However, get(), put(), and delete() return concrete results. The GAE low-level API provides a parallel set of methods that return results in a layer of indrection, the java.util.concurrent.Future<?> class. However, Future<?> is cumbersome to use because it wraps and rethrows all exceptions as checked exceptions.

Objectify provides a similar set of parallel methods, but they return Result<?> -- just like Future<?> but with sane exception handling behavior. Here are the salient parts of Objectify's API:

public interface Result<T>
{
    T get();
    Future<T> getFuture();
}

public interface Objectify
{
    ...
    public AsyncObjectify async();
}

public interface AsyncObjectify
{
    ...
    <T> Result<T> get(Key<? extends T> key);
    <T> Result<T> get(Class<? extends T> clazz, long id);
    ...
    <T> Result<Key<T>> put(T obj);
    ...
    Result<Void> delete(Object... keysOrEntities);
    ...
}

You get the picture. The AsyncObjectify interface has methods that parallel the synchronous Objectify methods, but return Result<?> instead. You can issue multiple parallel requests like this:

Objectify ofy = ObjectifyService.begin();
Result<Fruit> fruit = ofy.async().get(Fruit.class, "apple");
Result<Map<Long, Animal>> animals = ofy.async().get(Animal.class, listOfAnimalIds);
Result<Key<Fruit>> key = ofy.async().put(new Fruit("orange"));
Iterator<City> citiesIterator = ofy.query(City.class).filter("population >", 1000).iterator();

// All requests are executing in parallel

String color = fruit.get().getColor();  // calling Result<?>.get() blocks until request is complete

Considerations of Asynchronous Requests

Parallel requests must be used carefully:

• If you use Objectify's global memcache (the @Cached annotaiton), you must install the com.googlecode.objectify.cache.AsyncCacheFilter in your web application.
• You cannot have more than a fixed number of asynchronous requests going simultaneously. This number is documented in the Low-Level API documentation, currently 10. Additional requests will block until previous requests complete.
• All pending requests will complete before your HTTP request returns data to the caller. If you return from your HttpServlet.service() method while there are async requests pending, the SDK will block and complete these requests for you.
• This does not allow you to work around the 30s limit for requests (or 10m for task queue requests). Any async requests pending when a DeadlineExceededException happens will be aborted. The datastore may or may not reflect any writes.
• If you run up against DeadlineExceededException while using the global memcache, it is very likely that your cache will go out of sync with the datastore - even with the AsyncCacheFilter. Do not do this.
• The synchronous API is no more efficient than the asynchronous API. In fact, both Objectify's synchronous API and Google's low level synchronous API are implemented as calls to the respective async API followed by an immediate get().

Optimizing Storage

Indexes are necessary for queries, but they are very expensive to create and update. It costs, in api_cpu_ms, about 48ms to put() a single entity with no indexes. Each standard indexed field adds 17ms to this number. The indexes are written in parallel, so they do not add real-world time... but you'll see the real-world cost on your bill at the end of the week! Indexes also consume a significant amount of storage space - sometimes many times the amount of original data.

@Indexed and @Unindexed

By default, all entity fields except Text and Blob are indexed. You can control this behavior with @Indexed and @Unindexed annotations on fields or classes:

// By default, fields are indexed
public class Car
{
    @Id Long id;
    String vin;
    @Unindexed String color;
}

// This has exactly the same effect
@Unindexed
public class Car
{
    @Id Long id;
    @Indexed String vin;
    String color;
}

Partial Indexes

Often you only need to query on a particular subset of values for a field. If these represent a small percentage of your entities, why index all the rest? Some examples:

• You might have a boolean "admin" field and only ever need to query for a list of the (very few) admins.
• You might have a "status" field and never need to query for inactive values.
• Your queries might not include null values.

Objectify gives developers the ability to define arbitrary conditions for any field. You can create your own If classes or use one of the provided ones:

public class Person
{
    @Id Long id;
    String name;

    // The admin field is only indexed when it is true
    @Unindexed(IfFalse.class) boolean admin;

    // You can provide multiple conditions, any of which will satisfy
    @Unindexed({IfNull.class, IfEmptyString.class}) String title;
}

These If conditions work with both @Indexed and @Unindexed on fields. You cannot specify If conditions on the class-level annotations.

Check the javadocs for available classes. Here are some basics to start: IfNull.class, IfFalse.class, IfTrue.class, IfZero.class, IfEmptyString.class, IfDefault.class

IfDefault.class

IfDefault.class is special. It tests true when the field value is whatever the default value is when you construct an object of your class. For example:

public class Account
{
    @Id Long id;

    // Only indexed when status is something other than INACTIVE
    @Unindexed(IfDefault.class) StatusType status = StatusType.INACTIVE;
}

Note that you can initialize field values inline (as above) or in your no-arg constructor; either will work.

Custom Conditions

You can easily create your own custom conditions by extending ValueIf or PojoIf. ValueIf is a simple test of a field value. For example:

public static class IfGREEN extends ValueIf<Color>
{
    @Override
    public boolean matches(Color value)
    {
        return color == Color.GREEN;
    }
}

public class Car
{
    @Id Long id;
    @Unindexed(IfGREEN.class) Color color;
}

You can use PojoIf to examine other fields to determine whether or not to index! This example is inspired by the example in the Partial Index Wikipedia page, and will use a static inner class for convenience:

// We are modeling:  create index partial_salary on employee(age) where salary > 2100;
@Unindexed
public class Employee
{
    static class SalaryCheck extends PojoIf<Employee>
    {
        @Override
        public boolean matches(Employee pojo)
        {
            return pojo.salary > 2100;
        }
    }

    @Id Long id;
    @Index(SalaryCheck.class) int age;
    int salary;
}

You can examine the source code of the If classes to see how to construct your own. Most are one or two lines of code.

@NotSaved

If you would like to exclude a field value from being stored, you can use the @NotSaved annotation. The field will not be saved and will not occupy any space in the datastore. This works well in concert with IfDefault.class:

@Unindexed
public class Player
{
    @Id Long id;
    @Indexed String name;
    @NotSaved(IfDefault.class) RankType rank = RankType.PRIVATE;
    @NotSaved(IfDefault.class) int health = 100;
    @NotSaved(IfDefault.class) Date retired = null;
}

Note that @NotSaved values are not stored at all, so they aren't indexed and you can't query for them.

Polymorphism

NOTE: This requires Objectify v3.x

Objectify lets you define a polymorphic hierarchy of related entity classes, and then load and query them without knowing the specific subtype. Here are some examples:

@Entity
public class Animal {
    @Id Long id;
    String name;
}
	
@Subclass
public class Mammal extends Animal {
    boolean longHair;
}
	
@Subclass
public class Cat extends Mammal {
    boolean hypoallergenic;
}

Things to note:

• The root of your polymorphic hierarchy must be annotated with @Entity.
• All polymorphic subclasses must be annotated with @Subclass.
• You can skip @Subclass on intermediate classes which will never be materialized or queried for.
• You should register all classes in the hierarchy separately, but order is not important.
• Polymorphism applies only to entities, not to @Embedded classes.

In a polymorphic hierarchy, you can get() and query() without knowing the actual type:

Objectify ofy = ObjectifyService.begin();

Animal annie = new Animal();
annie.name = "Annie";
ofy.put(annie);

Mammal mam = new Mammal();
mam.name = "Mam";
m.longHair = true;
ofy.put(mam);

Cat nyan = new Cat();
nyan.name = "Nyan";
nyan.longHair = true;
nyan.hypoallergenic = true;
ofy.put(nyan);

// This will return the Cat
Animal fetched = ofy.get(Animal.class, nyan.id);

// This query will produce three objects, the Animal, Mammal, and Cat
Query<Animal> all = ofy.query(Animal.class);

// This query will produce the Mammal and Cat
Query<Mammal> mammals = ofy.query(Mammal.class);

Implementation Considerations

When you store a polymorphic entity subclass (but not an instance of the base type), your entity is stored with two additional, hidden synthetic properties:

• ^d holds a discriminator value for the concrete class type. This defaults to the class shortname but can be modified with the @Subclass(name="alternate") annotation.
• ^i holds an indexed list of all the discriminators relavant to a class; for example a Cat would have [["Mammal", "Cat]].

The indexed property is what allows polymorphic queries to work. It also means that you cannot simply change your hierarchy arbitrarily and expect queries to continue to work as expected - you may need to re-put() all affected entities to rewrite the indexed field.

There are two ways you can affect this:

1. You can leave some subclasses unindexed by specifying @Subclass(unindexed=true). You will not be able to query by these subclasses (although simple get()s work, and queries for indexed superclasses will return a properly instantiated instance of the subclass).
2. You can use @Subclass(alsoLoad="OldDiscriminator") to "reclaim" old discriminator values when changing class names. Note that this will not help with query indexes, which must be re-put().

Relationships

A relationship is simply a Key stored as a field in an entity. Objectify does not provide "managed" relationships in the way that JDO or JPA does; this is both a blessing and a curse. However, because Key is a generified class, it carries type information about what it points to.

There are fundamentally three different kinds of relationships in Objectify:

Parent Relationship

An entity can have a single Key field annotated with @Parent:

public class Person
{
    @Id Long id;
    String name;
}

public class Car
{
    @Id Long id;
    @Parent Key<Person> owner;
    String color;
}

Each Car entity is part of the parent owner's entity group and both can be accessed within a single transaction. When loading the child entity, the parent Key must be used to generate the child's key:

Objectify ofy = ObjectifyService.begin();

Key<Person> owner = new Key<Person>(Person.class, somePersonId);
Car someCar = ofy.get(new Key<Car>(owner, Car.class, someCarId));

Note that this is an inappropriate use of the @Parent entity; if a car were to be sold to a new owner, you would need to delete the Car and create a new one. It is often better to use Single Value Relationships even when there is a conceptual parent-child or owner-object relationship; in that case you could simply change the parent.

If you get() an entity, change the @Parent key field, and put() the entity, you will create a new entity. The old entity (with the old parent) will still exist. You cannot simply change the value of a @Parent key field. This is a fundamental aspect of the appengine datastore; @Parent values form part of an entity's identity.

Single-Value Relationship

In Objectify (and the underlying datastore), Keys are just properties like any other value. Whether it defines a one-to-one relationship or a many-to-one relationship is up to you. Furthermore, a Key field could refer to any type of entity class.

One To One

The simplest type of single-value relationship is one-to-one.

public class Person
{
    @Id String name;
    Key<Person> significantOther;
}

Objectify ofy = ObjectifyService.begin();
Person bob = ofy.get(Person, "bob");
Person bobswife = ofy.get(bob.significantOther);

Many To One

A Key field can represent a many-to-one relationship.

public class Employee
{
    @Id String name;
    Key<Employee> manager;
}

Objectify ofy = ObjectifyService.begin();
Employee bob = ofy.get(Employee.class, "bob");
Employee fred = ofy.get(bob.manager);

It looks identical to the one-to-one relationship because it is. The only difference is a conceptual one. What if you want to know all the employees managed by Fred? You use a query.

Objectify ofy = ObjectifyService.begin();

Iterable<Employee> subordinates = ofy.query(Employee.class).filter("manager", fred);

Multi-Value Relationship

The datastore can persist simple object types (Long, String, etc) and collections of simple object types. It can also persist collections (and arrays) of Keys. This creates an alternative approach for defining one-to-many (and many-to-many) relationships.

public class Employee
{
    @Id String name;
    Key<Employee>[] subordinates;
}

This is sometimes useful, but should be used with caution for two reasons:

1. Every time you get() and put() an object, it will fetch and store the entire list of subordinate keys. If you have large numbers of subordinates, this could become a performance problem.
2. Appengine limits you to 5,000 entries.
3. Because appengine creates an index entry for every value in the collection, you can suffer from Exploding Indexes.

Because appengine stores an index entry for each value in the collection, it is possible to issue queries like this:

Objectify ofy = ObjectifyService.begin();

// should contain Fred
Iterable<Employee> managers = ofy.query(Employee.class).filter("subordinates", bob);

The decision to use a Multi-Value Relationship will depend heavily upon the shape of your data and the queries you intend to perform.

Transactions

Working with transactions is almost the same as working with Objectify normally.

Objectify ofy = ObjectifyService.beginTransaction();  // instead of begin()
try
{
    ClubMembers cm = ofy.get(ClubMembers.class, "k123");
    cm.incrementByOne();
    ofy.put(cm);

    ofy.getTxn().commit();
}
finally
{
    if (ofy.getTxn().isActive())
        ofy.getTxn().rollback();
}

All data manipulation methods are the same as you would normally use.

Since entities in Objectify really are Plain Old Java Objects and transactions are tied to the Objectify object, it's easy to work with data inside and outside of transactions (or multiple transactions running in parallel!):

Objectify ofyNoTxn = ObjectifyService.begin();
Objectify ofyTxn = ObjectifyService.beginTransaction();
try
{
    Foo f = ofyTxn.get(Foo.class, "k123");
    Bar b = ofyNoTxn.get(f.barKey);

    if (b.wantsUp())
        f.increment();
    else
        f.decrement();

    ofyTxn.put(f);

    ofyTxn.getTxn().commit();
}
finally
{
    if (ofyTxn.getTxn().isActive())
        ofyTxn.getTxn().rollback();
}

You can interleave multiple transactions or nontransactional actions as long as you obey the the cardinal rule: Within a single transaction (defined by an Objectify object created with beginTransaction()), you may only read or write from a single entity group.

Yes, this means you can get() objects from a transactional Objectify and put() to a nontrasactional Objectify.

Lifecycle Callbacks

Objectify supports two of the JPA lifecycle callbacks: @PostLoad and @PrePersist. If you mark methods on your POJO entity class (or any superclasses) with these annotations, they will be called:

• @PostLoad methods are called after your data has been populated on your POJO class from the datastore.
• @PrePersist methods are called just before your data is written to the datastore from your POJO class.

You can have any number of these callback methods in your POJO entity class or its superclasses. They will be called in order of declaration, with superclass methods called first. Two parameter types are allowed:

• The instance of Objectify which is being used to load/save the entity.
• The datastore Entity which is associated with the Java POJO entity.

class MyEntityBase {
    String foo;
    String lowercaseFoo;
    @PrePersist void maintainCaseInsensitiveSearchField() { this.lowercaseFoo = foo.toLowerCase(); }
}

class MyEntity extends MyEntityBase {
    @Id Long id;

    @Transient Date loaded;
    @PostLoad void trackLoadedDate() { this.loaded = new Date(); }

    List<String> stuff = new ArrayList<String>();
    int stuffSize;   // indexed so we can query by list size
    @PrePersist void maintainStuffSize() { this.stuffSize = stuff.size(); }

    @PrePersist void doMore(Objectify ofy, Entity ent) { ... }
}

Caution: You can't update @Id or @Parent fields in a @PrePersist callback; by this time, the low-level Entity has already been constructed with a Key so it can be passed in to the callback as an optional parameter. You can, however, update any other fields and the new values will be persisted.

Migrating Schemas

It is a rare schema that remains unchanged through the life of an application. BigTable's schemaless nature is both a blessing and a curse - you can easily change schemas object-by-object on the fly, but you can't easily do it in bulk with an ALTER TABLE. Objectify provides some simple but powerful tools to help with common types of structure change.

The basic process of schema migration using Objectify looks like this:

1. Change your entity classes to reflect your desired schema.
2. Use Objectify's annotations to map data in the old schema onto the new schema.
3. Deploy your code, which now works with objects in the old schema and the new schema.
4. Let your natural get()/put() churn convert objects for as long as you care to wait.
5. Run a batch job to get() & put() any remaining entities.

Here are some common cases.

Adding Or Removing Fields

This is the easiest - just do it!

You can add any fields to your classes; if there is no data in the datastore associated with that field, it will be left at its default value when the class is initialized. This is worlds better than the exceptions you often get from JDO.

You can remove a field from your classes. The data in the datastore will be ignored when the entity is get(). When you next put() the entity, the entity will be saved without this field.

Renaming A Field

Let's say you have an entity that looks like this:

public class Person
{
    @Id Long id;
    String name;
}

You're doing some refactoring and you want to rename the field "name" to "fullName". You can!

public class Person
{
    @Id Long id;
    @AlsoLoad("name") String fullName;
}

When a Person is get()ed, the fullName field will be loaded either the value of fullName or name. If both fields exist, an IllegalStateException will be thrown. When put(), only fullName will be written.

Caveat: Queries do not know about the rename; if you filter by "fullName", you will only get entities that have been converted. You can still filter by "name" to get only the old ones.

Transforming Data

Now that you've migrated all of your data to the new Person format, let's say you now want to store separate first and last names instead of a single fullName field. Objectify can help:

public class Person
{
    @Id Long id;
    String firstName;
    String lastName;

    void importCruft(@AlsoLoad("fullName") String full)
    {
        String[] names = full.split(" ");
        this.firstName = names[0];
        this.lastName = names[1];
    }
}

You can specify @AlsoLoad on the parameter of any method that takes a single parameter. The parameter must be type-appropriate for what is in the datastore; you can pass Object and use reflection if you aren't sure. Process the data in whatever way you see fit. When the entity is put() again, it will only have firstName and lastName.

Caution: Objectify has no way of knowing that the importCruft() method has loaded the firstName and lastName fields. If both fullName and firstName/lastName exist in the datastore, the results are undefined.

Changing Enums

Changing enum values is just a special case of transforming data. Enums are actually stored as Strings (and actually, all fields can be converted to String automatically), so you can use an @AlsoLoad method to process the data.

Let's say you wanted to delete the AQUA color and replace it with GREEN:

public enum Color { RED, GREEN }    // AQUA has been removed from code but it still exists in the datastore

public class Car
{
    @Id Long id;
    Color color;

    void importColor(@AlsoLoad("color") String colorStr)
    {
        if ("AQUA".equals(colorStr))
            this.color = Color.GREEN;
        else
            this.color = Color.valueOf(colorStr);
    }
}

The @AlsoLoad method automatically overrides the loading of the Color field, but the Color field is what gets written on save. Note that you cannot have conflicting @AlsoLoad values on multiple methods.

Moving Fields

Changing the structure of your entities is by far the most challenging kind of schema migration; perhaps you want to combine two entities into one, or perhaps you want to move an @Embedded field into a separate entity. There are many possible scenarios that require many different approaches. Your essential tools are:

• @AlsoLoad, which lets you load from a variety of field names (or former field names), and lets you transform data in methods.
• @NotSaved, which lets you load data into fields without saving them again.
• @PostLoad, which lets you execute arbitrary code after all fields have been loaded.
• @PrePersist, which lets you execute arbitrary code before your entity gets written to the datastore.

Let's say you have some embedded address fields and you want to make them into a separate Address entity. You start with:

public class Person
{
    @Id Long id;
    String name;
    String street;
    String city;
}

You can take two general approaches, either of which can be appropriate depending on how you use the data. You can perform the transformation on save or on load. Here is how you do it on load:

public class Address
{
    @Id Long id;
    String street;
    String city;
}

public class Person
{
    @Id Long id;
    String name;

    @NotSaved String street;
    @NotSaved String city;

    Key<Address> address;

    @PostLoad void onLoad(Objectify ofy)
    {
        if (this.street != null || this.city != null)
        {
            this.address = ofy.put(new Address(this.street, this.city));
            ofy.put(this);
        }
    }
}

If changing the data on load is not right for your app, you can change it on save:

public class Address
{
    @Id Long id;
    String street;
    String city;
}

public class Person
{
    @Id Long id;
    String name;

    @NotSaved String street;
    @NotSaved String city;

    Key<Address> address;

    @PrePersist void onSave(Objectify ofy)
    {
        if (this.street != null || this.city != null)
        {
            this.address = ofy.put(new Address(this.street, this.city));
        }
    }
}

If you have an especially difficult transformation, post to the objectify-appengine google group. We're happy to help.

@Embedded

Objectify supports embedded classes and collections of embedded classes. This allows you to store structured data within a single POJO entity in a way that remains queryable. With a few limitations, this can be an excellent replacement for storing JSON data.

Embedded Classes

You can nest objects to any arbitrary level.

class LevelTwo {
    String bar;
}

class LevelOne {
    String foo;
    @Embedded LevelTwo two
}

class EntityWithEmbedded {
    @Id Long id;
    @Embedded LevelOne one;
}

Embedded Collections and Arrays

You can use @Embedded on collections or arrays:

class EntityWithEmbeddedCollection {
    @Id Long id;
    @Embedded List<LevelOne> ones = new ArrayList<LevelOne>();
}

Some things to keep in mind:

1. Do not use @Embedded to store collections or arrays of simple types. The datastore knows how to persist List<String>, Set<GeoPt>, etc without any special annotations.
2. You cannot recursively store @Embedded classes. That is, an @Embedded class cannot contain a field of its own type, either directly or indirectly.
3. An @Embedded array/collection cannot be nested inside of another @Embedded array/collection. It can, however, be nested inside any number of @Embedded classes.
4. Likewise, a native array/collection cannot be nested inside an @Embedded array/collection. The only way to put a collection inside a collection (ie, create a 2D structure) is to make part (or all) of the structure @Serialized (see below).
5. You should initialize collections. Null or empty collections are not written to the datastore and therefore get ignored during load. Furthermore, the concrete instance will be used as-is, allowing you to initialize collections with Comparators or other state.

Indexing Embedded Classes

As with normal entities, all fields within embedded classes are indexed by default. You can control this:

• Putting @Indexed or @Unindexed on a class (entity or embedded) will make all of its fields default to indexed or unindexed, respectively.
• Putting @Indexed or @Unindexed on a field will make it indexed or unindexed, respectively.
• @Indexed or @Unindexed status for nested classes and fields are generally inherited from containing fields and classes, except that:
• @Indexed or @Unindexed on a field overrides the default of the class containing the field.
• @Indexed or @Unindexed on a field of type @Embedded will override the default on the class inside the field (be it a single class or a collection).

@Indexed
class LevelTwo {
    @Indexed String gamma;
    String delta;
}

@Indexed
class LevelOne {
    String beta;
    @Unindexed @Embedded LevelTwo two;
}

@Unindexed
class EntityWithComplicatedIndexing {
    @Id Long id;
    @Embedded LevelOne one;
    String alpha;
}

If you persist one of these EntityWithComplicatedIndexing objects, you will find:

Note that one.two.delta is not indexed; the annotation on LevelOne.two overrides LevelTwo's class default.

Querying By Embedded Fields

For any indexed field, you can query like this:

Objectify ofy = ObjectifyService.begin();
ofy.query(EntityWithEmbedded.class).filter("one.two.bar =", "findthis");

Filtering works for embedded collections just as it does for normal collections:

Objectify ofy = ObjectifyService.begin();
ofy.query(EntityWithEmbeddedCollection.class).filter("ones.two.bar =", "findthis");

Entity Representation

You may wish to know how @Embedded fields are persisted so that you an access them through the Low-Level API. Here is an example:

class LevelTwo {
    String bar;
}

class LevelOne {
    String foo;
    @Embedded LevelTwo two
}

class EntityWithEmbedded {
    @Id Long id;
    @Embedded LevelOne one;
}

EntityWithEmbedded ent = new EntityWithEmbedded();
ent.one = new LevelOne();
ent.one.foo = "Foo Value";
ent.one.two = new LevelTwo();
ent.one.two.bar = "Bar Value";

Objectify ofy = ObjectifyService.begin();
ofy.put(ent);

This will produce an entity that contains:

You can see why query filters work the way they do.

For @Embedded collections and arrays, the storage mechanism is more complicated:

EntityWithEmbeddedCollection ent = new EntityWithEmbeddedCollection();
for (int i=1; i<=4; i++) {
    LevelOne one = new LevelOne();
    one.foo = "foo" + i;
    one.two = new LevelTwo();
    one.two.bar = "bar" + i;

    ent.ones.add(one);
}

Objectify ofy = ObjectifyService.begin();
ofy.put(ent);

This will produce an entity that contains:

This is what the entity would look like if the second and third values in the ones collection were null:

The synthetic ^null property only exists if the collection contains nulls. It is never indexed.

Schema Migration

The @AlsoLoad annotation can be used on any field, including @Embedded fields. For example, this class will safely read in instances previously saved with EntityWithEmbeddedCollection:

class Together {
    @AlsoLoad("foo") String partOne;
    @AlsoLoad("two.bar") String partTwo;
}

class NextEntity {
    @Id Long id;
    @AlsoLoad("ones") @Embedded List<Together> stuff = new ArrayList<Together>();
}

@AlsoLoad methods work as well, however you cannot use @Embedded on method parameters.

@Serialized

An alternative to @Embedded is to use @Serialized, which will let you store nearly any Java object graph.

class EntityWithSerialized {
    @Id Long id;
    @Serialized Map<Object, Object> stuff;
}

There are some limitations:

• All objects stored in the graph must follow Java serialization rules, including implement java.io.Serializable.
• The total size of an entity cannot exceed 1 megabyte. If your serialized data exceeds this size, you will get an exception when you try to put() it.
• You will not be able to use the field or any child fields in queries.
• As per serializaton rules, transient (the java keyword, not the annotation) fields will not be stored.
• All Objectify annotations will be ignored within your serialized data structure. This means @Transient fields within your serialized structure will be stored!
• Java serialization data is opaque to the datastore viewer and other languages (ie GAE/Python). You will only be able to retrieve your data from Java.

However, there are significant benefits to storing data this way:

• You can store nearly any object graph - nested collections, circular object references, etc. If Java can serialize it, you can store it.
• Your field need not be statically typed. Declare Object if you want.
• Collections can be stored in their full state; for example, a SortedSet will remember its Comparator implementation.
• @Serialized collections can be nested inside @Embedded collections.

You are strongly advised to place serialVersionUID on all classes that you intend to store as @Serialized. Without this, any change to your classes will prevent stored objects from being deserialized on fetch. Example:

class SomeStuff implements Serializable {
    /** start with 1 for all classes */
    private static final long serialVersionUID = 1L;

    String foo;
    Object bar;
}

class EntityWithSerialized {
    @Id Long id;
    @Serialized SomeStuff stuff;
}

Caching

Objectify provides two different types of caches:

• A session cache which holds entity instances inside a specific Objectify instance.
• A global cache which holds entity data in the appengine memcache service.

You must explicitly decide to use these caches. If you do nothing, every get() will read through to the datastore.

Session Cache

The session cache associates your entity object instances with a specific Objectify instance. You must explicitly enable it by passing in ObjectifyOpts to the ObjectifyService.begin() method:

    ObjectifyOpts opts = new ObjectifyOpts().setSessionCache(true);
    Objectify ofy = ObjectifyService.begin(opts);

Note:

• The session cache holds your specific entity object instances. If you get() or query() for the same entity, you will receive the exact same Java entity object instance.
• The session cache is local to the Objectify instance. If you begin() a new instance, it will have a separate cache.
• A get() (batch or otherwise) operation for a cached entity will return the entity instance without a call to the datastore or even to the memcache (if the global cache is enabled). The operation is a simple hashmap lookup.
• A query() will return cached entity instances, however the (potentially expensive) call to the datastore will still be made.
• The session cache is not thread-safe. You should never share an Objectify instance between threads.
• The session cache appears to be very similar to a JPA, JDO, or Hibernate session cache with one exception - there is no dirty change detection. As per standard Objectify behavior, if you wish to change an entity in the datastore, you must explicitly put() your entity.

Global Cache

Objectify can cache your entity data globally in the appengine memcache service for improved read performance. This cache is shared by all running instances of your application.

The global cache is enabled by default, however you must still annotate your entity classes with @Cached to make them cacheable:

@Cached
public class MyEntity {
    @Id Long id;
    ...
}

That's it! Objectify will utilize the memcache service to reduce read load on the datastore.

What you should know about the global cache:

• The fields of your entity are cached, not your POJO class itself. Your entity objects will not be serialized (although any @Serialized fields will be).
• Only get(), put(), and delete() interact with the cache. query() is not cached.
• Writes will "write through" the cache to the datastore. Performance is only improved on read-heavy applications (which, fortunately, most are).
• Negative results are cached as well as positive results.
• Transactional reads bypass the cache. Only successful commits modify the cache.
• You can define an expiration time for each entity in the annotation: @Cached(expirationSeconds=600). By default entities will be cached until memory pressure (or an 'incident' in the datacenter) evicts them.
• You can disable the global cache for an Objectify instance by creating it with the appropriate ObjectifyOpts.
• The global cache can work in concert with the session cache.
• Remember: The session cache caches entity Java object instances, the global cache caches entity data.

Warning: Objectify's global cache support prior to v3.1 suffered from synchronization problems under contention. Do not use it for entities which require transactional integrity, and you are strongly advised to apply an expiration period to all cache values.

The cache in 3.1 has been rewritten from scratch to provide near-transactional consistency with the datastore. Only DeadlineExceededException should be able to produce synchronization problems.

For more commentary about the new v3.1 cache, see MemcacheStandalone.

Example

Andrew Glover wrote an excellent article for IBM developerWorks: Twitter Mining with Objectify-Appengine, part 1 and part 2.

Now, read the BestPractices.