• Design Decisions
• Basic API
• Permanent Objects
• Persistence Requirements
• Persisting Functions
• Persisting Userdata
• Writing code which is compatible with the save / load system

Design Decisions

To understand some of the design decisions in the persistance logic, it is useful to review the objectives of the persistance logic:

1. Allow the game to be saved to a file, and then later reloaded from said file
2. Work with as many major implementations of Lua as possible (5.1 and 5.2 from lua.org, LuaJIT 1 and 2, etc.).
3. Have a platform independent file format (endianness, size of data types, Lua implementation, etc.)
4. Have a file format which is somewhat forward and backward compatible between revisions

The implication of the second objective is that the persistance library must use only the Lua 5.1 public API. There were two obvious paths to go down for implementing persistance:

1. Write code for persisting every part of the game to binary, and then depersisting it again
2. Take a snapshot of the Lua universe, and write code for persisting every type of Lua value to binary, and then depersisting it again

The second option was chosen. The reason being that there are only a small number of different Lua data types, whereas there are a much larger number of game parts. There are also new game parts being implemented as time goes on, whereas Lua data types are fixed in number. With this decision made, LuaPluto was one option which seemed to fit the bill, however further inspection revealed that Pluto did not satisfy our objectives:

• Pluto uses the private Lua internals, limiting it to lua.org's Lua 5.1 and other very similar implementations of Lua. As a result, Pluto is incompatible with LuaJIT 2, and is probably incompatible with with lua.org's 5.2.
• Pluto persists functions to Lua bytecode, which kills platform independence (and introduces security concerns - malicious bytecode can be dangerous).
• Pluto's writing of binary integers seems to have no regard for different endianness and different sizes of the underlying types on different platforms.

Despite these problems, the implementation of Pluto is nice, and so there are similarities between Pluto the bespoke persistance library implemented for CorsixTH.

Basic API

persist.dump(value, permanent_objects)

Takes a single Lua value, and a table of object => key relations, and returns a string representing value.

persist.load(data, permanent_objects)

Takes a string previously created by persist.dump, and a table of key => object relations, and returns the value originally passed to persist.dump.

For simple values (numbers, strings, etc.), persist.dump will just prepend a minimal header to the value. For complex values (tables, functions, userdata, etc.), persist.dump will persist the value, as well as (recursively) everything that the value references: keys and values and metatables for tables, upvalues and environments for functions, metatables and environments for userdata, etc.

persist.dofile(filename)

Equivalent to the standard dofile function, with the added effect of interpreting --[[persistable:name]] decorators on functions. See the section on persisting functions for further information.

Permanent Objects

In following the massive web of references from the original object, persist.dump will often encounter objects which should not, or cannot, be persisted. The solution to this is the permanent objects table. When persist.dump stumbles upon an object which is present in the permanent objects table, it will persist the key associated with the object instead of persisting the object, and will not follow any references from the object.

In order to be depersist the value, persist.load must be given an inverted permanent objects table, so that it can turn keys back into values. For every permanent key persisted by persist.dump, there must be some value associated with that key in the table passed to persist.load, though the value does not need to be the same as the one originally encountered by persist.dump. For example, if the application is closed between a savegame being taken and it being loaded, then all of the permanent objects will naturally be different, though they should represent the same things.

Note that while permanent objects are not recursively persisted, if some other persisted object contains a reference to something inside of the permanent object, then that part of the permanent object will get persisted.

Persistence Requirements

As can be seen in the above table, the are complex requirements on functions and on userdata, which are explained in more detail below.

Persisting Functions

C functions must be permanent objects - they cannot be persisted any other way. Lua functions can be persisted as permanent objects, or as explicitly named persistable functions.

If a Lua function is instantiated exactly once (which is usually true for top-level functions in source files), then it can be persisted as a permanent object. To do this, it must be given a name which is unique across all other permanent objects, and unchanging in future revisions. Global functions and class methods are automatically marked as permanent objects, using their existing names. Local functions can be marked as permanent objects by defining them using syntax like the following:

local f; f = persistable"unique name"( function(x, y)
  return x^2 + y^2
end)

Where the original, non-persistable function was:

local function f(x, y)
  return x^2 + y^2
end

If a Lua function is instantiated a variable number of times (which is usually true for anonymous functions and local functions defined within other functions), then it can be persisted, as long as it conforms to certain rules. Firstly, the file containging the function must have been loaded by persist.dofile (rather than loadfile, require, etc.) - unless you are doing some major restructuring, then this point will not need to be considered. Secondly, the function must be given a name which is unique across all other non-permanent persistable functions. This name must be given as a comment immediately before the function keyword, in the format --[[persistable:unique name]]. As an implementation detail, there can only be one function with a persistable decorator per source file line. Lastly, the function cannot have any strong upvalues.

Note: The term closure means an instance of a function, along with a value for each upvalue of that function (and also along with an environment table).

An upvalue (a local variable used within a function which is not defined in that function) is strong if it is written to by at least one closure, and is read from by any other closure(s). Otherwise, an upvalue is weak (that is, an upvalue is weak if it is only in one closure, or it is in multiple closures but not written to). The important point is that weak upvalues can be separated with no observable differences in behaviour (if Lua 5.2 was set as a minimum requirement, then upvalues could be rejoined, allowing strong upvalues to be persisted too).

In the following example, value is a weak upvalue, as it is read-from and written-to, but only by one closure:

function make_counter(starting_value)
  local value = starting_value
  return --[[persistable:counter_function]] function()
    value = value + 1
    return value
  end
end

In the following example, self is a weak upvalue, as it is only read-from, despite being in multiple closures (the fields of self are written to, but the upvalue itself is not written to):

local self = {x = 0, y = 0}
local --[[persistable:mover_x]] function moveX(x)
  self.x = self.x + x
end
local --[[persistable:mover_y]] function moveY(y)
  self.y = self.y + y
end

In the following example, value is a strong upvalue, as it is read-from by one closure, and written-to by a different closure:

local value
local function set(...)
  value = ...
end
local function get()
  return value
end

When a non-permanent function is persisted, all of it's upvalues are persisted. One issue to watch out for is when the persisted function can call other local functions (excluding those defined within itself), as local functions which it might call are recorded as upvalues, and hence get persisted, and thus require marking as permanent or persistable.

Persisting Userdata

Note: This section can be ignored if only writing Lua code, as new types of userdata can be made by new C code.

Userdata which are zero-size (like those created by the standard, though undocumented, newproxy function) can persisted, provided that their metatable (if present) does not contain a reference back to the userdata, and that their __depersist_size metafield (if present) is set to 0.

Other userdata must have a metatable in order to be persisted, and as previously, this metatable cannot contain a reference back to the userdata. The metatable must contain at least the following fields:

• __depersist_size - an integer to be passed to lua_newuserdata in order to re-create the userdata during depersistance. Given that the size of a userdata is often platform-dependant, userdata metatables are usually permanent objects (which also avoids references from the metatable back to the userdata). The presence of this field is what causes the requirement of the metatable not referencing the userdata, as that would create a circular reference before the userdata could be recreated.
• __persist - a C function which takes two arguments: the userdata being persisted, and a LuaPersistWriter userdata. If the userdata needs to persist extra data beyond it's metatable and environment table, then this function must swap the position of its arguments, cast the LuaPersistWriter userdata to a pointer, and then call methods on it. For example:

static int l_layers_persist(lua_State *L)
{
    THLayers_t* pSelf = luaT_testuserdata<THLayers_t>(L);
    lua_settop(L, 2);
    lua_insert(L, 1);
    LuaPersistWriter* pWriter = (LuaPersistWriter*)lua_touserdata(L, 1);
    pWriter->writeByteStream(pSelf->aiLayerContents, 13);
    return 0;
}

• __depersist - a C function which takes two arguments: the userdata being depersisted, and a LuaPersistReader userdata. If this function returns true, then it will be called again with just a single argument (the userdata being depersisted) after all other non-userdata values have been fully depersisted, which is useful if part of the depersistance process depends on other values which might only be partially depersisted at the time of the first call. This function should, in most cases, call a constructor on the userdata, as the memory contents of the userdata is undefined otherwise. If __persist wrote anything, then this function must swap the position of its arguments, cast the LuaPersistReader userdata to a pointer, and then call methods on it, being sure that in comparision to __persist, it calls the equivalent methods and in the order order. For example:

static int l_layers_depersist(lua_State *L)
{
    THLayers_t* pSelf = luaT_testuserdata<THLayers_t>(L);
    lua_settop(L, 2);
    lua_insert(L, 1);
    LuaPersistReader* pReader = (LuaPersistReader*)lua_touserdata(L, 1);
    new (pSelf) THLayers_t; // Call default constructor
    if(!pReader->readByteStream(pSelf->aiLayerContents, 13))
            return 0; // If a read fails, do not attempt any further reads
    return 0;
}

The metatable may also contain:

• __pre_depersist - a C function which takes the userdata being depersisted as an argument. This function is called before __depersist, and more importantly, also before the userdata's environment table is depersisted. Most of the time this is not needed, but it can be very useful in cases where the "reference graph" has a cycle containing multiple userdata and the order of depersistance of said userdata is important. When such cycles exist, one of the userdata in the cycle must be depersisted first, and if that first userdata references a second userdata via its environment table, and then the second userdata calls LuaPersistReader::readStackObject() to get the first userdata, then the second userdata will be seeing the first in an uninitialised state, unless the first used __pre_depersist to initialise itself before its environment table was depersisted.

Writing code which is compatible with the save / load system

The most important thing is to ensure that any functions which might get referenced by a game object are persistable, as detailed in the section on persisting functions.

If compatibility of savegames is desired between different versions of the source code (i.e. taking a savegame, then moving to a newer revision of the source, then loading said savegame), then there are other things to consider:

• Remember that any new fields on a table (i.e. ones which weren't used when the save was taken, but acquired a use in the new code) may be nil after loading a savegame from a previous version. This must be gracefully handled at all places where the field is used.

For example, if the Hospital class didn't have a reputation field, and a save was taken, then in some newer version of the code, the reputation field got initialised in the Hospital constructor and used throughout the code, and then said save was loaded, then the reputation field would be nil in all Hospital instances, despite there being no way of that happening normally.

• If new upvalues are added to a non-permanent persistable function (for example, the new code uses a local variable from a parent function which the old code didn't), then when an older savegame is loaded, accesses to these new upvalues will act like accesses to globals with the same names. Hence the introduction of new upvalues should be avoided if savegame compatibility is desired. Alternatively, if new upvalues need to be introduced, and savegame compatibility needs to be maintained, then a copy of the original persistable function can be made somewhere, and the new code given a new persistable name.

• Non-permanent persistable functions should not be deleted - move them to the end of the file or to a new file specially for deprecated persistable functions.

If a non-permanent persistable function disappears from the source code, then any savegames which include an instance of that function will fail to load. Similarly, if a permanent function disappears or is renamed (included those which are implicity named by being global or class methods), then any savegames which include a reference to that function will fail to load, though references to implicitly named permanent functions are rare.

For major changes, the cost of maintaining compatibility with savegames of previous versions will be too large. For minor changes, trying to maintain compatibility is usually a worthwhile goal.

Another thing to bear in mind is that the memory addresses of Lua objects (i.e. the result of calling tostring on functions and tables) should not be used for anything important, as these addresses will change across a save/load.