XML Database for Unity (Part 1)

Unity manages its data using a data structure called a scene graph.  Anyone who has worked a bit in Unity (or probably any other 3D DCC tool) understands what a scene graph is: there’s a Scene which contains any number of GameObjects in a spatial hierarchy.  These GameObjects have a collection of Components which in turn reference Assets or other GameObjects.  This setup is very intuitive and convenient, particularly if your game is broken into a number of levels.  But what about games with data that doesn’t live in a scene graph, or which builds its levels at runtime?

Many games don’t have the concept of a “level,” at least at design-time.  Strategy games, Roguelikes and Minecraftlikes (I went there) all build levels at runtime out of pieces which don’t exist in a scene graph.  Even games that do have levels in many cases still have to manage lots of data which doesn’t live in a level (UI icons, character customization pieces, server-side data representations, etc).

Unity provides the Resources and Asset Bundles APIs for loading assets that don’t live in a scene.  These are great APIs (albeit a bit boilerplate), but they are only half of the equation.  How do we author and organize this data?  How does the game’s logic access and filter it?  These are questions I will address with an XML Database.

I have built two sample Unity projects to illustrate this problem.  I recommend you download the examples from GitHub here.  In this article, we will be looking at the SimpleExamples project.  The AssetBundleExamples project will be the subject of Part 2.

The example projects are the beginnings of a strategy game (go figure) with a procedurally generated map.  The map has four different types of terrain Tiles: water, grass, dirt and stone, and any number of Features which can be placed on the tiles (trees, rocks, gems, etc).  When you run the game, it looks like this:

Procedural Map
Our Procedural map example. Special thanks to Daniel Cook for the awesome art.

There are two examples in the SimpleExamples project.  The first, in the Example1 directory, is a “naive” implementation of the procedural map.  The second example, in the Example2 directory, illustrates how to create the same map using an XML Database.

Example1

Each example has a script which builds the map, and inherit from the MapBuilder class.  This base class is responsible for generating a height field using a simple Perlin noise algorithm.  After the height field is built, the InitMapObjects() virtual method is called which is responsible for instantiating Tiles and Features based on the height field.  We’ll override this method in each example.  After the map objects are initialized, the MapBuilder places the shadows.  We’ll make use of this in our subclasses.  Let’s look at the business of MapBuilder class, the InitHeights() method.  All the rest can be ignored for now.

This method is very simple.  We initialize the HeightField int array with our Width and Height, then iterate over each Tile position.  At each position, a “noise space position” is computed by normalizing the Tile position and multiplying it by the NoiseFrequency parameter.  A noise value is then sampled using Unity’s Perlin noise API.  The sample is between 0.0 and 1.0, and needs to be converted to an integer height value before it’s stored in the array.  This is accomplished by multiplying the noise value by 3.0 (the number of Tile types – 1) and rounding to int.

Once InitHeights() is finished, InitMapObjects() is called.  In the base class, this method is abstract, so we’ll have to go to our subclass to see what it’s doing.  In Example1, the subclass is called MapBuilder1 (creative, right?).

In MapBuilder1, there are three public members which would be editable in the Unity inspector: FeatureChance, TileLoadStrings and FeatureLoadStrings.  FeatureChance is the chance (from 0 – 100) that a Feature will be placed on a Tile.  TileLoadStrings and FeatureLoadStrings are arrays of strings which are paths to prefabs in the Resources directory.  More on this later.

The InitMapObjects() method is similar to the base class’s InitHeights() method, but instead of generating height values, we loop over each position and instantiate a Tile prefab and sometimes a Feature prefab.  Resources.Load<T> is used to load the prefab given its path string.  That prefab is then passed into the InstantiateMapPrefab() method defined in the MapBuilder base class.  This method instantiates the prefab, then places it on the map with the correct sorting depth.

Let’s look at the inspector for MapBuilder1:

MapBuilder1 Insepctor
The inspector for MapBuilder1

We’ve added resource paths for the four Tiles and a path for each Feature in their respective arrays.

Organizing your assets as path strings in arrays on a GameObject is totally acceptable.  I’ve shipped a game like this.  In my experience, though, there are a few problems with this approach:

  • Prefabs and ScriptableObjects are very hard to merge.  Source control is extremely important when you’re working on a team.  Even though Unity can serialize its assets in a “merge friendly” JSON file, we all know that merging a Unity asset can be a giant pain in the ass, and should only be attempted by those who really understand the file format.  All Unity assets should set up for mutually exclusive checkout in your source control system – only one person should be able to work on an asset at a time.  If all of your data is stored in a prefab or ScriptableObject and you’re using mutually exclusive checkout, that asset becomes a bottleneck for your authoring pipeline.  If you’re not using mutually exclusive checkout (looking at you, Git) then it becomes a time bomb waiting to explode.
  • This solution is not scalable.  For our simple example, we have four Tiles and seven Features, but imagine if we had hundreds.  Can you imagine dealing with a list in the Unity inspector that is hundreds of elements long?  Neither can I.
  • The data relationships are hidden in an asset.  To see the relationship between the MapBuilder and the assets it consumes, you have to open the editor and look.  This is not ideal for engineers and makes the data hard to validate externally.
  • The data is not platform agnostic.  Obviously, since we’ve defined our Tiles and Features in a Unity GameObject, we can’t use that data externally (like on a server) without some extra work.

Ideally, we would like to be able to define Tiles and Features outside of Unity all together, in a massively scalable, merge-friendly and platform agnostic manner.  XML to the rescue!

Example2

Let’s take a look at Example2.  This example externalizes the ideas of Tile and Feature into XML and loads that XML into a database at runtime.  The database is queried for appropriate entries by the MapBuilder2 script when the map is constructed.

There are a few new objects in this example.  In the Example2 scene, a new object has been added called GameController with a script of the same name.  This object is responsible for initializing the database when the game starts.  A new file called Infos.cs has been added.  This file defines two new classes: TileInfo and FeatureInfo, each inheriting from the DatabaseEntry class.

The TileInfo and FeatureInfo classes are simple and are marked up with XML attributes.  These attributes tell the C# runtime how to deserialize these objects from an XML document.

In the StreamingAssets/XML/ directory, there is a file called MapData.xml.  This is the place where all of the TileInfos and FeatureInfos used by the MapBuilder are defined.

The XML version of the info and the C# version of the info match – anything marked with the [XmlElement] attribute in the C# info class becomes an XML element in the XML file.  Each info entry also as an ID attribute.  This is used to reference specific entries in the database.  More on IDs later.

In our previous example, a Feature was instantiated as per the FeatureChance member of the MapBuilder1 class.  If FeatureChance was met, a random Feature was chosen from the FeatureLoadStrings list.  We have made two significant improvements in Example2.  First, each TileInfo includes its own FeatureChance, so now each Tile can have a different probability of dropping a Feature.  Additionally, each TileInfo keeps its own list of Features in an array called PossibleFeatures.  This way, a Tile can define which set of Features can appear on it.  This is accomplished by using the DatabaseEntryRef<T> type, which allows DatabaseEntries to reference each other.  More on this later.

Now that we’ve seen how objects are defined in the XML database, both on the C# and XML side, let’s look at how the gameplay logic queries for the data.  Here’s what the InitMapObjects() method looks like in MapBuilder2:

It’s very similar to MapBuilder1 but has some important differences.  Instead of pulling path strings from arrays as in Example1, we’re querying the database for entries that meet certain criteria.  These queries result in an array of entries that match, which we can select randomly from.  This line says it all:

We’re saying “give me all the TileInfos in the database that match the height at this position.”  The .Where() clause, where we’re doing the filtering, is part of the System.Linq namespace.  Linq is super awesome, and a full explanation is beyond the scope of this article, but in general you use Linq to make SQL-style queries into C# data structures.  The .Where() clause takes a “predicate” – a piece of code that returns true if the data in question should be included in the result.  In this example, we’ve used a lambda function as the predicate, but could have easily used a method instead.

By using TileInfo and FeatureInfo database entries instead of raw strings, we have a lot more data available to use than we previously did.  When we drop a Feature, we’re actually selecting randomly from the TileInfo’s array of FeatureInfos.  You can easily see how new data and relationships could be added.  This is very powerful, indeed.

Database Implementation

So how does this all work?  We’ve seen a Database singleton being used to get DatabaseEntry subclasses.  Let’s look at its implementation.

The Database implementation lives in three files found in Common/Script/Database/ – Database.cs, DatabaseFactory.cs and ID.cs.  Let’s look at Database.cs first.  Here’s the complete code listing:

This file defines three classes: DatabaseEntry, DatabaseEntryRef<T> and Database.  The Database stores collections of DatabaseEntry subclasses.  Each DatabaseEntry has an ID, which the user provides in the XML definition. When an entry is requested, two Dictionary lookups are made: first, the collection of DatabaseEntries is looked up based on the entry type.  Then, the DatabaseEntry itself is looked up using the provided ID (see the GetEntry<> and GetEntries<> methods).

The Database class also provides the ReadFiles() method for recursively loading XML files from a given root directory.  Once all files are loaded, DatabaseEntryRefs are resolved and an overridable OnPostLoad() method is called on each DatabaseEntry, in case a DatabaseEntry subclass needs to do some fancy initialization after it’s loaded.  This part of the Database implementation could easily be extended to pull XML files from other locations (like a server).

The DatabaseEntryRef<> class provides a way to make references between DatabaseEntries.  This was used by TileInfo example2 to reference FeatureInfos.  These references are read in as ID strings, converted to IDs and resolved after loading is complete.  DatabaseEntryRefs can be implicitly converted to the DatabaseEntry they point to or its Entry property can be used to access the entry.

The next class we’ll look at is DatabaseFactory.cs.  This class is responsible for building a list of factory functions which take XML as a parameter, and instantiate and initialize the correct DatabaseEntry subclass.

Registration of factory functions happens automatically in the CreateTypeMap() static method, which is called during static initialization.  This method iterates through each object type defined in the currently loaded assembly, and builds a factory function for all objects that are DatabaseEntry subclasses.  When the user calls Database’s ReadFiles() method, the DatabaseFactory’s Create method is called to do the actual instantiation and deserialization.

Finally, let’s take a look at ID.cs.  Every DatabaseEntry has an ID.  It is the way entries are stored in Database’s key-value collections, and the means by which the user looks up specific entries.

ID is basically a string but with one important difference.  IDs must be constructed with the static CreateID() (the constructor which takes a string is private, while the default constructor returns an “empty” ID).  When you create an ID via the CreateID method, it first checks if an ID with the provided string has been created already (all created IDs are stored in the static idTable).  If one already exists, that one is returned instead of a new one being created.  This has two benefits.  First, we prevent an explosion of strings and save some memory and GC thrash.  Second, and most importantly, comparing IDs is not comparing strings, but comparing memory addresses.  If you call CreateID() twice with the same string, the same object will be returned both times.  ID’s Equals() method has been overridden to perform a “reference equals” instead of a “value equals.”  This makes looking up DatabaseEntries by their ID very fast.

Why This is Awesome

Externalizing game data like this is extremely powerful.  Designers can work independently of artists, and in many situations don’t even need seats of Unity to modify the game!  This applies the same for modders.  Putting your game’s data in XML gives you modding support for free.

The XML doesn’t have to be stored locally as it is in the example I’ve presented.  It could just as easily come from a server.  This makes updating your game easy, especially if you aren’t adding new art assets.  Simply change the XML on the server, and you’re done!  You could make balance changes to your game without having to resubmit your app.

The XML representation of your game data can be shared between server and client, eliminating the need for duplicate versions.  The Database implementation is not dependant on any Unity features, so you could even use the same Database code on both the client and server if both are written in C#.

XML is also easily validated either in Unity at design-time or runtime, or by external validators.  We’ll take a look at validating XML content at design-time in Part 2 of this article.  C# also has robust support for XML schemas.  In fact, using a tool called xsd which comes with Mono, you can generate XML schemas from C# classes or C# classes from XML schemas.  The database implementation in these examples doesn’t do any schema validation for simplicity’s sake, but it would be easy to add.

Limitations

The database implementation presented here uses C# containers internally to store data, not an actual database backend like SQL.  This is fine if all of your game data can be resident in memory.  If you have so much data in the database that you run out of memory, a better data storage solution would need to be implemented.

What’s Next?

In Part 2 of this article, I will use this XML database system to create a build pipeline for Asset Bundles and an asynchronous asset loading API.  Stay tuned!