Search Unity

In-depth Cache Server

October 26, 2012 in Engine & platform | 9 min. read
Placeholder image Unity 2
Placeholder image Unity 2
Share

Is this article helpful for you?

Thank you for your feedback!

In Unity 3.5 we shipped the first version of the Cache Server, and with 4.0 we're making it even better (the way it works is the same). I want to go a bit in-depth to explain what it actually does, because it’s pretty awesome and can save you lot of time.

It makes importing projects insanely fast.

If you pull changes from version control and everyone in your team uses the cache server, everything will already be cached and your imports will be almost instantaneous.

If you switch between platforms, you no longer need to wait for a long time while Unity re-compresses textures. Switching platforms becomes almost instant.

It works with any version control system: SVN, Perforce, Asset Server, Git, Hg etc.; or if you don't use version control. The only requirement is that you enable .meta files in the project.

The Cache Server helps you make this go away:

Version control in Unity

In Unity all source assets are stored in the "Assets" and "ProjectSettings" folders. Since these two folders contains all source assets, they must be in version control and they are the only folders that need to be in version control. The "Library" folder should not be under version control. In order to use Version Control in Unity you also need to enable .meta files. This can be done in the Edit -> Project Settings -> Editor menu.

The Cache Server is not a replacement for version control. In fact it does not touch source assets at all! It is a way of caching the import step. Thus it works with any version control system out there, simply because it does not interfere with it in any way.

What does the import step do?

Unity has a completely automatic asset pipeline. Whenever a source asset like a .psd or .fbx file is modified, Unity automatically reimports it. During the import step Unity creates a file with the imported data. Various things are generated in this import step, for example a .psd file will import the texture and compress it for the target platform. A Texture2D class object will be generated and stored in the "Library/metadata" folder. Unity also generates a preview for the texture which is used when picking assets. The generated data is not under version control, since it can always be regenerated. Some asset types will reimport when switching platforms, for example textures require a reimport when switching to a platform with different compression scheme. Meshes on the other hand are independent of the target platform.

The import step sometimes does very complex operations. For example when importing a .psd file for iOS we generate PVR compressed texture automatically. Or we could generate secondary mesh UV data for lightmapping. In some cases it can take seconds for each asset. This time can add up and thus the cache server can give you massive import speed benefits.

In Unity each asset is identified by a GUID (a globally unique identifier). The GUID is generated when an asset is first imported and stored in the .meta file. We do this so we can move any asset without losing any references to it (all asset references are GUID-based, not folder or file name based).

What does the Cache Server do?

The cache server accelerates the import step. It does not touch source data. The cache server is independent of version control. It is even independent of your current project.

Instead of actually importing an asset Unity will re-use an imported asset another team member or yourself has previously imported and uploaded to the cache server. The Cache Server is completely automatic and happens as part of the import pipeline. How much benefit you get from the cache server depends on the type of assets. Textures and large model files usually take the most time to import thus the speed up will be greatest there. Anything from 2x-100x can be expected!

The first thing Unity does when it is about to import an asset is to generate an MD5 hash of all source data.

For a texture this consists of:

  • The source asset: "myTexture.psd" file.
  • The meta file: "myTexture.psd.meta" (stores all importer settings).
  • The internal version number of the texture importer.
  • A hash of all version numbers of all Asset Postprocessors.

Anything that can affect the resulting imported data will be hashed. The Cache Server is a very simple file server that looks up assets by the GUID & Hash of that asset. On the cache server you might have many cached files for the same texture. The cache server removes assets that have not been used for a period of time automatically. It uses a simple least recently used scheme for throwing away unused caches.

If you switch Unity from iOS to Standalone mode for example, the imported texture will get a different hash. When switching the platform Unity calculates the new hash of all textures and tries to download them from the cache server. If they are available they will be used, otherwise it will import the texture and upload the imported result to the cache server.

Once it is cached on the server it can be used next time someone switches to the platform or imports that asset. Thus switching platforms becomes pretty much instant.

Cache Server performance

The Cache Server itself is very simple. It is essentially a streaming file server. If you setup a cache server for a big team, the important considerations are:

  • Lots of RAM so the cache server can keep all assets in disk cache and serve them instantly
  • Fast disk so if it's not in RAM cache it can be served quickly
  • Ensure the cache server is at least 3-4x larger than the Library/metadata folder in your project
  • The network connection to the machines using Unity must be fast.

Since it is purely a cache that accelerates the import and all the data can always be re-generated automatically by reimporting, it usually makes sense to ensure the cache server is local. Some people setup cache servers that are used remotely, this can make sense with a very fast internet connection. It is best to profile it and see what is fastest. But for the best performance I'd recommend  to have the cache server in the same location with a wired Ethernet connection. There is no problem with having multiple cache servers for one project. It is simply a cache, there are no conflicts if you have multiple servers.

Once the data is downloaded, the actual integration of the data usually takes less than 1 ms.

Essentially it caches the entire import step including preview generation, meaning other people using the same cache server can reuse the already imported asset without having to go through the full import process. In Unity the asset importers often perform complex tasks for example, when importing a .psd file in iOS we generate compressed PVRTC textures automatically for you so you get the best performance on the device. Or we have an option to generate secondary UV data for your models for lightmapping purposes automatically. These tasks can easily take several seconds and when you have a ton of assets it can quickly add up.

Cache Server on Windows

The Cache Server is built on top of node.js. The windows file system & file locking behaviour on windows make the cache server work not very efficiently on windows. I suggest that you install your cache server on a Linux or OS X server instead.

What not to do in the Cache Server?

The Cache Server does not handle asset dependencies. In fact, Unity's asset pipeline does not deal with the concept of asset dependencies either; it is built in a way to avoid dependencies. AssetPostprocessors are a common technique used to customize the asset importer to fit your needs; for example you might want to add MeshColliders to some GameObjects in an FBX file based on their name or tag.

However, it is also easy to use AssetPostprocessors to introduce dependencies. For example you might use data from a text file next to the asset to add additional components to the imported game objects. This is not supported in the Cache Server! If you want to use the cache server you you will have to remove dependency on other assets in the project folder. Since the Cache Server doesn't know anything about the dependency in your postprocessor, it will not know that anything has changed thus use an old cached version of the asset.

In practice there are plenty of ways how you can do asset postprocessing that works well with the Cache Server.
You can use:

  • The Path of the imported asset
  • Any import settings of the asset (including string AssetImporter.userData)
  • The source asset itself or any data generated from it passed to you in the asset postprocessor.

Modifying materials that already exist might cause trouble as well. When using the cache server Unity validates that the references to materials are maintained. But since no postprocessing calls will be invoked, the contents of the material can not be changed when a model is imported through the cache server. Thus you might get different results when importing with or without cache server. It is best to never modify materials that already exist on disk from an asset postprocessor.

The issue with modifying materials that already exist is that, if you download an fbx file through the cache server, then there is no import process running for it, this means that if you rely on resetting the generated materials to some generated defaults every single time the model importer runs, then this asset postprocessor will not be run, when importing a cached fbx file.

Where do I get it?

The Cache Server is part of the Team License: http://unity3d.com/unity/team/

You can download it here: http://unity3d.com/unity/team/cacheserver/

How to setup the Cache Server: http://docs.unity3d.com/Documentation/Manual/AssetCacheServer.html

Conclusion

The Cache Server gives you massive import time speed ups. If you work on a large project folder, using it is incredibly important. It makes your life better, seriously. Use it!

October 26, 2012 in Engine & platform | 9 min. read

Is this article helpful for you?

Thank you for your feedback!

Related Posts