LZ4 AssetBundle Still Cost A Lot Of Memory On Android Devices

I build the assetbundle with LZ4 format.,and load the assetbundle with LoadFromFileAsync(Not call unload).When i test on editor(Windows PC),it work fine,and i check the memory from profile,the serializedFile's size is small(About 0.5M),But when i test on android device, the serializedFile's size is very big(about 60M,And it became small when i unload the AssetBundles).Could anybody tell me why or what can i do with it?
PS:The doc said"Mem: no extra memory is used. Perf: reading from disk." in this situation.
Unity Version：5.3.1

 

Hi, thanks for your feedback!

Because of slow reads there might be excessive buffers created during realtime decompression on a preloading thread which were not reused (this was a bug which was fixed in 5.3.2) - I would recommend to upgrade to the latest stable version of Unity.

If problem still exists after upgrade, to help us narrow down a problem - do you have a data for uncompressed bundles? Do you see a memory jump immediately after LoadFromFileAsync finishes or after you loaded some objects from a bundle? Could you please file a bug with a project in such case?

Also as a side note - Android player is a bit special as it has extra decompression layer from apk file and uses a different caching policy for the SerializedFile itself. Thus there might be an additional overhead, but <=1MB overall.

 

Thank for your reply!
1." Thus there might be an additional overhead, but <=1MB overall." Do you mean every single loaded assetBundle will cost extra 1M(<=) or all of loaded assetBundles' cost in total ?
2.After update unity to 5.3.4, we still have the problem, The serializedFile's size of our game is still big(now about 70m) on android devices. Then we do some test, we found everytime after call LoadFromFile(no need to call loadAsset ), The serializedFile's size will grow 0.5M, no matter what size the origin assetbundle size is(Even load a very small assetBundle(20K), It will aslo cost 0.5M).So when we load a lot of assetbundle to memory the sizes will became very terrible(eg. After loaded 100 assetbundle,the serializedFile's size will bigger than 50M). Is this normal?

 

Right now for Android we use 2x256KB buffers per SerializedFile (besides decompression buffers), compared to 2x7KB on PC. So it is 0.5MB per bundle (assuming each normal bundle has a single serialized file) being kept alive until bundle is unloaded. This buffer is large for Android because of a custom apk compression layer, and we hold larger buffers to save apk decompression time.
We are looking into ways to make it consistent with other platforms.

 

Taking advantage of the subject, I am not sure about the performance of LZ4.

This is what the documentation says

Unity also supports LZ4 compression, which results in larger compressed file sizes, but does not require the entire bundle to be decompressed before use. LZ4 is a “chunk-based” algorithm, and therefore when objects are loaded from an LZ4-compressed bundle, only the corresponding chunks for that object are decompressed. This occurs on-the-fly, meaning there are no wait times for the entire bundle to be decompressed before use. The LZ4 Format was introduced in Unity 5.3 and was unavailable in prior versions.
​

When loading an already cached LZ4 asset bundle to the memory, is it going to use less memory compared to the other compressions?

Searching over the internet seems like LZ4 is about performance of decompression, and not memory optimization(load to memory only the parts we need).

Thanks in advance.

 

If a bundle was cached to disk LZ4 compression saves only a disk space.
Depending on disk read speed and compression ratio, it might either improve or degrade loading performance.
LZ4 bundle is loaded to memory by 128KB (by default) chunks when we actually read the data from it (scene loading or assets loading). We allow 5 loading streams with up to 4 128KB buffer chunks overall independent on how many LoadSceneAsync or LoadAssetsAsync requests you have in flight.
Uncompressed and lz4 compressed bundles when cached are being read on demand with a fixed predictable overhead (which is quite large for Android at the moment due to additional internal cache). Lzma compressed bundles during caching are being decompressed and recompressed (if Caching.compressionEnabled == true) to lz4 while being stored into the cache. So for all cached bundles, regardless of compression scheme there is a relatively fixed overhead (capped transient decompression buffer for lz4) during loading data from them.

When we load a bundle completely to memory (e.g. WebGL or LZMA compressed bundle which requires full decompression) using WWW (not WWW.LoadFromCacheOrDownload) it might actually save a memory, as a bundle will be recompressed to in-memory lz4 bundle. That's the only case when it actually saves RAM.

 

I use LZMA bundle, but found that the serializedFile's size is also 0.5MB each bundle like LZ4 format.
Could you tell me the reason?

 

Hi!

Yes. I assume you are talking about Android. On Android a cache per each SerializedFile is 512KB. That size was introduced as an optimization which compensates slow disk reads and zip decompression for data in Resources folder. That also had a undesirable effect on bundles.
In 5.6 this is changed and size is reduced to 8KB.

 

I have tested in 5.6 with Android device.

Test Flow.

1. Build bundle in 5.6
2. Build player in 5.6
3. Just 'AssetBundle.LoadFromFile' 80 bundles.
4. SerializedFile Profiling..

Cache buffer size is still large.

I want more information about "In 5.6 this is changed and size is reduced to 8KB".

Can you help me?