How to check the hash code of an Assetbundle to validate

Hi guys.

The topic may not totally related to the Assetbundle Graph Tool, but it will still affect the people, so I hope we can get the answer here.

When building the assetbundles, the manifest will give out the computed hash128:

Code (CSharp):

1.                     Hash128 myHashCode = aManifest.GetAssetBundleHash(aAssetbundleBuildArray[i].assetBundleName);
2.                     string myHashCodeString = myHashCode.ToString();

That's nice and we can also use that hash value if it is already present in our download cache:

Code (CSharp):

1. Caching.IsVersionCached(aWww.url, myHashCode)

But how are we supposed to get the hashcode from the assetbundle to validate? Afaik, the hash code is not present in any form in the assetbundle, but we have to get the hash code from outside (file downloaded from server is our solution).

That doesn't help to validate that the downloaded data is correct though and here we come to my problem. Even with the hash code, I can't validate that the data my Client has downloaded is actually the correct one.

Am I missing something here? Would it be possible that Unity adds the hash code into the Assetbundle when creating the bundles? Is there any other way to validate the data (I've not found any way yet, as WWW.bytes is not readable for me and the Assetbundle property is not serializable to do a MD5 check by myself).

Thanks 4 help,
Patrick

 

Ok, thanks, I'll try that and report

 

@Reichert

Yes, I tried and it works, but not in the way I would like it to have. I thought about writing about it several times now, but I wanted to talk first to one of the Assetbundle System engineers if my implementation of our system is using the AB system wrong or if we missed something.

tl:dr
I assume the AB system suggests to have a "preloading mechanism" which loads and verifies all assetbundles, before the actual game starts, rather than check on the fly if an assetbundle is up2date or not (which is what we are doing at the moment).

long story:
We use the assetbundles to load battleground (base) and assets (custom additions) when the player is entering a mission, as well as for static assets (UI atlasses, music,...). While the former is only loaded when used, the later is loaded when the player starts the game.

Now the biggest problem we face is, that the player has to get from somewhere outside that an assetbundle has been updated. We use a small textfile (well, kind of small, but still 300 kb for us) which contains all data we need to tell the client that the assetbundle is updated: CRC, hash value and a size for our loading screen display. The progress of downloading and updating looks like:

1. player starts game, downloads "assetbundle_version.cfg".
2. game parses config and knows now the latest and correct CRCs, hash values and sizes for the assetbundles.
3. game gets on the preloading screen, checks and downloads static assetbundles (UI atlas, music,...) if needed.
4. player can login into game and decides what he would like to do.
5. player wants to play a mission, enters the mission and comes to a loading screen.
6. game checks which assetbundles are actual needed for that mission and tries to download the assetbundles (battleground, models, textures, materials,...) if needed (client knows the CRCs and hashvalues from the start of the game).

Now in the last step, we face our problem: if we update our assetbundles on our ftp while the player is playing, he could potentially download assetbundles with a different CRC than the client expects: the client downloaded the configuration file only at start of the game and we updated the assetbundles after that.

Now to fix that, we could download the configration file each time we want to check if we need to download assetbundles, but that would mean to download 300 kb each time the player wants to enter a mission (which happens quite often and is not suitable for a mobile game). Maybe we can change our WWW.LoadFromCacheOrDownload to not check the CRC and hashvalues later, after the game started.

In my conclusion, the current main problem in the assetbundle system is, that the hashvalues and CRCs are given in from outside. In a perfect world, we would simply tell the game to download an assetbundle from our ftp and it checks from a small download header if that is already the data the game knows and if it is not corrupted. No passing in an external configuration file.

Well, now I wrote the story before talking to the pros, but I'll post a follow up to this

Patrick

 

Ok, talked to Ryan at the Unite Europe and he also suggested something like the small version file. It sounds like a solution we can try.

I'm very happy that you are already working on some solutions for the problem. I've to admit though, if that will be some kind of Unity Service, we most probably have to stay away from it (legal stuff etc). But I'm looking forward to it anyway

Thanks again,
Patrick

 

@PAHeartBeat Your asset bundle issue is EXACTLY the same issue we ran in our company's game. The only difference is that I store our asset bundles' version/CRC hashes in a database, and I have a Python script return the version/CRC pair for that asset bundle for the specified platform. It's tedious and annoying because I have to make a call to our REST API to get the version/crc pair before I load an asset bundle. Not only that, but I have to manually open up all of the manifest files, so that I can copy and paste them into the database until I write a tool to do that for me.

I create a UnityWebRequest object with UnityWebRequest.GetAssetBundle(), and call req.Send(). Then, if an updated asset bundle is found, all Unity does to let you know is throw an exception saying there's a CRC mismatch as if it downloaded incorrectly. It's nice that there's a way to check if it downloaded correctly, but the problem is that this is the EXACT same mechanism needed to update the asset bundle too... I'm not checking for the AssetBundle's integrity to see if it downloaded correctly in this context. I literally want to just download the latest version from the server, or pull from the cache. I haven't found much documentation on what that "version" code is for, but it seems to be Unity's version code for the asset bundle's file format for forward compatibility in the future (from what I've gathered).

 

Hello guys. I want to ask it to you, because I haven’t found working examples on the internet. I’m also interested in AssetBundle’s hash which is stored in a file with the .manifest extension. I always have to check if there is an update available on the server (check if the file have changed). If I’m not mistaken, there is a class AssetBundleManifest and a method GetAssetBundleHash, but my attempts to use it result in failure. I cannot get AssetBundleMaifest.

I used this source as an example: https://docs.unity3d.com/ScriptReference/Caching.html.
Despite using local loading in the example, I still cannot do it. I tried to load both the file with the .manifest extension and the AssetBundle file itself, what could be the problem?

May be you could show me a working example of getting AssetBundleManifest and calling GetAssetBundleHash.

 

@ViktorAcadem
You get the Hash from AssetBundleManifest.GetAssetBundleHash which you store in a file. You have to download the file later then, so your client will know which Hash values are the current ones on the server.

The AssetBundleManifest comes from your BuildPipeline.BuildAssetBundles call.

Pretty much straight forward on creation side.

On client side later, you use WWW.LoadFromCacheOrDownload and use the Hash value as parameter.

 

@pahe
WWW.LoadFromCacheOrDownload TO BE DEPRECATED (Use UnityWebRequest)
https://docs.unity3d.com/Manual/AssetBundles-Native.html

Following the recommendations I use UnityWebRequest, but the problem is not here.

The problem is that I don’t correctly use AssetBundleManifest.GetAssetBundleHash, it always leads to the error. I would like to see the applying of this method on the working example. As a result of calling BuildPipeline.BuildAssetBundles, we get two files, one with the .manifest extension, the other main one without the extension. Which one should be loaded to get hash using AssetBundleManifest.GetAssetBundleHash? The thing is that I need to track the updates on the server and, if necessary, load AssetBundle. For this purpose, I store not only the AssetBundle file on the server, but also a text file with the following data: crc, version, hash.

I read the saved file as follows:

Code (CSharp):

1. public IEnumerator LoadManifest(string path)
  
2. {
      
3.    using (WWW www = new WWW(path))
      
4.    {
          
5.       yield return www;
        
6.       if (!string.IsNullOrEmpty(www.error))
          
7.       {
              
8.            errorMessage = www.error;
              
9.            yield break;
          
10.       }
          
11.       ReadFile (www.text);
      
12.     }
  
13. }
    
14.  
15. private void ReadFile(string text)
16. {
      
17.    string[] readtext = text.Split ("\ n"[0]);
      
18.    uint crc = uint.Parse(readtext [0]);
      
19.    int version = int.Parse(readtext [1]);
      
20.    string hash = readtext [2];
      
21.    Manifest = new AssetInfo (crc, version, hash);
  
22. }


Despite this solution, I would like to learn how to get hash correctly using AssetBundleManifest.GetAssetBundleHash. I would be very grateful if you would show by example how AssetBundleManifest.GetAssetBundleHash works, when downloading a new file from the server, or at least locally, because I didn’t find working examples.

 

Thanks for the info, didn't know that. I'll have a look into that. I guess that with 2017 the function is deprecated, as we're on 5.6.x and I don't get that warning. Anyway, it should work both ways.

 

Do you have a working example of using AssetBundleManifest.GetAssetBundleHash in order to get Hash? Could you, please, show this example to me? Thank you in advance.

 

Well, you mean something like this?

Code (CSharp):

1. Hash128 aHashCode = aManifest.GetAssetBundleHash(aAssetbundleBuild.assetBundleName);
2. var aHashCodeString = aHashCode.ToString();
3. Debug.Log(aAssetbundleBuild.assetBundleName + " Hash: " + aHashCodeString + " CRC: " + aCrc + " filesize: " + aFileLength);
4.  

 

Yes, right, but I have a problem with lines earlier, when getting AssetBundleManifest (LoadAsset returns null).



Code (CSharp):

1. AssetBundle manifestBundle = AssetBundle.LoadFromFile(manifestBundlePath);
2. AssetBundleManifest manifest = manifestBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");



manifestBundlePath - is this a path to the file with a .manifest extension? There are some who points the way to the main file without an extension. Which file does manifestBundlePath point to? I tried different options, but it doesn’t work. I can’t understand how to get AssetBundleManifest correctly. 

How did you get aManifest? Which file did you download?

 

Hm. Do you need the manifest from runtime or to editor time?

Editor time (when creating the assetbundles) will give you the manifest when creating the Assetbundle:

Code (CSharp):

1. var aManifest = BuildPipeline.BuildAssetBundles(anAssetbundleOutputPath, aAssetbundleBuildArray, aBuildOptions, aBuildTarget);
2.  

For runtime, you have to download the manifest file and unpack it:

Code (CSharp):

1. assetBundleManifest = aDownload.assetBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");
2.  

Hope this helps. If not, please add some more info at which point (editor/runtime) you need the manifest file.

 

I'm interested in getting the manifest file at run time by downloading the file from the server. But I can’t even download it from the device’s memory. Here is an example of how my logic works:



As a result of the BuildPipeline.BuildAssetBundles work, I got the main file (filename) and the manifest file (filename.manifest). Then I move the received files to the project folder (StreamingAssets) and try to download them using the following lines:

Code (CSharp):

1. 

string path = Path.Combine(Application.streamingAssetsPath, "filename.manifest");
  
2. AssetBundle manifestBundle = AssetBundle.LoadFromFile(path);
  
3. AssetBundleManifest manifest = manifestBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");

4. Hash128 hash = manifest.GetAssetBundleHash(«bundleName»);



When I try to download "filename", I have an error in the 3rd line (manifest = null)

When I try to download "filename.manifest", I have an error "Unable to read header from archive file"

Where is the error?

How to download correctly a manifest file?

 

"filename.manifest" is no assetbundle, but the manifest file in text format. You need to download "filename", which is an assetbundle which contains the manifest file as AssetBundleManifest.

So, although you get an error when downloading "filename", that would be the correct way. I'm not using AssetBundle.LoadFromFile, but that should still work the same way. I'd suggest you debug that assetbundle by manifestBundle.LoadAllAssets() and give out what name and type is in it. Everything else looks fine in my opinion.

Edit: if you can't get it to work, I'll try to find some time at the weekend to create a showcase project for it.

 

Using the function manifestBundle.LoadAllAssets (), we get an array that contains only compressed GameObject. There is no AssetBundleManifest class in this array.


Here is an example of viewing all the downloaded resources:



Code (CSharp):

1. string path = Path.Combine(Application.streamingAssetsPath, "filename");
2. AssetBundle manifestBundle = AssetBundle.LoadFromFile(path);
3. object[] objects = manifestBundle.LoadAllAssets();
4. foreach(var obj in objects){
5.      Debug.Log(obj.GetType().ToString());
6. }

The contents of the console:

Code (CSharp):

1. UnityEngine.GameObject
2. UnityEngine.Debug:Log(Object)



Maybe I need to use special settings for AssetBundle build?

I build using the following lines:

Code (CSharp):

1. BuildPipeline.BuildAssetBundles (path, BuildAssetBundleOptions.None, EditorUserBuildSettings.activeBuildTarget);

 

Alright, I've created a small test project now, to make sure that I'm not talking about project specific stuff

Testproject
My project is very simply, only containing a Material, and empty folder called "Assetbundles" and an editor script to build the assetbundles. The editor script is under the Editor folder and looks like:

Code (CSharp):

1. public class abCreator
2. {
3.     [MenuItem("Assetbundles/Create Assetbundles")]
4.     public static void createAssetbundles () {
5.        
6.         BuildPipeline.BuildAssetBundles("Assetbundles", BuildAssetBundleOptions.None, BuildTarget.StandaloneWindows64);
7.     }
8. }

The material is created empty and I setup the assetbundle settings for it via inspector (in our project we do that via code, but it works the same way).



Now, when I use the editor script to create the assetbundles, I get the following files:



mymats is the assetbundle with my material.
mymats.mats (mymats.mats.manifest is the real name, .manifest is only removed in the project view) is the manifest file in text format for my assetbundle.
The first Assetbundles is the assetbundle with the manifest for my assetbundles which are contained in the output folder. (This is the assetbundle you need to download later, to get the manifest from!)
The second Assetbundles is the manifest file (again, .manifest is not shown in the project view) in text format, so you can see what is inside (afaik only for you to see what is actually inside later and if you want to check if GUIDs are set correctly, dependecies are set correctly, ...).

What do you need in the end?
You only need the two assetbundles. The Assetbundles assetbundle contains the manifest you need later. The mymats assetbundle contains your material.
The other two files (both manifest files in text format) which were created are AFAIK solely for debugging, if you want to check what is inside and if everything is correct referenced or you just want to see how it looks. (Please correct me if I'm wrong here).

Hope that helps

 

Thanks for the detailed series of answers. I understood where I made a mistake. As expected, I took the wrong files to get AssetBundleManifest, mistakenly assuming that it is in "mymats".

 

I will ask here as it is similar subject. (to not open new topic).
Let's say i have bundle called "flags".
I have custom updater -> i'm comparing hash of local "flags" bundle with one on server.
I'm getting hash from local "flags" bundle this way:

Code (CSharp):

1.  AssetBundleManifest myManifest = localAssetBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");
2.         Hash128 localHash = myManifest.GetAssetBundleHash("flags");

localAssetBundle is AssetBundles file creted while constructing bundles.

If hash not match -> game is downloading "flags" server version and save it as local.
Should i download and save as local "AssetBundles" and "AssetBundlesManifest" files?
Downloading and saving only "flags" and "flags.manifest" will give false (not matched hash) on next update check. However instantiated object is "new" - same like on server.
Or should i ignore "AssetBundles" and "AssetBundles.manifest" files and take local hash direct from "flags.manifest" ?
hope you know what i mean

 

Hi,

I was wondering about your usage of the GetAssetBundleHash function.
According to the documentation it is not recommend to use it for versioning:

"Note that while AssetBundleManifest.GetAssetBundleHash() may also be used for this purpose, we don’t recommend this function for versioning, as it provides just an estimation, and not a true hash calculation)."

Is it reliable from your experience?

 

We use it in dozens of applications as a versioning tool and have yet to run into problems.

 

Good to hear that, thanks!

 

No, it isn't reliable。
I met the same assets different content has the same hash.

 

In my case, bundle downloaded successful, but manifest is always null

Code (CSharp):

1. AssetBundle bundle = DownloadHandlerAssetBundle.GetContent(requestVideo);
2. AssetBundleManifest manifest = bundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");

 

the asset bundle manifst is a separate file procuces by the build and it can be found in the assetbundles file

 

I second Dred236's comment- I have found this to be unreliable. I also fairly regularly encounter situations where the same hash value is assigned for bundle files that aren't exactly the same. I'm looking into other options now.

 

So my issue is that I have an assetbundle I created for webGL for a scene to load. My code to load that and launch the scene works as expected, until I make a new version of that assetbundle and upload it to my server. If I use the same browser I had loaded the first version of the assetbundle then the browser seems to use that cached version, even though the one on the server has changed.

So looking at the docs there appears to be one of two ways to "force" it to load a new version of an asset bundle that had been cached locally. Either I pass in a uint "version" and bump that each time I create another version of the asset bundle, OR I pass in a hash for the assetbundle. My guess is internally Unity looks at the local cached version, compares the hash, and if it doesn't match what was used in the function call it goes out to the server to grab the new version.

Assuming I have that correct, I need to store the hash for the latest version somewhere on the server; similar to the config file mentioned above by another poster. However, I noticed that in the assetbundle.manifest file it has the hash for the asset bundle. So why not just upload the .manifest file along with the matching assetbundle, then *somehow* parse the .manifest file to get that hash. At that point I can load the assetbundle by passing the name and hash I read from the manifest file.

Another post above says "you don't need to do that to get the hash, just load the assetbundle and use the available function to get a manifest". Well, that doesn't work if my browser returns a local cached file.

So, why can't we load the assetbundle.manifest file to get the hash for the bundle, then when we call UnityWebRequestAssetBundle.GetAssetBundle() use that hash to ensure we get the correct bundle? All I would need is a way to properly parse the remote .manifest file and the rest is easy. No one seems to have shown how to do this step.

I can skip it by creating a separate "config" type file with a bundlename, hash pair, but why go to the trouble if this information is already available in the .manifest file?

Perhaps I totally misunderstand the entire assetbundle system, and there is a much simpler way to do what I want. Any help would be greatly appreciated!

 

thats how i used it also.. parsing the manifest to get hash (this was few years ago, not sure if still valid)
https://github.com/UnityCommunity/U...ets/Scripts/AssetBundles/AssetBundleLoader.cs

 

Depending on what build pipeline you use you may already have an assetbundlemanifest generated for you, I answered a similar question here:

https://forum.unity.com/threads/how...er-with-unitywebrequest.1028164/#post-6660664

Yes, you could probaby parse the .manifest file (which is normaly used by the build system for incremental builds I think), but then you wouldn't have dependency info like an assestbundlemanifest file does.