IndexedDB files and WWW

Hi,

I have faced a problem with this. Imagine this situation:

Correct situation

1. Write to a file via "File.WriteAllText" -> This get's stored on IndexedDB.
2. Try to read it with "File.ReadAllText" -> Gets the file from IndexedDB

Wrong situation

1. Write to a file via "File.WriteAllText" -> This get's stored on IndexedDB.
2. Try to read it with "WWW" -> Tries to load it with "XMLHttpRequest" and I get an error that says "cannot load file:///idbfs/..."

I understand why it triggers that error but I also think that Unity could handle the "url" protocol used in WWW o UnityWebRequest so it can know if it needs to resolve it with "XMLHttpRequest" (!= file://) or "File API" or what you use for that (== file://).

This way will be consistent with other platforms where you can load local file system urls with WWW o UnityWebRequest.

 

I'm also interested in this.

 

@jonas-echterhoff or anyone?

 

Ok thanks.

 

I'm glad to hear that.

Thanks.

 

@Marco-Trivellato any news on the matter? This feature would also help with audio encoding issues, as, aside from WWW.audioClip, there seems to be no other ways to efficiently encode audio files at runtime (AudioClip.SetData with samples offset is not supported on WebGL).

 

Ah, It's nice that IO is now properly working in WebGL (and looks like we don't need to call syncfs manually anymore?), but I've meant using UnityWebRequest (WWW) over local path in WebGL. I think the topic's author meant the same thing. We need UnityWebRequest to be able to load files over the local IO in WebGL. It works in standalone and mobile platforms, but not in WebGL. The use case would be, for example, loading an audio clip from the IndexedDB cache via UnityWebRequestMultimedia.GetAudioClip.

 

Ah, okay. Anyway, thank you for the update

 

Has something changed in the latest versions of Unity? I tried your code in 2019.1 (in an empty project) and it seems like the text file is not saved to the persistent storage. Same with PlayerPrefs.Save - after reloading the data is lost.
Converted the project to 2017.3 and it worked fine.

 

ExternalEval doesnt work you need to make it a plugin although it doesnt seem to save anyway i think the problem is that every single time you reload unity the persistentDataPath changes

Code (CSharp):

1. mergeInto(LibraryManager.library, {
2.  
3.      SyncFiles : function()
4.      {
5.          FS.syncfs(false,function (err) {
6.          });
7.      },
8.     ReadFiles : function()
9.      {
10.          FS.syncfs(true,function (err) {
11.              SendMessage('MyGameObject', 'FinishedSnycing');
12.          });
13.      }
14. });
15.  

Code (csharp):

1.  
2. using System.Runtime.InteropServices;
3.  
4. #if UNITY_WEBGL && !UNITY_EDITOR
5. [DllImport("__Internal")]
6. private static extern void SyncFiles();
7.  
8. #endif
9.  
10.         SyncFiles();
11.  

 

Didn't test it in the latest 2018.3.x patch releases, but it's been working fine for me throughout 2017-2018 releases: https://github.com/Elringus/UnityCommon/blob/master/Assets/UnityCommon/Runtime/Utilities/IOUtils.cs

 

Thank you, this is helpful to me.

 

@Marco-Trivellato do we still need to manually invoke `FS.syncfs` via a JS plugin to persist the changes? Are there any plans to do that automatically or add a managed API, so we could do that via C# without external plugins?

 

I'm on 2018.4 - when accessing Application.persistentDatapath in WebGL, should I be using

UnityWebRequest.Get(Application.persistentDatapath + "/myFile.txt")

in a coroutine, yielding, and getting the text of the downloadHandler, OR should I just - as the example above shows - just use

File.ReadAllText(Application.persistentDataPath + "/myFile.txt")

? My code was using File.ReadAllText initially and I kept getting a null result despite logging indicating that the file at

Application.persistentDataPath + "/myFile.txt"

existed. When I switched to using the UnityWebRequest/yielding method, my NREs vanished.

That makes sense to me since UWR (previously WWW) is how you normally stream files (local or remote) but I found it odd that answers I've read specifically when dealing with IndexDB/persistence seem to differ, with no explanation for this discrepancy (that I've found)

 

@Marco-Trivellato
Can we have some news about that ! I really need to be able to load some ressources from persistentDataPath in WebGL (some mp3 with UnityWebRequestMultimedia for example) but can't do it because UnityWebRequest will always rework the URL thinking it's a relative path and adding the host at the start.

Is there a way to prevent the rework of the URL, or maybe another way to change the URL for the same path but without starting with "/" ?

 

bump

 

bump

 

It's a shame that UnityWebRequest can not read `file://` URLs on Web platform.

Currently, the only way to do this, is to send your file's byte array to JS, create a Blob out of it, then create a link for that Blob (using URL.createObjectURL()), then pass in that link to UnityWebRequest, which is too complicated.

But... speaking of which, UnityWebRequest should support loading assets from memory, which would solve all use-cases.

 

Did you guys create a bug report for this? If you don't, Unity will probably not track this request...

 

No, I did not. This thread is from 2017, and I don't expect Unity to do anything about it ...

I solved my problem using the solution I posted above.

Technically, it would be a feature request (not a bug), so they would not prioritize it.

 

I think it's a bug, because it works on other platforms but not webgl. They should fix it to work the same on webgl.

 

The method above is feasible and inspiring, and is exactly what unity is doing in the Audio.js inside the WebGLSupport folder except the UnityWebRequest part. So I managed to load the compressed audio data in memory by adding the following code to JS_Sound_Load function:

Code (JavaScript):

1. JS_Sound_Load: function (ptr, length, decompress, fmodSoundType) {
2.     if (WEBAudio.audioWebEnabled == 0)
3.         return 0;
4.     //inject audio data start
5.     var injectedAudioData = window.injectedAudioData;
6.     if (injectedAudioData)
7.     {
8.         ptr = injectedAudioData.ptr;
9.         length = injectedAudioData.length;
10.         delete window.injectedAudioData;
11.     }
12.     //inject audio data end
13. #if USE_PTHREADS
14. ...

and added the inject function to .jslib

Code (JavaScript):

1. mergeInto(LibraryManager.library, {
2.     InjectAudioData: function (data, length) {
3.         console.log("InjectAudioData "+data+"|"+length);
4.         window.injectedAudioData = {
5.             ptr: data,
6.             length: length
7.         };
8.     },
9.     EraseAudioData: function (){
10.         delete window.injectedAudioData;
11.     }
12. });

then call it like this

Code (CSharp):

1. JavaScriptBridge.InjectAudioData(data, data.Length);
2. dummyClip.LoadAudioData();
3. JavaScriptBridge.EraseAudioData();

The dummyClip is a small piece of audio that was imported into unity as Compressed In Memory, so that we can relace the content and leave everything else unchanged.
Still, this method only works for WebGL, and a native implementation of loading binary data in memory as compressed audio clip is highly appreciated.