Data Upgrade Script Data Upgrade

Very interesting feature. However i dont like the fact that every time you change data layout, you have to reimplement lost fields in upgrader with exactly same type and name without optional way to restore these infos inside Upgrade call.

Well, disciplined programmers surely will change upgrader immediately after every layout change but would be nice if there was a safeguard. Could be possible to implement something like this?

Code (CSharp):

1. public void Upgrade(ref ProjectileLauncher launcher, int version, SerializedProperty[] lostFields) //or oldFields if hard/impossible to filter out not lost fields

i believe this would enable querying lost fields' name and type and even get their value before layout change.

 

I heard about this at Unite and am excited by this feature and what I've been reading in the discussion.

1. I will vote as well for a "force this process" API call which definitively updates the serialized data it can (by path or guid is fine) (as well as doing any other serialization changes). I can think of several projects where we would have benefited from having this API even without this data upgrade feature. Specifically when updating Unity versions or data fields, we'd run into the issue where files would be "changed" by re-serialization on artists/designer machines, but the change would be meaningless to that individual, so they wouldn't know exactly what to do with those changes.
2. One problem we have is guid changes, almost always caused by VCS .meta file mangling. We've also had times where needing to convert MonoBehaviour/ScriptableObject references between a dll and loose scripts was needed. Is that a potential extension or a tool that could be added to this, or as a utility to repair that sort of damage in lieu of text file search/replace across the project? (Bonus points if there was some sort of API that could let us know if Unity had *ever* seen a file with a particular guid, and what it knew about that file)
   
3. Is there any recourse here if member names are the same between versions of the data? Ex: Previously a field of type Sprite was called "image" and later it is changed so that the field is of a scriptable object type instead, but it remains named "image". If you put a field in the updater called "image" of the Sprite type, would that get the contents of the property "image" in the serialized if it was a Sprite, or would it just not be able to do anything useful in that case?
4. Perhaps it's present somewhere in this thread, but is there an order of precedence between FormerlySerializedAs and the updater field names, when it's trying to pass along data into fields?

 

On a related note. In current versions of Unity, I use my own JSON serialization system to achieve the same goal - support for data upgrade. One of the biggest problems is saving/loading references to nested assets. In my case, Sprites that are within a Texture asset (you know, the little triangle fold out on the Texture that). There is no public API for getting the FileID, except for undocumented Unsupported.GetLocalIdentifierInFile method, which is something I'd like not to rely on. Also, there is no API allowing to get the reference to a Sprite asset even if I have both the GUID of the Texture asset and the FileID of the Sprite. This results in some horribly inefficient code like this:

Code (CSharp):

1. public static Object GetAssetDatabaseObject(string guid, int localIdentifier) {
2.     string assetPath = AssetDatabase.GUIDToAssetPath(guid);
3.     if (String.IsNullOrEmpty(assetPath))
4.         return null;
5.  
6.     // This actually loads the whole Texture in the memory,
7.     // even if I am really only interested in Sprites
8.     Object[] objects = AssetDatabase.LoadAllAssetsAtPath(assetPath);
9.     if (objects == null || objects.Length == 0)
10.         return null;
11.  
12.     for (int i = 0; i < objects.Length; i++) {
13.         Object obj = objects[i];
14.         int objectLocalId = obj.GetLocalIdentifierInFile();
15.         if (objectLocalId == localIdentifier)
16.             return obj;
17.     }
18.  
19.     return null;
20. }

So... While you guys are at it, perhaps you could add an official API to get the FileID, and to get a reference to an Object by its GUID and FileID combination? That would at least help developers to implement their own data upgrade system.

Thanks.

 

Have you tried abusing the recent changes to EditorJsonUtility? I think it will be able to translate between a (manually generated) string with guid and fileid into an actual object reference.

 

Any news on this? It would be really nice for this to be supported out of the box.

 

Having to call EditorJsonUtility just to get an object reference is an overkill. After all, EditorJsonUtility does this via some internal API, so why not just expose the single missing piece as a public API?

 

Just gave the version from the 2017.3 beta a go.

Our first commit on this project is from mid March, and the Assets folder has 6969 files in it, totaling at 2.27 GB. After running ForceReserializeAssets, 4199 of those files were changed. Looking over the data, you've added an externalObjects field to .meta files, probably in 2017.3, so that means that every single .meta file has been changed.

After removing meta files from the equation, and rounding a bit, we have just under 3500 files, and about 750 of them were changed by running ForceReserializeAssets - over a fifth! The first commit was with editor version 5.5.1f1, so it's jumped quite a few versions.

Question - when we open an old project in a new version of the editor, Unity goes through all the files to check which ones needs to be updated already. Could we perhaps get the option to also force reserialization when that happens? I'd be happy with a command line option to do that.
Now (afaik) we have to open the project, wait for Unity to parse all files to check which one needs to be updated, and then run ForceReserializeAssets to essentially have Unity do the same thing again, just this time rewriting all files instead of just those where it's absolutely necessary. Having just one parse-all-files round would be nice.

 

I'm talking about the "Your project was last opened with a different setup of Unity (...) this may require re-import" thing. I was under the impression that it checked through files to figure out which ones to change. It does take a very long time, and it shows an asset import progress bar, but maybe I'm wrong?

 

More feedback on the ForceReserializeAssets API:

- Again, it's very good! Thank you!
- Running the method seems to update the timeCreated field on .meta files? I ran it once, commited everything, and ran it again. Quite a few files seems to have had their timeCreated field updated to when I ran the command. Is that intentional?
- There's something going on with rotation values. When running the reserializer twice, I get changes in some prefabs and scenes that looks like this:



Which is slightly annoying.

All in all, when running it the second time, I got 90 changed files, most of which had updated timestamps or minor changes to floating point values.

 

I'll see if I can figure out something. Note that the time created field wasn't updated on everything, but instead on... idk, about a hundred assets (out of thousands).

I'll run the reserializer on all the files over lunch, and then try to apply it to a single file, see what happens.

 

Okay, I've got a repro case. How do you want me to send it? It's a smallish project, where both timeCreated and m_LocalRotation gets changed for some files when I run reserialize all a second time.

 

Case 984239

 

that m_localRotation issue must be old. Euler to Quaternion to Euler issues, or?

 

It didn't even register in my head as an issue, because we get precision differences in the serialisation practically every time we change major(-ish) Unity version, very much including m_LocalRotation. I guess I should be a little more trigger happy with bug reports!

 

Thanks for working on this feature!

 

Hello,

The link in the first page is for Unity 5.4.0f3. Where newer version can be found ?

Thanks

 

Any update on the progress of this? I'm working on a fairly large open source project where I'll want to change how the data's layed out in the backend in ways that [FormerlySerializedAs] won't be enough to handle. The data upgrader interface would be great to have in this situation.

I assume you're all really busy with jobs and ecs and stabilizing the new .NET an all that, so I'm not expecting anything in the near future. I'm just wondering if this is something that's still worked on and could be expected within a couple of major releases.

 

Well this is a bummer, I got all excited for upgrading our project to improve workflow, merging and a bunch of stuff. Now I realise the build is just for 5.4

While I am here, I will throw my hat into the ring and give a description of how I was expecting this feature to work. I hoped unity would give us a hook that worked the same/similar way to [FormerlySerializedAs]. The attribute would provide the (previous) Serialized data and the fieldinfo of the field. That way we could manipulate the data in the way we need to fit the new fieldtype.

​

What we will probably end up doing is doing it through property drawers - find the new/old type through property relatives, manipulate the data, same new type data.

i.e. we want to change string fields to ScriptableObjects

Code (CSharp):

1. [hideinInspector]//added
2.  
3. string old
4.  
5. [ChangeThis]
6. SomeData new;

Code (CSharp):

1. Change This property Drawerer
2.    if objectreference == null
3.       Then Get old field string value
4.       make new SomeData using that value
5.       set new value

but I guess that may not fit everyone's use case


I hope unity addresses this issue soon

 

I know this thread has fallen off the edge of the world, but with every new Unity project I still wish there was a feature like this. We constantly refactor our components and there is no way around changing data and types while iterating on gameplay features.

The simplest case of renaming a field is handled alright by the FormerlySerializedAs attribute, but often we also switch types or between a single reference to an array of references and so on.

 

There's a somewhat hacky way of doing this that we use when this is necessary. It only works for ScriptableObjects, though, a MonoBehaviour solution would be much more involved (you'd have to manually iterate all prefabs and scenes). Anyway, recently posted this to a discord channel, so I'll copy it over here:

Code (csharp):

1.  
2. // starting with this SO, where you want myData to be an int - ie. "0" to 0
3.  
4. public class TestScript : ScriptableObject {
5.     public string myData;
6. }
7.  
8. // change to this:
9.  
10. public class TestScript : ScriptableObject, ISerializationCallbackReceiver {
11.     public string myData;
12.  
13.     public int myNewData;
14.  
15.     public void OnBeforeSerialize() {
16.         if (int.TryParse(myData, out int intVal))
17.             myNewData = intVal;
18.     }
19.  
20.     public void OnAfterDeserialize() { }
21.  
22.     [MenuItem("Temp/Transfer data")]
23.     public static void TransferData() {
24.         var allInstances = AssetDatabase.FindAssets("t:TestScript").Select(AssetDatabase.GUIDToAssetPath).ToArray();
25.         AssetDatabase.ForceReserializeAssets(allInstances);
26.     }
27. }
28.  
29. // call TransferData, save project
30. // change to this:
31.  
32. public class TestScript : ScriptableObject {
33.     [FormerlySerializedAs("myNewData")]
34.     public int myData;
35.  
36.     [MenuItem("Temp/Transfer data")]
37.     public static void TransferData() {
38.         var allInstances = AssetDatabase.FindAssets("t:TestScript").Select(AssetDatabase.GUIDToAssetPath).ToArray();
39.         AssetDatabase.ForceReserializeAssets(allInstances);
40.     }
41. }
42.  
43. // call TransferData, save project
44. // change to this:
45.  
46. public class TestScript : ScriptableObject {
47.     public int myData;
48. }

 

@superpig, I read through this thread again and came over this post:

I never thanked you properly for that answer, so thank you! It did bring up a question, though.

I recently wrote an asset post processor that edits animations for some very specific fbx. files. That works great, but I'm a bit unsure how I should make those changes get picked up when team members pull changes to the post processor. Right now I write a value to assetImporter.userData to make the file get reimported, but that feels dirty?

As you wrote in that post, and according to the documentation, you check the version of the post processor, and if that changes, "assets associated with this asset postprocessor will be reimported".

The problem is that I don't really know what "assets associated with this asset postprocessor" means. Does it have to do with what methods the asset post processor implements? You wrote "AssetPostprocessors that may affect that asset".

I tried to override GetVersion for my post processor and set it to 1, but that didn't cause any assets to get reimported

 

Tried. base.GetVersion() is 0. Setting it to any other int value (2, 3, 560909834, uint.maxValue) doesn't have any effect either.

 

#1153110

Although, solving that bug won't really make it useful. I've got an asset post processor that's supposed to target a very small set of .fbx files (files that need animation transitions to be stepped). I don't really want to reimport every single .fbx file in the project. That takes, what, 30 minutes?
Ideally, we would be able to associate asset post processors with certain folders, or use other kinds of filters that are cheap to check, but reduce the number of assets they work on.

 

Are there any news on the original Script Data Upgrade feature? It would be really useful to me at this moment to have a way of upgrading the data when upgrading scripts. Thanks!

 

A few projects later and I’m still hoping for a way of upgrading serialized data cleanly. It’s a recurring issue and always difficult to work around. Just wanted to leave my vote here to make this feature a thing!

 

It looks to me like Unity is now investing heavily in DOTS, where there is a better separation between code and data, and between authoring data and runtime data, with converters etc; overall I can imagine it being much easier to refactor DOTS data programmatically.
All this to say that, as much as I'd have liked to have that feature, I wouldn't hold my breath on Unity investing much on MonoBehaviour-related tech, which is on its way to the grave...

 

If I'm forced to learn a new engine (and DOTS is exactly that) then I'll review all the available options first. I don't feel like going back to programming using low-level, C-style coding with structs and indexes instead of classes. For the first time in 10 years I'm considering that I might have to switch to Unreal at some point in the future if that DOTS-thing goes too far.

 

I still want this data upgrader, even for DOTS. There’s always going to be serialized data in Unity. It’s always going to be YAML and the serializer changes really really slowly, so no matter what the tech stack of the future will be, at the bottom there will be serialized assets that hold data that needs to change as the design of our games changes. There’s really no way around it, even if my data is really nicely organized as flat lists and sub-scenes, still need to upgrade my bool array to an int array to enums when things grow.

Please revisit this tool!

 

Agreed. Despite the proposed workarounds, I still haven't found a successful solution to cases like this:

1. Standard component serialized normally.
2. New properties are added with some default values.
3. Upgrading existing instances require different default values in the new properties than newly created instances.

I can't figure out a way to determine whether the serialized data has the new properties or no in order to apply the proper default values.

 

Wow this is cool. I was just thinking about how awesome it'd be to be able to version serialized assets.

I mean, it is called an "AssetDatabase", and databases do have migrations...

 

Addressables

Addressables package is one of the biggest reason (imo) this should be a thing. Currently if you want to change a direct reference to a

AssetReference

you need to find all instances of the object you edited, and re-serialised it.

Imagine a world where you could change the data type, add a little code to convert one type to another, and never have to worry about that object instance you forgot to manually re-serialise. Script Data Upgrade.

 

That pretty much sums it up.

 

Resurfacing this thread, but by any chance is this feature still exists ? With another name or API maybe ? It seems very nice and must-have feature, but it seems it has been abandoned ... Why ?

 

The ability to force a reserialize a list of assets was a very big step towards solving the problem. See my post here for how we solve it. Note that iterating all scenes and prefabs and updating them is super easy, unlike what I write in that post, you just feed the results from AssetDatabase.FindAssets("t:scene") and AssetDatabase.FindAssets("trefab") into AssetDatabase.ForceReserializeAssets.

It falls apart a bit if there's a long time between when you introduce the ISeralizationCallbackReceiver and you know that all the data using the old format has been pushed to master, but that problem will probably only be really big on larger teams.

 

Thanks ! Your solution is nice, it's a good-enough workaround. But it's a shame Unity dropped that feature ...