Community Format Proposal: .UNITYMESH

I think we've seen a lot of content importers for Unity that don't rely on stuff like asset packages. For example, sound, texture files, etc. What we haven't really seen is a good model importer.
I think it would be neat if as a community we could organize our own format for the purpose. I propose something that mirrors the Mesh class in Unity, so as to be almost a straight binary dump of the data contained in the Mesh. This means the data is already in the correct format Unity expects, and means data importers and exporters will be much easier to write.

The data is organized into lumps. The first lump, Header, looks like this:

Code (csharp):

1.  
2. {
3.     char[3] signature = UNM
4.     byte version = 0
5.     uint vertexLumpOffset --pointer to start of vertices lump
6.     uint uv1LumpOffset --pointer to start of uv1 lump
7.     uint uv2LumpOffset --pointer to start of uv2 lump
8.     uint submeshesLumpOffset --pointer to start of submeshes lump (submesh lump contains pointers to a triangle lump per submesh)
9.     uint normalsLumpOffset --pointer to start of normals lump
10.     uint tangentsLumpOffset --pointer to start of tangents lump
11.     uint colorsLumpOffset --pointer to start of colors lump
12.     uint boneWeightsLumpOffset --pointer to start of bone weights lump
13.     uint bindPosesLumpOffset --pointer to start of bind poses lump
14. }

And the lump formats:

Code (csharp):

1.  
2. VERTICES
3. {
4.     ushort vertexCount
5.     float v0.x
6.     float v0.y
7.     float v0.z
8.     float v1.x
9.     [etc]
10. }
11.  
12. UV(1,2)
13. {
14.     float uv0.x
15.     float uv0.y
16.     float uv1.x
17.     [etc]
18. }
19.  
20. SUBMESHES
21. {
22.     uint submeshCount
23.     uint triangle0LumpOffset
24.     uint triangle1LumpOffset
25.     uint triangle2LumpOffset
26.     [etc]
27. }
28.  
29. TRIANGLES
30. {
31.     uint triangleCount
32.     uint v0
33.     uint v1
34.     uint v2
35.     [etc]
36. }
37.  
38. NORMALS
39. {
40.     float v0.x
41.     float v0.y
42.     float v0.z
43.     float v1.x
44.     [etc]
45. }
46.  
47. TANGENTS
48. {
49.     float v0.x
50.     float v0.y
51.     float v0.z
52.     float v0.w
53.     float v1.x
54.     [etc]
55. }
56.  
57. COLORS (stores Color32 as a single int, one byte per component)
58. {
59.     uint c0
60.     uint c1
61.     uint c2
62.     [etc]
63. }
64.  
65. BONE WEIGHTS
66. {
67.     uint b0.index0
68.     uint b0.index1
69.     uint b0.index2
70.     uint b0.index3
71.     float b0.weight0
72.     float b0.weight1
73.     float b0.weight2
74.     float b0.weight3
75.     uint b1.index0
76.     [etc]
77. }
78.  
79. BINDPOSES
80. {
81.     float bp0.row0.x
82.     float bp0.row0.y
83.     float bp0.row0.z
84.     float bp0.row0.w
85.     float bp0.row1.x
86.     [...]
87.     float bp0.row3.w
88.     float bp1.row0.x
89.     [etc]
90. }
91.  

The idea behind this format is that, as I mentioned, it mirrors Unity's Mesh class making it very easy to import. It also supports skinned meshes, submeshes, etc.
Unity Indie users could use this as a way to implement streaming assets, perhaps with an export plugin that allows them to export UNITYMESH files from a mesh inside the Unity Editor, then this can be placed on an external source such as a web server. You might even create 'asset packages', basically ZIP files containing UNITYMESH files, sound files, and textures.

Thoughts?

 

I actually wrote an open source OBJ importer with the idea that I'd be able to import assets at runtime with it. Of course, they lack bones and animations, so OBJ isn't the best format. It's just really easy.

I don't have time or motivation to help with this project, but I wish you luck at it.

 

I'd also take a look at Assimp and the Collada format.

As you said, the benefit is that it would be much easier to load and export the model in the format you've proposed. The downside is that you have to write, for each modeling program, an exporter/importer for that format.

I personally like the Collada format as it supports bones and animations. It's also an open format that is widely supported. It is xml based, so the files are larger than binary equivalents however. Currently I've implemented a bastardized version of the Collada spec for my 3D tileset files. The tricky bit with Collada files is pairing the accessors with the data. It's not bad, but it can get kinda verbose rather quickly.

I too wish you luck with this project, it sounds interesting.

 

Yeah.... I think this wouldn't be so much for a general model format (although certainly it could be used for that if someone were dedicated enough to write exporters for popular modeling packages, which I'm not), but rather a useful base for streaming assets in games that don't have access to Pro. That is, you could export your mesh from Unity with an editor extension, upload the UNITYMESH file to your content server, and download it at runtime. It could also be packed into ZIP files along with sound files and images as a sort of "asset package"

 

For that, it sounds like you'd really just need to write a de/serializer for the Unity Mesh class. For that matter, you could take a little extra time and write a conversion method to/from the Assimp in-memory format and you'd have your general model format.

Basically, Assimp is a library that provides conversion methods for virtually all model formats to a single, in-memory format. You can then (as some people have done) serialize that in-memory format to some other format (such as Json in one example). Assimp is written in C++, but there's a C# wrapper sitting out there somewhere (granted, it uses P/Invoke).

 

Won't work for me unfortunately, I've only got an Indie license so I can't use C++ plugins.

 

I was more assuming one would write an external program for conversion to/from other formats to/from the proposed .unitymesh format.

 

doesn't unity have a maximum of 65535 vertices per mesh? you could reduce the vertexCount variable to a ushort if so.

 

Yes it does, and good point.

 

Instead of pre-defined lumps and header i'd go for a chunk based ID approach instead, this means you can include or exclude any chunks as you see fit.

Its used in the old 3ds format (see here) and isn't really much more complex to parse than a pre-defined header. Each chunk defines what type it is and the offset to the next chunk, then you have the actual chunk data. IT also means its extendible as you can add you own custom chunk ID's. These ID's and chunks are simply ignored when not defined in your importer so they still have backwards compatibility.

 

So a chunk might be:

ushort ChunkID
uint ChunkLength
byte[] ChunkData

A first pass might be the importer gathering the data for every chunk, and then attempting to parse chunks that have recognized IDs.

That sounds reasonable. Might actually end up slightly easier as well...

 

Yeah pretty much, though i'm not sure you need passes for importing, you just loop through the chunks as you come to them, extract data for the ones you know, skip the ones you don't. You don't expect chunks to be in any particular order, though obviously you try to be consistent.

As for a chunk it can be developed further so that you have


ushort ChunkID -- e.g. 0001 for vertices
uint ChunkLength
-- Actual chunk data in this case for vertices
ushort vertexCount -- number of vertices
float v0.x
float v0.y
float v0.z
float v1.x
... etc
ushort ChunkID -- e.g. 0002 for uv's
uint ChunkLength
-- Actual Chunk for UV
ushort UVChannelIndex -- which uv channel this is for
ushort UVCount -- number of UV coordinates
float uv0.x
float uv0.y
... etc
ushort ChunkID
uint ChunkLength
.. etc


where chunkLength is the number of bytes of the chunk data, it should not include the chunkID or the uint of the chunkLength. So this way you read the first ushort to get the first chunkID, then you read the next uint to get the chunk length, then you decide if you know what to do with this chunk. If not skip ahead by chunkLength bytes, if you do then parse the chunk.

One issue to keep in mind is how to deal with strings. You can either expect to read by byte until you get a zero byte or split any strings into a subchunk that includes the length of the string in bytes.

e.g.
ushort ChunkID -- e.g. 0024 Material
uint ChunkLength
byte materialNameLength - assuming max of 255 characters.
bytes[] materialname
etc

 

if you do the chunks, then you don't need vertex count, triangle count, or submesh count in their respective chunks, as you already know how long that chunk is.

 

Thanks for the pointer on using chunks, guys. I was originally basing this off of my prior experience writing a parser for Quake maps (where everything is stored in a lump, and there's basically an offset to each lump included near the start of the file). But chunks is proving to be much easier - I've started writing this with a sort of plugin-style architecture, where you write "plugins" for each chunk ID, and this plugin is responsible for parsing the chunk data. Makes things very modular, and easy to extend.

 

I would be very interested in this project, I currently working at a serializer for my meshes and I think using a common format is a very clever idea.

Also for importing an easier option would be to do an Editor extension to export to UNITYMESH...

Keep us posted...

 

I've worked on this myself, and can roundtrip a mesh with all data identical. (tested on a rigged mesh with two UV sets and vertex color information)

I changed a bit how the uv and submesh/triangle information was stored from what was originally posted, as they couldn't have been stored as they were in order to fully support all the mesh data.

https://github.com/jbruening/UnityMesh

project is MIT license.

 

I would have the extension as .UMESH because long extensions suck.

 

the processor doesn't care about the file extension, it just reads from a Stream object. So long as the stream follows the format, it'll work.

 

Ah, very cool. I had written most of my importer, but never got around to writing an exporter. Nice to see a full GIT project doing the full range
Maybe I'll write an Editor utility out of this that lets you pick a source Mesh and a destination file to write the .UMESH (or .UNM, or whatever) format to.

 

Did you ever get around to expanding on this at all? A utility to export models from the editor would be handy. In fact, a way to recreate prefabs from a stream, including the material, scripts, etc would be excellent. There's another thread looking into replacements for asset bundles, and this would help a lot.

http://forum.unity3d.com/threads/203094-Unity-native-prefab-texture-streaming

 

I already made one about a month ago. It's in the github repo. https://github.com/jbruening/UnityMesh/blob/master/scripts/editor/ExportUnityMesh.cs

it's not exactly difficult to use, as you can see. You use one static function and pass in a stream object.

 

I would have liked to get around to this, but naturally enough my life seems to have had other ideas as I've been quite busy lately.

 

Bringing up an old thread, but i stumbled across it.
Im using blender and the fbx exporter is a bit hit a miss.
Would it be possible to make a direct unity exporter from blender.
So blender saves out in the unity format....?

 

The import step is required; there's no option to plug the native format directly into the editor, assuming anyone even knows what it is (aside from Unity employees...hopefully). I wouldn't be surprised if the format changed somewhat from one version to another anyway.

--Eric