Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Reducing script compile time or a better workflow to reduce excessive recompiling

Discussion in 'Scripting' started by guavaman, Aug 20, 2012.

  1. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    SOLUTIONS: Each of these solutions reduce compile time. Use one or all of them together to get the biggest speed increase.

    #1 - Convert project to C# (extremely effective)
    I was using UnityScript. Convert your project to C# -- it compiles 4X FASTER than UnityScript! Compile times went from 15s to 3.5s. See this post for some numbers.

    #2 - Convert some/all scripts to DLLs (very effective but complex)
    NOTE: This solution involves a complicated setup using externally compiled DLLs. While it works, it's probably far too complicated for most users. Compile times reduced from 15s to 2-3 seconds or less. See my answer here.

    #3 - Convert ALL remaining .js files to C# (somewhat effective)
    I was able to reduce compile times an additional 3.25 seconds by converting ALL lingering .js files in my project to C#, including Standard Assets. See this post for details.

    Overall, using all 3 methods above, my script compile times are now down from 15 seconds every time I update a script to just 2 seconds. That's a 7.5X performance increase! See my happy face over there? :)

    ----------------------------

    Original Post:

    A common problem is that as a project grows in size, so does the time it takes to compile the scripts. When one change is made to any script, all must be recompiled. The result is a maddening amount of waiting during development and testing. My current project has 3.78MB in 164 script files (not including the default stuff in Standard Assets and Pro Standard Assets). It takes 15 seconds to compile scripts every time I make a tiny change, which can translate into hours of wasted time per day waiting. I'm currently working on a Core i7 3.6 + 12GB mem. BTW, I'm coding in Unityscript.

    Questions:
    1) Are there any ways to reduce compile times apart from getting a new CPU? I doubt getting an SSD would help here. I already know about this: Overview: Script compilation (Advanced)

    2) I'm curious about other people's compile times compared to number of scripts / megs of scripts, specifically with regards to C#. I'm using Unityscript, and I'm wondering if it would be significantly faster compiling in C#. I don't think there's any way possible I could convert my project to C# at this point, but maybe it would help in future projects.

    3) I feel like I'm missing something with regards to a smart workflow. I did some work on C# in Visual Studio recently (non-Unity stuff) and found it to be very quick and efficient because of real-time syntax error underlining, real-time error warnings, break points and debugging, and excellent IntelliSense. Working in Unity is a completely opposite experience. I feel stuck in the mud. As far as I've found, there's no syntax error underlining or other error warnings without doing a compile (read: waiting...), MD debugging integration is frustrating so I haven't used it (it always tries to open a second Unity instance), and I can't get MD to ever give me consistent results with code completion (it works maybe 5% of the time for me). Testing or bug fixing new code is always an endless back and forth between MD and the editor play button with a bunch of Debug.Log() calls thrown in as needed to find the problem, and of course there's all the waiting to compile just to add or remove a log call each time. (I know about the inspector debug mode to see vars, but on scripts with tons of arrays or complex objects it always crashes my machine so I can never use it.) Adding and debugging one small feature can take a day or more sometimes because of this slow workflow. Maybe I'm doing things all wrong, I don't know. I'm curious how some of you go about your routine when working.

    4) I'm currently getting about 600 warnings every time I compile. It's all trivial stuff, but I'm wondering if this might be making it take a bunch of extra time to compile and whether it would be worth taking the time to try to clean up the warnings. Or maybe there's a way to suppress warnings so it doesn't have to warn me every time? (Edit: Actually 90% of these warnings are things like "System.Collections is never used" in a Monobehavious script. I can't very well strip out the include from Monobehaviour to make this warning go away.)

    5) In MD, there's an option to Compile Assembly. I assume that means it's just going to compile whats in that particular assembly and not the whole project. Is there any way possible to split up your scripts into smaller assemblies so I could just re-compile the ones I'm working on at the moment. (I doubt it and I hope this question doesn't sound too dumb. It's a just shot in the dark. :p )

    Thanks!
     
    Last edited: Sep 20, 2012
    DrKucho likes this.
  2. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,350
  3. andererandre

    andererandre

    Joined:
    Dec 17, 2010
    Posts:
    683
    4) I would prefer fixing the warnings over deactivating them. If you fix all warnings right when they first appear it's no big deal and won't waste a lot of time. The moment will come when you assign a variable to itself and a warning saves your ass. Just manually assign null to all values you only set in the editor to fix the "will always have its default value null" warnings.

    5) You can use DLLs in Unity. You could compile some parts of your code into a DLL, possibly some of your utility classes that are pretty much complete and tested as they are.
     
    Novack likes this.
  4. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    I can't speak for Unityscript, nor whether switching to C# would improve your build times.

    At least with C#, I find it far more productive to build DLLs in Visual Studio and deploy them to the Unity project. I have some scripts to help with this which I might put on the asset store sometime - the scripts provide a UI inside Unity that lets you control some integration with the Visual Studio project, so it can make it automatically deploy the assemblies, add references to UnityEngine/UnityEditor in a manner that's independent of the Unity install directory, etc. I also have a smart launcher which knows how to launch or focus the right Visual Studio process for the script being edited - by default Unity tries to use its own auto-generated solution file, which is rubbish. The launcher also allows a fall back to MonoDevelop if the file is not C# code (e.g. UnityScript, or shader code).

    Visual Studio with Resharper is just insanely more productive than MonoDevelop. When I lost my last job, where we had this setup, I really wanted to give MonoDevelop a chance, but after about eight months of casual development I just couldn't stand it any more. The worst bit is when its auto-completion goes wrong and just decides to delete random characters from your last few lines of code, making it uncompilable...
     
  5. Tseng

    Tseng

    Joined:
    Nov 29, 2010
    Posts:
    1,217
    @ 3) Yes, when you want to debug your code, you must close your Unity Editor (not MonoDevelop) and then start it. In order for MonoDevelop to be able to debug in play mode it must attach itself to the Unity Editor process. That's why it tries to start a new instance (and fails if your Editor is already running)
     
  6. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    I've never had any trouble attaching MonoDevelop to a running Unity process, but I also don't do it very often and certainly not as a matter of course, as it seriously slows down code execution.
     
  7. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Thanks for the replies everyone! :)

    Unfortunately, this only applies to C#. I did try it myself and it doesn't compile. Also, Iooked through MSDN's list of warning and error codes and can't find anything for the biggest offenders, the namespace not used warnings. These warnings can potentially appear in every single .js file you make because the Monobehaviour class has built-in imports of various things like System.Collections, UnityEngine, and UnityEditor. If you don't use something from each namespace, you get a warning. UnityEditor is the silliest because the vast majority of your scripts are probably not going to be editor scripts so you'd never use anything from UnityEditor namespace and therefore get the warning in most of your files. I found a very ugly workaround which I posted here which involves putting a dummy function in every script to use each namespace. Yuck.

    Hmm, I agree disabling them en masse is a bit heavy-handed. Unfortunately, most of the warnings come from Monobehaviour and are not fixable without the ugly hack I posted above. I suppose it's better than nothing. I'll "fix" them and see if it helps compile times.

    Now this is intriguing... I'm trying to imagine how this could be implemented to speed up compile time.
    The majority of my scripts are Monobehaviour scripts. Utility classes only account for a handful of files and are not the bottleneck, so making DLLs out of those wouldn't help much. I imagine I couldn't pre-compile my Monobehaviour scripts as I'd have no way to assign them to objects at that point, correct? (Edit: I just found out this is not correct -- you can drag/drop monobehaviours from a pre-compiled dll as seen in this video.) Possibly some of the functions of each monobehaviour script could be broken out into separate non-monobehaviour classes, the data shared via objects, and those pre-compiled, although that would take a heck of a lot of reworking over all these scripts. Also, it doesn't seem to me like there would be a way to compile UnityScript into a DLL... maybe MD can do it. This is worth more research.

    Edit: Since you can drag and drop monobehaviours onto objects from within a pre-compiled DLL, I can potentially see replacing all my finished scripts with DLLs which would make compiling super fast. Now I just have to figure out if its possible to compile a UnityScript file into an external DLL...

    Wow, sounds like you've refined your workflow very well. Your C# tools wouldn't help me with this project, but after all this is done I'm certain to be done with MonoDevelop. I'd really like to have a more efficient way of working like that.

    I'm curious, what are the types of things you deploy via DLL and those you cannot and must use Unity to compile? About what percentage of your code is deployable via DLL?

    I get more frustrated with MonoDevelop every day, but there's no alternative for me. I'm surprised you can even use auto-completion. I've never had it delete random chars, but just about everything about it is broken for me. It's REALLY slow (like 3-5 second delay when you start typing), it almost never suggests the things I want like local variables or class properties, it's horribly inconsistent about what it does suggest to the point of being totally unusable. Again, we have that C# UScript gap here and most of the very experienced people on this board are using C#, so I just chalk it up to that.

    I should give this another chance and see if its usable. I guess I just have to get into the habit of starting Unity via MD -> Debug, so long as it doesn't try to open another Unity instance each time I try to debug. My project is big so starting Unity can take several minutes.
     
    Last edited: Aug 20, 2012
    DrKucho likes this.
  8. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    Almost everything works fine from precompiled DLLs these days. There are three rough edges I haven't got a solution for - one, you need two debugging files next to the DLL if you want MonoDevelop to be able to debug it, which is slightly cluttering. Two, the scripts within the DLL appear as a flat list, as if they were files inside the DLL "directory" - the flat list is annoying, it's a shame it's not broken down by namespace. And three - when you double-click a script in the DLL, Unity doesn't try to launch the right source file and line number as usual - it tries to open the DLL itself.

    None of those are very important though. Double-clicking on error messages in the console window works fine, and that's the most important bit.

    Yes, it's hard to guess how much difference that makes.
     
  9. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Agreed. Those downsides are not bad at all. If smart usage of this could save me 5-10 seconds off my compile time I'd be ecstatic.

    There is one big question in my mind about implementation (assuming I am able to compile DLLs from MD in Uscript to being with, which I will try doing next). I really doubt there's a way I could substitute the pre-compiled DLL version of a class for the already assigned reference made by drag and drop of the script from Unity. It seems to me you'd have to start your development using the pre-compiled technique. Pretty much if its a monobehaviour based script, I'm kind of out of luck as those almost always have some serialzed data.
     
  10. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Re: Compiling a DLL from MonoDevelop in UnityScript

    Okay, it does work! You cannot choose to compile to library from the MonoDevelop GUI, but you can set it in the *.unityproj file:

    Code (csharp):
    1. Change:
    2. <OutputType>Exe</OutputType>
    3.  
    4. To:
    5. <OutputType>Library</OutputType>
    6.  
    Lots of other stuff can be set in the unityproj file. Just open Assembly-UnityScript-firstpass.unityproj created by Unity in the root of your Unity project for more settings you can use.

    Excellent! (in Mr. Burns voice) Can't wait to implement...
     
    Last edited: Aug 21, 2012
  11. JamesLeeNZ

    JamesLeeNZ

    Joined:
    Nov 15, 2011
    Posts:
    5,616
    I read on here once that someone changed all the un-nessesary public var's to private and it improved things..
     
  12. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    Yes, it will lose track of the references. If you save assets in text format, you might be able to pull some tricks to programmatically fix up the broken references afterwards, but it won't be fun - or, it will be the quirky kind of fun, not the mainstream kind.
     
  13. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I just want to update the thread with the rest of my findings on the topic of precompiled DLLs.

    If you use UnityScript, it's not possible to compile a DLL Unity can actually use. Even though it's possible to compile a DLL, you cannot set a compile target and therefore will always get a .Net 4.0 DLL which may cause errors in Unity. See my post here for details. (Note: The particular error I was having has a workaround, but I still don't know whether using .Net 4.0 dlls in Unity is a good idea for other reasons.)

    Regardless, since my project is so far along and makes use of so many MonoBehaviours with serialized data, it's not really possible to offload much into external DLLs without destroying all that work setting up prefabs and serialized data. This is the kind of thing that's only really useful if designed for at the beginning of a project.

    It's not even feasible to offload things like managers into an external DLL at this point because they make use of so many classes that are defined in the Scripts folder (classes with serialized data like MonoBehaviours). A compiled DLL cannot communicate two-ways with classes defined after it in the same way that scripts in the Standard Assets folder cannot communicate with scripts in the normal Scripts folder. To provide a link between these, it would be possible to make base classes for everthing in the DLL that the Scripts could inherit from, but again, this is a design time issue as changing any of the classes with serialized data now to inherit from a different base class would destroy the serialized data.

    Which brings me all the way back to square one. It appears there's no real solution to making Unity compile significantly faster if you're stuck using UnityScript.

    (Edit: Well, I came up with a workaround that uses the same internal compiler Unity does and outputs a .NET 2.0 library files without the annoying errors. See this post for details. )
     
    Last edited: Aug 25, 2012
  14. Tseng

    Tseng

    Joined:
    Nov 29, 2010
    Posts:
    1,217
    You can even offload MonoBehaviours to a DLL (the DLL of course needs to reference UnityEngine.dll (found in C:\Program Files (x86)\Unity\Editor\Data\Managed) but you probably have to wrap them around a class inside your script folders, i.e.

    Code (csharp):
    1.  
    2. // in your DLL
    3. public class SomeClass : MonoBehaviour {
    4.     // your code
    5. }
    6.  
    7. // in your script folder:
    8.  
    9. public class SomeClass : YourDLLNameSpace.SomeClass {
    10. // can be empty, everything is already in the DLL's class
    11. }
    12.  
    This would allow you to move even w/o losing the references, maybe even create a script that will create the script files. But it's harder to debug this way
     
    Novack likes this.
  15. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I knew extending the class would work as I mentioned above but I had no idea it would keep the serialized data references! I just tested it out and indeed it does. Thanks! So maybe there is some possibility for this if I can possibly find a way to force .Net 2.0 compilation. (Currently exploring all the stuff in C:\Program Files (x86)\Unity\Editor\Data\Mono\lib\mono\2.0 to see if I can find the way Unity does its internal compile pass).

    Oh yeah, and namespaces don't work in UnityScript as far as I can tell, so I have to rename the base class to SomeClass_Base or something like that, which isn't that bad.

    If the .mdb file works, at least you should get line numbers. But trying to compile a MonoBehaviour to DLL is giving me the good old IEnumerable error with pdb2mdb (a known error I've seen mentioned many times before).

    Edit: My new workaround using Unity's internal command line compiler instead of MonoDevelop actually outputs an MDB file for problematic classes like MonoBehaviour.
     
    Last edited: Aug 25, 2012
  16. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I'd like to report that two weeks later, my problem is solved! I've reduced my compile time from 15 seconds to 2-3 seconds. Thanks for your help everyone in getting this figured out.

    In a nutshell, I moved every script I possibly could to external DLLs, though this was no easy task, especially because I'm using UnityScript. Moving to DLLs is pretty easy for basic utility classes, but most of the speedup comes from moving MonoBehaviours to DLLs which is much more difficult considering this was not planned for at the beginning of the project.

    Because of how I structured things, I can keep the scripts I'm currently working on as scripts in Unity and everything I'm mostly done with as DLLs. Once I'm done with them, I can shift them over into the DLL side. (Its just a little bit easier to work on things that need frequent changes as scripts rather than DLLs, but it's also workable just to work on everything as DLLs, though I doubt using Unity's debugger would work from precompiled DLLs).

    I'll outline my steps for anyone interested. Note that this requires Unity Pro unless you can figure out how to edit the binary scene and prefab files. Also, I assume you are comfortable writing your own small tool programs outside of Unity to help with certain small tasks like parsing text files.

    ----------------------------------------------------------

    STEP 1: *** BACK UP YOUR PROJECT! ***

    ///////////////////////////////////
    STEP 2: PREPARING YOUR DLL SOLUTION
    ///////////////////////////////////

    Create a new solution in MonoDevelop. Inside this solution, create one project for each category of your scripts. I organized mine into a 30 projects by type arranged just as they had been as scripts. (The more code you have in one category, the longer it will take to compile.) Copy all the scripts you want to convert to DLLs into their respective project folders.

    Common libraries:
    Add references to one or both UnityEngine.dll and UnityEditor.dll in all your projects that need them. (In Windows, find them in Unity\Editor\Data\Managed.) Also add System from the Packages tab.

    Utilities:
    I had to do a little re-organizing to make sure I had a sensible dependency chain set up. My utility classes, enums, and some hardcoded data tables would need to be referenced by most/all other projects, so I organized accordingly. Add references to these projects in all your other projects that need them.

    MonoBehaviours:
    I wanted to be able to keep my existing code entirely as is. Because much of the code in my MonoBehaviour classes relies on other MonoBehaviour-extended types (being used in other scripts), there are a lot of cross dependencies. I didn't want to have to wait forever for one gigantic project to compile every time I made a change (how Unity does it), so I chose to split up all MonoBehaviours into two classes each -- one base class and one class extended from the base class. I then added references the BaseClasses project to every other project that needed access to those classes (all the MonoBehaviour-containing projects). The base classes can be passed around and accessed by any MonoBehaviour that needs them.

    About Base Classes:
    Because of the size of my project, splitting up all the MonoBehaviours into base and extended classes required that I write a little tool to help me with the task. Essentially I went through all the script files and extracted all class properties/fields and all public methods and generated a large file containing all the base classes. For properties, I copied the property declaration and assignment (if any) as is so as to . For the methods I just created stubs (blank methods) out of them. (When one of these stub methods is called through the base class, the extended class's override method will be called instead.) All the game code will be in the extended classes. The base classes just provide a base framework.

    I used the original class name for my base classes because I didn't want to change any references in code. After I got my base classes file made, I modified each script file:
    Comment out the properies because these need to be only set in the base.
    Rename the class (and their .js/.cs files) with a suffix ("_Ext" in my case) to differentiate it from the base class. (You will never have to call these extended classes in code, just the base with the name you're used to.)
    Make the class extend the new base class.

    MonoDevelop configuration for C# users:
    Make sure your projects are set to output to .NET 2.0 or 3.5 or you MIGHT get some errors when using the dlls in Unity. Also make sure you enable optimization. (VStudio: project properties -> Build -> optimize code, MonoDev: project properties -> Compiler -> Enable Optimizations). If you do not enable this, your code will run slower.


    MonoDevelop configuration for UnityScript users:
    Getting DLLs to compile from UnityScript is a bit of a challenge and carries with it a couple of issues I'll get to later. To get MonoDevelop to output a Library instead of an Exe you can't use the GUI like in C#. Instead, open all your .unityproj files in your new solution and change the line <OutputType>Exe</OutputType> to <OutputType>Library</OutputType>.

    Compiling and Merging:
    Unity has a big problem which makes it impossible to just import your compiled DLLs if you're using the base class method outlined above. Unity will not recognize any subclasses in a DLL extended from a MonoBehaviour-based base class in a separate DLL. See this post for details.

    In order to get around this problem, you must merge your BaseClasses.dll with all your other BaseClass-extended DLLs. This way all the bases and their subclasses exist within one DLL by the time Unity gets it and there is no problem. (Well, except for the fact that your dropdown list will contain ALL your classes not in alphabetical order which is kind of ugly if you need to assign them.)

    To facilitate this, MonoDevelop projects should be set up in a particular way. Set all projects to output to solution/bin/merging so every DLL will be in one place when compiled. Also, right click on every reference in every project and uncheck "Local Copy" -- this will prevent MD from copying all the dependencies into this output folder with every compile. Also, copy the common dependencies to this folder or a subfolder: UnityEngine.dll, UnityEditor.dll, and for UnityScript users, Boo.Lang.dll and UnityScript.Lang.dll (find them in Unity\Editor\Data\Mono\lib\mono\2.0 in Windows).

    Microsoft ILMerge is the tool to use for merging the dlls. It runs from the command line. It will also merge PDB files so you will have line numbers reported from Unity for bug testing, etc. Create a batch file to merge the base class and any other MonoBehaviour-derrived class dlls together into one dll. I call mine Core.dll. You don't have to merge your utility classes as they can just be copied as is. (I recommending forcing ILMerge to ouput a .NET 2.0 class with the switch /targetplatform:v2.)

    It's unfortunate you have to go through this extra step, but merging only takes 2-3 seconds even with the 160 or so MonoBehaviours I have in my project and I only have to do it if I change something in a MonoBehaviour-derrived class.

    PDBs, MDBs, and Merging:
    Compiling with MonoDevelop outputs .PDB files, but Unity wants .MDB files. ILMerge also requires PDB files and outputs a merged PDB files. This PDB file must be converted to an MDB file for Unity. Unity comes with pdb2mdb.exe (in Unity\Editor\Data\Mono\lib\mono\2.0\ in Windows) which can convert PDBs to MDBs (well, most anyway). Run it from the command line pdb2mdb Assembly.dll, but your current directory MUST be the directory the .dll and .pdb reside in or you'll get an error. Convert your merged DLL and any non-merged utility class assemblies DLLs you have as well. (Add all this to your batch file created in the previous step.)

    NOTE: pdb2mdb is buggy and doesn't work on all assemblies. I had some problems trying to convert one of my assemblies (IEnumerable error), so I ended up having to make my own pdb2mdb converter using Mono.Cecil which you can get here. It was pretty easy with help from here on where to start.

    NOTE 2: .PDB files are output in the format AssemblyName.PDB, however .MDB files should be formatted AssemblyName.dll.mdb in order for Unity to recognize them. Plan for this accordingly.

    Copy to your Unity project:
    An addition to the merging and PDB converting batch file, also add copying the merged .dll + .mdb and any other loose .dlls + .mdbs you have to your UnityProject/Assets/Assemblies folder. (Anywhere under Assets is fine.)

    Workflow:
    I made a few helper programs and a pretty nice batch file to make this process smoother for me. After compiling a change to any of my DLLs, I just run a single batch file from a hotkey or any convenient launcher to detect changed DLLs, re-merge if necessary, and copy all the changed DLLs and MDBs to the Unity folder, and let me know of any problems such as locked files, etc. It's quick and only adds one additional button press beyond telling MD to compile. (I COULD add it to an "After Build" command but I don't want it running every time necessarily.)

    Result:
    Once your structure is set up, you should be able to compile each individual project relatively quickly instead of having to wait for everything to compile every time. You can also work on new scripts in the Unity Assets folder just like before, but you won't have to wait for hundreds of other scripts to compile because they're already dlls. Those scripts can later be moved to DLL format too once you're finished with them.

    Note about .NET version for UnityScript users:
    I have encountered an error before with some .dlls compiled from MonoDevelop because MonoDevelop ALWAYS compiles to .NET 4.0 even if you try to force it to use .NET 2.0. Most of the time this isn't a problem, but if you see some errors like in this post you may have to change some code to work around them OR resort to using Unity's internal compiler to compile instead of MonoDevelop to force it to compile in .NET 2.0 as I outlined here. But I recommend against doing this as it requires a TON of extra steps and a series of custom programs to help make the workflow less awkward. In the end I abandoned this approach for many reasons AND the side benefit of using ILMerge to merge your DLLs is you can force it to output a .NET 2.0 even though the source DLLs are .NET 4.0.

    /////////////////////////////////////////////////////////////////////////////
    STEP 3: Converting your project to use the DLLs instead of the loose scripts
    /////////////////////////////////////////////////////////////////////////////

    First, change a couple of settings in the editor: (Requires Unity Pro)
    Edit -> Project Settings -> Editor
    Version Control: Meta files
    Asset Serialization: Force text

    Version control: Meta files will make Unity output .meta files for every asset which contain the important GUID for the asset. Asset serialization: Force text will make it so Unity outputs text files for all the assets which will make it far easier for us to find/replace all the references to the old loose scripts across the project. (Note: It will take quite a while for Unity to generate these files on a large project.)

    Replacing Script References:
    All .prefab and .unity (scene) files must have their references to the old loose scripts replaced with references to the new classes in the DLLs.

    Unity uses a combination of a fileID and a guid to determine what class is referenced on a prefab or in a scene. These references are stored in the .prefab and .scene files and show up as a line of text because of the asset serialization setting above. This way, references can be easily changed in all objects across the project.

    See my post here for information about the reference format.

    The trick here is that you will have to build a reference table. You need to know both the guids/fileIds for all your loose scripts and the guids/fileIds for all your new extended classes. Once you get these, it's a simple matter of search and replace across the project referencing the table.

    In order to build the table, I suggest starting a new Unity project and copy all your old loose scripts WITH their .meta files to it. Create one gameobject and apply all your scripts to it. Create a prefab out of it. Write a small editor script to give you a list of all components on the object and reverse the order of the list because the prefab stores them in reverse order of how Unity's inspector shows or the output of a for(Component c in GetComponents(Component)) loop. Filter out any components that may have been added automatically as a result of assigning a MonoBehaviour like CharacterController, etc. Then parse the prefab file looking for m_Script: {fileID: # guid: # lines and copy out all the fileIds and guids. Finally compile it into a table.

    Delete the gameobject, prefab, and scripts and do the same thing again with the DLLs this time (also remember to bring the meta files as well so the guid will match when you copy it out.) Assign all the extended MonoBehaviour classes ("_Ext" classes in my case) to a gameobject and repeat. You don't need to do it for the base classes because you are never going to assign these directly to anything, just the extended ones.

    I suggest backing up your .prefabs and .scenes again now in case something goes wrong and you need to re-do the reference replacement. It's a lot less hassle than restoring the entire backup if you have a lot of other assets.

    Now with your table in hand, write a program to search the UnityProject/Assets folder for all .unity and .prefab files, search for fileId + guid pairs, and replace them with the fileId and guid of the matching extended class from the dll. (Using the suffix "_Ext" allowed me to easily find the replacement in the table.)

    Now open the project in Unity and verify the MonoBehaviours converted properly by looking at some prefabs. Make sure they were replaced with the expected class and that all the serialized data fields are there. Also check and make sure any serialized object references that point to a script type now point to the new extended type.

    If everything looks good, delete loose scripts.

    ---------------------------------------------------------------

    DONE!

    Your compile times should now vastly improved.

    Also, as a bonus, the reference replacement process could also be used when converting an entire project to another language. I'm seriously thinking about switching my whole project to C# so I can finally have a decent IDE to work on.

    EDIT: Since I made this post, I have indeed converted my project to C# and boy was I surprised at the results. C# compiles between 4X and 12.5X faster than UnityScript! Had I known this before, I wouldn't have bothered with the complex DLL setup because C# is fast enough to just keep using Unity's loose script scheme. Using the loose script setup, C# is 4X faster, and using the many-dll's set up, C# is 12.5X faster at compiling. Either way you go, you can't loose if you just bite the bullet and convert everything to C#.
     
    Last edited: Sep 28, 2012
  17. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    Excellent. It'll be interesting to hear in the future how maintainable your new setup is.

    Did you ever time how long it took to compile all your original scripts into one DLL? There's no real reason to suspect it would be any quicker than when Unity triggers the compile, but it might be easy to test and time.
     
  18. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Hehe... Yeah, I hope it works out. I suppose it's not difficult to go in reverse and put it back as separate scripts if it turns out to be awkward.

    Yes. Actually, when compiling the entire solution at once, it's quite a bit slower than Unity's internal compiler. Not sure why. I've had 3 setups so far:
    1) Unity compiles all non-editor scripts as one project (default Unity setup) - 15 seconds (this is not an accurate comparison since it wasn't actually recompiling ALL scripts, just the stuff in the Scripts folder)
    2) All 30 projects in MonoDevelop compiling with the Unity command line compiler via batch file: 27 seconds
    3) All 30 projects in MonoDevelop compiling with MonoDevelop: 44 seconds

    Additionally, it takes about 4 seconds to merge the files into the DLL.

    So it's a tradeoff. If you're going to be recompiling everything all the time, it is slower, probably partly because it's a bunch of projects now instead of like 4-5 like in Unity's solution. But because I will only usually compile one project at a time, it will only take about a total of 5-6 seconds if its something that needs merging. If it's not a merging class (a utility class) its near instant. Or, if I'm working on loose scripts instead of DLLs, 2-3 seconds because there's so little left in the Scripts folder that needs compiling every time.

    Probably the most awkward part of the whole setup is having to have all the fields and their default assignments in the base class.
     
  19. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    550
    The biggest corcern if it falls down is that you'll lose future work due to scene corruption or something. Hopefully not! But be careful with source control and/or backups.

    I meant putting all the scripts into one (or two) DLLs, rather than building lots of tiny DLLs.

    A little over a year ago I was working on a codebase with a few other developers, building DLLs in Visual Studio, and we evolved to a point where we had one DLL per logical functionality unit, totalling perhaps 40 or 50 DLLs. Solution build times in Visual Studio were then pretty bad - even when few files had changed, it took it a while to convince itself that most of the libraries didn't need rebuilding. Even within Visual Studio, this was bad for unit testing, as the test cycle time was dominted by all these no-op builds.

    One of the developers restructured the solution, breaking it into the smallest number of DLLs that was possible. It was less elegant, but build times were much better with only about 5 DLLs to build (one was agnostic, not using Unity; one used UnityEngine; one used UnityEngine and UnityEditor; another was C# 4.0 with WPF; and there were one or two executables that obviously required separate projects for each).
     
  20. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I'm pretty paranoid about backing things up, so that should help. The fact I have the prefabs and scenes in text format is huge relief because I know that even if things get twisted up, it's at least possible to get them untwisted. Still, the only scenario I can think of that might cause my serialized data to get corrupted is if Unity suddenly decides to change fileIds or guids on one of my classes. I don't know how that would happen, but I will watch out for it since numberkruncher did mention having some prefab corruption when compiling dlls from Visual Studio. (I'm currently working on getting my project over to C#, so that may be an issue for me too.)

    Oh. No, I didn't test that but I can. Anyway, that's what Unity did by default that made my compile times so annoying in the first place, so I don't think I'd have any use for that setup.

    I see. That jives with what I was seeing. The more projects, the slower the compile time. But it works for me because I will only usually want to compile one project at a time. Anyway, I'm curious to know just how much overhead it adds. I'll post my numbers once I do the test.
     
  21. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    @gfoot

    Okay, here are the test numbers compiling 162 .js files as one project:
    MonoDevelop - 14.50 s
    Using Unity's command line - 13 s

    There's a bit of a difference between MD and the command line version, but not too bad when only dealing with 1 project.

    Also, with my numbers in the post 2 up from this one, the "2) All 30 projects in MonoDevelop compiling with the Unity command line compiler via batch file: 27 seconds", there was also the difference that at the time I was compling it this way, I was referencing the actual compiled DLLs and not the other projects. This may have reduced the overhead compared to MD having to check if each project needed to be recompiled, similar to what you mentioned with Visual Studio.

    Command line compiling is faster, but I ended up having too many problems doing it the that way because I was running MonoDevelop at the same time and it kept locking various dlls, usually after switching back and forth between Unity, making me have to restart MD all the time to re-compile. PITA.
     
  22. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I just finished converting my project to C# and I wanted to add compile times:
    163 .cs files, 29 projects (a little bit has changed):
    Rebuild entire solution.

    Visual Studio 2010 Express: 3.5 seconds
    MonoDevelop: 3.5 seconds

    I'm blown away at the compile speed difference between C# and UnityScript. With separate projects, MD compile was 44 seconds for the whole solution. C# is 12.5X faster at compiling than UnityScript! Good lord, even if you ignore all the other reasons for using C#, this one is enough in and of itself to convince me C# is the right choice for any sizable project.

    Edit: THIS IS HUGE On a whim, I just tried moving all my C# scripts back into a new Unity project and not use my precompiled DLLs. To my complete shock, now it only takes Unity 3.5 seconds to compile the C# scripts!!! I did the same with the .js scripts and it took a full 15 seconds to compile! This means that when using Unity in the way it was intended (loose scripts), C# compiles a full 4X FASTER than UnityScript! That's the final nail in the coffin for US as far as I'm concerned. I could have avoided wasting that week+ converting my project to DLLs if I had known this in the first place.
     
    Last edited: Sep 17, 2012
  23. Demigiant

    Demigiant

    Joined:
    Jan 27, 2011
    Posts:
    3,239
    Awesome discovery, thanks for reporting it :)

    I used to compile all my code as DLL (mainly to use all of Visual Studio's power, and because I hate the loose namespace-less Unity way of managing scripts), but finally went back to using loose scripts due to pdb2mdb not supporting Enumerators (and thus Coroutines), sigh.
     
  24. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    I didn't test with Coroutines, but I was getting some IEnumerable errors using the PDB2MDB that came with Unity. I found out how to make my own (going both ways, MDB2PDB too) using Mono.Cecil which you can get here. It was pretty easy with help from here on where to start. I didn't see the error anymore, but I can't say for sure it would work with coroutines.

    Edit: Yoyo has found a better solution to the Coroutine/IEnumerable problem with PDB2MDB. See his post here.
     
    Last edited: Dec 8, 2012
  25. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Convert ALL remaining .js files in the prject to C#. Don't mix JS and C#.
    When I actually got all my scripts and dlls together in my game project, I was seeing a bit worse compile time than I had recorded before -- 6 seconds (probably because my previous tests didn't include Standard Assets and such). I was able to reduce compile times from 6 seconds to 2.75 seconds by converting ALL the .js files in my project to C#, including Standard Assets. Even if I had one lingering .js file remaining, the compile times were not as good. After removing the last .js file, Unity automatically removes all 4 .unityproj projects from the solution. After a restart of Unity, compile times were much faster.

    Note: When converting the Standard Assets to C#, be sure to enable Version Control : Meta Files and Asset Serialization : Force Text (Pro only features). Then copy your standard assets to another folder (make a solution), convert them to C#, delete the .js files BUT keep the *.js.meta files and rename them to *.cs.meta. This way nothing will break in your objects and the default references on the scripts will still be kept.
     
    Last edited: Sep 20, 2012
  26. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    guavaman, would you be willing to share the source for your pdb2mdb replacement? This would make a nice addition to the Unity community wiki. I'm sure anyone working with Visual Studio would be very appreciative. :)
     
  27. Demigiant

    Demigiant

    Joined:
    Jan 27, 2011
    Posts:
    3,239
    +1

    That would be awesome :) (I tried following your - guavaman - links to make the modification, but I always postponed it since it didn't look that easy to me)
     
  28. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    EDIT: Don't bother with this code! Yoyo has found a MUCH simpler solution. See this post.

    Sure. But my implementation probably isn't exactly suitable for the Wiki. It's not very robust and doesn't do much error checking or anything. Also, last time I used it, as I recall, there was some other issue with the conversion not working with ILMerge. It's been a while since I messed with it because I pretty much gave up using external DLLs because of issues related to Unity not handling MonoBehaviour inheritance from DLLs properly.

    You need to reference Mono.Cecil.MDB and Mono.Cecil.PDB for it to compile. Those can be downloaded from here -- the MDB and PDB sources are in the Symbols folder of the zip.

    Code (csharp):
    1. // Program.cs
    2. using System;
    3. using System.Collections.Generic;
    4. using System.Linq;
    5. using System.Text;
    6. using System.IO;
    7. using Mono.Cecil;
    8. using Mono.Cecil.Cil;
    9. using Mono.Cecil.Pdb;
    10. using Mono.Cecil.Mdb;
    11. using Mono.CompilerServices.SymbolWriter;
    12.  
    13. namespace MDB2PDB {
    14.     class Program {
    15.         static void Main(string[] args) {
    16.             if (args.Length < 1 || args.Length > 2) {
    17.                 Console.WriteLine("Usage: Assembly.dll (w/ Assembly.dll.mdb in same folder) [mode = mdb/pdb]");
    18.                 Console.WriteLine("Output: Assembly.dll.pdb");
    19.                 Exit();
    20.             }
    21.  
    22.             PDBConverter pbdConverter = new PDBConverter(args);
    23.  
    24.             Log.CloseFile();
    25.             //Pause();
    26.         }
    27.  
    28.         static internal void CriticalError() {
    29.             Log.CriticalError();
    30.             Log.CloseFile();
    31.             Pause();
    32.             Exit();
    33.         }
    34.  
    35.         static public void Exit() {
    36.             Environment.Exit(1);
    37.         }
    38.  
    39.         static private void Pause() {
    40.             Console.WriteLine("\nPress any key to exit...");
    41.             Console.Read();
    42.         }
    43.     }
    44.  
    45.     public class PDBConverter {
    46.         private enum FileType { MDB, PDB };
    47.         private string fileExt = ".mdb";
    48.         private string targetExt = ".pdb";
    49.         private ConvertMode convertMode = ConvertMode.MDB2PDB;
    50.  
    51.         private enum ConvertMode { MDB2PDB, PDB2MDB };
    52.  
    53.         public PDBConverter(string[] args) {
    54.             bool isPath = false;
    55.             string fullPath = Path.GetFullPath(args[0]);
    56.             string mode = "";
    57.             if(args.Length > 1)
    58.                 mode = args[1].ToLower();
    59.            
    60.             // Determine if arg is a directory or file
    61.             if (!File.Exists(fullPath) || fullPath.EndsWith(Path.DirectorySeparatorChar)) {
    62.                 if(Directory.Exists(fullPath))
    63.                     isPath = true;
    64.             }
    65.  
    66.             if (mode == "pdb" || mode == ".pdb") {
    67.                 convertMode = ConvertMode.PDB2MDB;
    68.                 fileExt = ".pdb";
    69.                 targetExt = ".mdb";
    70.             }
    71.  
    72.             if (isPath)
    73.                 ProcessPath(fullPath);
    74.             else {
    75.                 ProcessFile(fullPath);
    76.             }
    77.         }
    78.  
    79.         private void ProcessPath(string path) {
    80.             if (!Directory.Exists(path)) {
    81.                 Log.Error("Directory " + path + " does not exist!");
    82.                 Program.CriticalError();
    83.             }
    84.  
    85.             string[] files = Directory.GetFiles(path, "*.dll", SearchOption.TopDirectoryOnly); // no recursive
    86.             if (files.Length == 0) { // no files found
    87.                 Log.Error("No DLL files found in " + path);
    88.                 Program.CriticalError();
    89.             }
    90.  
    91.             // Process each dll file
    92.             for (int i = 0; i < files.Length; i++) {
    93.                 ProcessFile(files[i]);
    94.             }
    95.         }
    96.  
    97.         private void ProcessFile(string assemblyPathAndFile) {
    98.             if (convertMode == ConvertMode.MDB2PDB)
    99.                 ProcessFileMDB2PDB(assemblyPathAndFile);
    100.             else
    101.                 ProcessFilePDB2MDB(assemblyPathAndFile);
    102.         }
    103.  
    104.         private void ProcessFileMDB2PDB(string assemblyPathAndFile) {
    105.             if (!File.Exists(assemblyPathAndFile)) { // make sure file exists
    106.                 Log.Error(assemblyPathAndFile + " does not exist!");
    107.                 return;
    108.             }
    109.            
    110.             if(!assemblyPathAndFile.ToLower().EndsWith(".dll")) {
    111.                 Log.Error(assemblyPathAndFile + " is not a DLL file! Skipping.");
    112.                 return;
    113.             }
    114.  
    115.             string assemblyFileName = Path.GetFileName(assemblyPathAndFile);
    116.             string path = Path.GetDirectoryName(assemblyPathAndFile);
    117.  
    118.             if (!File.Exists(assemblyPathAndFile + fileExt)) { // no mdb/pdb file found, just skip it
    119.                 Log.Write(fileExt + " file not found for assembly " + assemblyPathAndFile + ". Skipping.");
    120.                 return;
    121.             }
    122.  
    123.             // Open files and save
    124.             try {
    125.                 var readerParameters = new ReaderParameters { ReadSymbols = true, SymbolReaderProvider = new MdbReaderProvider() };
    126.                 var assemblyDefinition = AssemblyDefinition.ReadAssembly(assemblyPathAndFile, readerParameters);
    127.  
    128.                 Guid guid = Guid.NewGuid();
    129.                 string tempAssembly = path + Path.DirectorySeparatorChar + guid.ToString();
    130.  
    131.                 var writerParameters = new WriterParameters { WriteSymbols = true, SymbolWriterProvider = new PdbWriterProvider(), };
    132.                 assemblyDefinition.Write(tempAssembly, writerParameters); // write temp files
    133.  
    134.                 File.Delete(tempAssembly); // delete the temp assembly file
    135.                 FileInfo tempAssemblyInfo = new FileInfo(tempAssembly + targetExt);
    136.  
    137.                 string pdbTargetFile = path + Path.DirectorySeparatorChar + Path.GetFileNameWithoutExtension(assemblyFileName) + targetExt;
    138.                 if (File.Exists(pdbTargetFile)) // delete existing pdb file
    139.                     File.Delete(pdbTargetFile);
    140.                 tempAssemblyInfo.MoveTo(pdbTargetFile); // rename temp pdb to final Assembly.pdb
    141.  
    142.                 // Match DLLs timestamp on PDB
    143.                 //FileInfo pdbInfo = new FileInfo(pdbTargetFile);
    144.                 //FileInfo dllInfo = new FileInfo(assemblyPathAndFile);
    145.                 //pdbInfo.CreationTime = dllInfo.CreationTime;
    146.                 //pdbInfo.LastWriteTime = dllInfo.LastWriteTime;
    147.                 //pdbInfo.LastAccessTime = dllInfo.LastAccessTime;
    148.  
    149.                 Log.Success();
    150.             }
    151.             catch(Exception e) {
    152.                 Log.Error(e.Message);
    153.                 Program.CriticalError();
    154.             }
    155.         }
    156.  
    157.         private void ProcessFilePDB2MDB(string assemblyPathAndFile) {
    158.             if (!File.Exists(assemblyPathAndFile)) { // make sure file exists
    159.                 Log.Error(assemblyPathAndFile + " does not exist!");
    160.                 return;
    161.             }
    162.  
    163.             if (!assemblyPathAndFile.ToLower().EndsWith(".dll")) {
    164.                 Log.Error(assemblyPathAndFile + " is not a DLL file! Skipping.");
    165.                 return;
    166.             }
    167.  
    168.             string assemblyFileName = Path.GetFileNameWithoutExtension(assemblyPathAndFile);
    169.             string path = Path.GetDirectoryName(assemblyPathAndFile);
    170.             string sourceDebugFile = path + Path.DirectorySeparatorChar + assemblyFileName + fileExt;
    171.  
    172.             if (!File.Exists(sourceDebugFile)) { // no mdb/pdb file found, just skip it
    173.                 Log.Write(fileExt + " file not found for assembly " + assemblyPathAndFile + ". Skipping.");
    174.                 return;
    175.             }
    176.  
    177.             // Open files and save
    178.             try {
    179.                 var readerParameters = new ReaderParameters { ReadSymbols = true, SymbolReaderProvider = new PdbReaderProvider() };
    180.                 var assemblyDefinition = AssemblyDefinition.ReadAssembly(assemblyPathAndFile, readerParameters);
    181.  
    182.                 Guid guid = Guid.NewGuid();
    183.                 string tempAssembly = path + Path.DirectorySeparatorChar + guid.ToString();
    184.  
    185.                 var writerParameters = new WriterParameters { WriteSymbols = true, SymbolWriterProvider = new MdbWriterProvider(), };
    186.                 assemblyDefinition.Write(tempAssembly, writerParameters); // write temp files
    187.  
    188.                 File.Delete(tempAssembly); // delete the temp assembly file
    189.                 FileInfo tempAssemblyInfo = new FileInfo(tempAssembly + targetExt);
    190.  
    191.                 string targetFile = path + Path.DirectorySeparatorChar + Path.GetFileName(assemblyPathAndFile) + targetExt;
    192.                 if (File.Exists(targetFile)) // delete existing pdb file
    193.                     File.Delete(targetFile);
    194.                 tempAssemblyInfo.MoveTo(targetFile); // rename temp mdb to final Assembly.dll.mdb
    195.  
    196.                 // Match DLLs timestamp on MDB
    197.                 //FileInfo debugInfo = new FileInfo(targetFile);
    198.                 //FileInfo dllInfo = new FileInfo(assemblyPathAndFile);
    199.                 //debugInfo.CreationTime = dllInfo.CreationTime;
    200.                 //debugInfo.LastWriteTime = dllInfo.LastWriteTime;
    201.                 //debugInfo.LastAccessTime = dllInfo.LastAccessTime;
    202.  
    203.                 //MethodSymbols sym = new MethodSymbols(assemblyDefinition.MetadataToken);
    204.  
    205.                 Log.Success();
    206.             }
    207.             catch (Exception e) {
    208.                 Log.Error(e.Message);
    209.                 Program.CriticalError();
    210.             }
    211.         }
    212.     }
    213. }
    214.  
    Code (csharp):
    1. // Log.cs
    2. using System;
    3. using System.Collections.Generic;
    4. using System.Linq;
    5. using System.Text;
    6. using System.IO;
    7.  
    8. namespace MDB2PDB {
    9.     static public class Log {
    10.         static public int errors = 0;
    11.         static public int successes = 0;
    12.         static private StreamWriter sw;
    13.         static private List<string> failures = new List<string>();
    14.  
    15.         static public void Error(string s) {
    16.             errors++;
    17.             Console.ForegroundColor = ConsoleColor.Red;
    18.             Console.WriteLine("ERROR: " + s);
    19.             Console.ResetColor();
    20.             if (sw != null) sw.WriteLine("ERROR: " + s);
    21.         }
    22.  
    23.         static public void Success() {
    24.             successes++;
    25.         }
    26.  
    27.         static public void Write(string s) {
    28.             Console.WriteLine(s);
    29.             if (sw != null) sw.WriteLine(s);
    30.         }
    31.  
    32.         static public void OpenFile(string logFile) {
    33.             try {
    34.                 sw = new StreamWriter(logFile);
    35.             }
    36.             catch (Exception e) {
    37.                 Console.WriteLine("Could not create log file!");
    38.                 Console.WriteLine(e.Message);
    39.             }
    40.         }
    41.  
    42.         static public void CloseFile() {
    43.             ReportSuccesses();
    44.             ReportFailures();
    45.             ReportErrors();
    46.  
    47.             if (sw != null)
    48.                 sw.Close();
    49.         }
    50.  
    51.         static public void Failed(string s) {
    52.             failures.Add(s);
    53.         }
    54.  
    55.         static public void CriticalError() {
    56.             Console.ForegroundColor = ConsoleColor.Red;
    57.             Log.Write("\nCritical error!\n");
    58.             Console.ResetColor();
    59.         }
    60.  
    61.         static private void ReportFailures() {
    62.             if (failures.Count == 0) return;
    63.  
    64.             Log.Write("\n" + failures.Count + " failures:");
    65.             foreach (string s in failures)
    66.                 Log.Write(s);
    67.         }
    68.  
    69.         static private void ReportErrors() {
    70.             if (errors == 0) {
    71.                 Console.ForegroundColor = ConsoleColor.Green;
    72.                 Log.Write("No errors encountered!");
    73.                 Console.ResetColor();
    74.                 return;
    75.             }
    76.  
    77.             Console.ForegroundColor = ConsoleColor.Red;
    78.             Log.Write("\n**** FAILED! ****");
    79.             Log.Write(errors + " errors encountered!");
    80.             Console.ResetColor();
    81.         }
    82.  
    83.         static private void ReportSuccesses() {
    84.             if (successes == 0) return;
    85.  
    86.             string plural = "";
    87.             if (successes > 1)
    88.                 plural = "s";
    89.  
    90.             Log.Write("\nSuccessfully converted " + successes + " file" + plural + "!");
    91.         }
    92.     }
    93. }
    94.  
    Hope it helps. If you find it useful and clean it up, please repost the code here.
     
    Last edited: Dec 8, 2012
    SugoiDev and PDZ like this.
  29. Demigiant

    Demigiant

    Joined:
    Jan 27, 2011
    Posts:
    3,239
    Thanks a lot guavaman :) Will try that soon, and surely repost it in case I should clean it up.
     
  30. Maxii

    Maxii

    Joined:
    May 28, 2012
    Posts:
    45
    Wow! A potential fix for pdb2mdb! I can't wait! :p
     
  31. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Fantastic, thanks for the code! (Wish I'd been paying more attention to this thread, I just spent this morning downloading Visual Studio 2012 Express, getting the Mono.Cecil nuget package, and building my own converter ...)

    My converter is even less robust than yours, but the guts of it seem to be the same (read pdb symbols, write temp assembly with mdb symbols, move mdb symbol file to correct name). My mdb generator works for an assembly with just one source file, but if I add a second source file then when I step into code I end up in the wrong file -- possibly MonoDevelop is expecting an older version of the .mdb file format? Have you run into anything like this?

    Update: I downloaded Mono.Cecil 0.9.4 and rebuilt it from the source. I am now able to step into an assembly with two sources files, but my "real" DLL still doesn't work -- either doesn't find source at all, or jumps into random code. I chose 0.9.4 because this is the version of Mono.Cecil that comes with the MonoDevelop that comes with Unity, and it seemed like a good idea to match versions. I also tried with the "master.archive" version of Mono.Cecil (linked in guavaman's post above) but that didn't help either.
     
    Last edited: Dec 5, 2012
  32. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Sorry you're running into all these problems. Unfortunately its been a while since I worked on any of this since I abandoned using DLLs because of Unity's issues with inherited Monobehaviours, so I can't remember clearly what I did and didn't do and what worked or not. I seem to recall having some problems getting certain DLLs to jump to line numbers when double clicking on errors. I'm not sure if I found a solution. As far as I remember, though, the line numbers were at least accurate.
     
    Last edited: Dec 5, 2012
  33. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Oh well, thanks again for your effort on this. At present I'm thinking of moving all non-MonoBehaviour code into external assemblies, so the inheritance issue isn't a showstopper for me right now. I've met Jb Evain a couple of times, the author of Mono.Cecil (and UnityVS), so perhaps I'll see if he knows what my issue might be.
     
  34. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Quick update -- if I build code with unity\3.5.6\Editor\Data\MonoBleedingEdge\lib\mono\2.0\mcs.exe (*not* Mono\lib\mono\2.0\gmcs.exe, which just falls over with an unhandled exception) then I get a dll and a matching mdb which supports source level debugging in MonoDevelop. I'm going to see if I can set up a .csproj file for code editing in Visual Studio with an external build via mcs. Will post my results ...
     
  35. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    This seems like a much simpler approach. Instead of messing around with Mono.Cecil, I can just use mcs.exe (which comes with Unity) to generate an assembly from a custom Build target. This gives me nice editing and code navigation in Visual Studio, with source-level debugging in MonoDevelop, which is what I wanted. (Well, I'd rather have source-level debugging in Visual Studio ...) For some reason gmcs.exe didn't work, I had to use mcs.exe out of the MonoBleedingEdge folder.

    Here's my complete csproj file for reference ...
    Code (csharp):
    1. <?xml version="1.0" encoding="utf-8"?>
    2. <Project ToolsVersion="4.0" InitialTargets="" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    3.   <!-- Common Properties -->
    4.   <PropertyGroup>
    5.     <!-- Default configuration if not specified. -->
    6.     <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    7.     <!-- Default platform if not specified, or if set to "web" default. -->
    8.     <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    9.     <Platform Condition=" '$(Platform)' == 'web' ">AnyCPU</Platform>
    10.     <ProductVersion>8.0.30703</ProductVersion>
    11.     <SchemaVersion>2.0</SchemaVersion>
    12.     <ProjectGuid>{61A89EEC-36B9-49ED-8431-D6A7DBDD9EA7}</ProjectGuid>
    13.     <OutputType>Library</OutputType>
    14.     <AppDesignerFolder>Properties</AppDesignerFolder>
    15.     <RootNamespace>MyNamespace</RootNamespace>
    16.     <AssemblyName>MyAssembly</AssemblyName>
    17.     <TargetFrameworkVersion>v3.5</TargetFrameworkVersion>
    18.     <TargetFrameworkProfile>
    19.     </TargetFrameworkProfile>
    20.     <FileAlignment>512</FileAlignment>
    21.     <!-- Look up Unity install folder, for locating managed assembly references. -->
    22.     <UnityInstallFolder>$([System.IO.Path]::GetDirectoryName($(registry:HKEY_CURRENT_USER\Software\Unity Technologies\Unity Editor 3.x\Location)))</UnityInstallFolder>
    23.     <IsUnityBuild Condition="Exists('$(MSBuildProjectDirectory)\..\..\unity')">true</IsUnityBuild>
    24.     <UnityGamePluginsDirectory Condition="$(IsUnityBuild)">$(MSBuildProjectDirectory)\..\..\unity\Assets\Plugins</UnityGamePluginsDirectory>
    25.   </PropertyGroup>
    26.   <!-- Debug Properties -->
    27.   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    28.     <PlatformTarget>AnyCPU</PlatformTarget>
    29.     <DebugSymbols>true</DebugSymbols>
    30.     <DebugType>full</DebugType>
    31.     <Optimize>false</Optimize>
    32.     <OutputPath>bin\Debug\</OutputPath>
    33.     <DefineConstants>DEBUG;UNITY_3_5</DefineConstants>
    34.     <ErrorReport>prompt</ErrorReport>
    35.     <WarningLevel>4</WarningLevel>
    36.     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    37.   </PropertyGroup>
    38.   <!-- Release Properties -->
    39.   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    40.     <PlatformTarget>AnyCPU</PlatformTarget>
    41.     <DebugType>pdbonly</DebugType>
    42.     <Optimize>true</Optimize>
    43.     <OutputPath>bin\Release\</OutputPath>
    44.     <DefineConstants>UNITY_3_5</DefineConstants>
    45.     <ErrorReport>prompt</ErrorReport>
    46.     <WarningLevel>4</WarningLevel>
    47.   </PropertyGroup>
    48.   <PropertyGroup>
    49.     <StartupObject />
    50.   </PropertyGroup>
    51.   <!-- Standard stuff helps with reference plumbing and such. Even though we're not using the
    52.       standard Build target, we'll keep this anyway. -->
    53.   <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
    54.   <!-- Source -->
    55.   <ItemGroup>
    56.     <Compile Include="**\*.cs" />
    57.   </ItemGroup>
    58.   <!-- References -->
    59.   <ItemGroup>
    60.     <Reference Include="System" />
    61.     <Reference Include="UnityEngine">
    62.       <HintPath>$(UnityInstallFolder)\Data\Managed\UnityEngine.dll</HintPath>
    63.       <Private>False</Private>
    64.     </Reference>
    65.   </ItemGroup>
    66.   <Target Name="Build">
    67.     <ItemGroup>
    68.       <RspFile Include="$(MSBuildProjectDirectory)\mcs.rsp" />
    69.       <RspOptions Include="-reference:UnityEngine" />
    70.       <!-- Note: need to escape the semi-colon, otherwise we get multiple lines. -->
    71.       <RspOptions Include="-define:$(DefineConstants.Replace(';', '%3B'))" Condition=" '$(DefineConstants)' != '' " />
    72.       <RspOptions Include="-debug+" Condition=" '$(DebugSymbols)' == 'true' " />
    73.       <RspOptions Include="-optimize+" Condition=" '$(Optimize)' == 'true' " />
    74.       <RspOptions Include="-warnaserror+" Condition=" '$(TreatWarningsAsErrors)' == 'true' " />
    75.       <RspOptions Include="-lib:$(UnityInstallFolder)\Data\Managed" />
    76.       <RspOptions Include="-out:$(OutputPath)$(AssemblyName).dll" />
    77.       <RspOptions Include="-platform:$(Platform.ToLower())" />
    78.       <RspOptions Include="-target:$(OutputType.ToLower())" />
    79.     </ItemGroup>
    80.     <Message Text="Writing @(RspFile) ..." />
    81.     <WriteLinesToFile File="@(RspFile)" Lines="@(RspOptions);@(Compile)" Overwrite="true" Encoding="Unicode" />
    82.     <Message Text="Building with mcs.exe ..." />
    83.     <MakeDir Directories="$(OutputPath)" />
    84.     <Exec Command="$(UnityInstallFolder)\Data\MonoBleedingEdge\lib\mono\2.0\mcs.exe @@(RspFile)" />
    85.     <Delete Files="@(RspFile)" />
    86.     <CallTarget Targets="AfterUnityBuild" Condition="$(IsUnityBuild)" />
    87.     <Message Text="Built $(OutputPath)$(AssemblyName).dll" Importance="High" />
    88.   </Target>
    89.   <Target Name="AfterUnityBuild" Condition="$(IsUnityBuild)">
    90.     <ItemGroup>
    91.       <TargetFiles Include="$(OutputPath)$(AssemblyName).*" />
    92.     </ItemGroup>
    93.     <Copy SourceFiles="@(TargetFiles)" DestinationFolder="$(UnityGamePluginsDirectory)" />
    94.   </Target>
    95. </Project>
     
  36. Demigiant

    Demigiant

    Joined:
    Jan 27, 2011
    Posts:
    3,239
    Mhmm great things happening here, thanks to you both :) Maybe you should create a dedicated thread about this, so everybody could use it as a reference.
     
  37. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Sigh, after all that, Jb Evain (author of pdb2mdb, Mono.Cecil, and UnityVS) emailed me to say I should be using:
    Editor\Data\MonoBleedingEdge\lib\mono\4.0\pdb2mdb.exe
    and not:
    Editor\Data\Mono\lib\mono\2.0\pdb2mdb.exe

    And it turns out the reason my own pdb2mdb converter wasn't working is because it generates a new output assembly which I was discarding, and then trying to use the generated mdb with a mismatched assembly.

    Oh well, I learned a bunch about Mono.Cecil and MSBuild, so it wasn't entirely wasted effort. I'll tidy up my build file and post a new thread on how to edit in Visual Studio and debug in MonoDevelop.
     
  38. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    So using this version of pdb2mdb.exe doesn't throw any errors with coroutines and IEnumerables and such? o_O
    When you start the new thread, can you post a link in here in case someone stumbles on this thread from a search engine? Thanks!
     
  39. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
  40. Maxii

    Maxii

    Joined:
    May 28, 2012
    Posts:
    45
    Guavaman, above you referred to 'Unity's issues with inherited MonoBehaviours in external .dlls'. I've recently encountered the editor no longer showing my MonoBehaviours in .dlls once I started inheriting from my own MonoBehaviourBase class. However, I can't seem to find any discussion about the subject although I know I've seen some before. Can you perhaps point me to a thread or two?

    Your work with Yoyo on pdb2mdb was OUTSTANDING! I've been banging my head against that brick wall for 4 months now, and, all of a sudden it's solved! Can't believe no one from Unity mentioned it ...

    Thanks for any help you can provide.
     
  41. guavaman

    guavaman

    Joined:
    Nov 20, 2009
    Posts:
    5,605
    Here is the initial post where I discovered the bug: http://forum.unity3d.com/threads/14...-MonoDevelop?p=1022960&viewfull=1#post1022960
    There was also some discussion at UnityAnswers here: http://answers.unity3d.com/questions/240985/subclassed-monobehavior-in-external-dlls-not-recog.html

    For a while I was merging my DLLs with ILMerge so Unity wouldn't choke on the inherited Monobehaviours, which is why I thought I had to try to try to rewrite PDB2MDB in the first place. But the clunky workflow eventually forced me back to using a bunch of loose .cs scripts in the end.

    Unfortunately there isn't a solution. Unity support acknowleged it was a bug, but I haven't heard anything on a fix. :(

    My bug report is as follows (contains some additional details such as how to make it crash and how serialized data isn't saved):

    --------------------------

    1) What happened

    a) Failure to recognize MonoBehaviour classes
    b) Reproducible crash

    Working with DLLs:
    Unity is unable to recognize classes extended two steps from MonoBehaviour if each class resides in a different DLL. Tested in both C# and UnityScript.

    Example:
    AssemblyA.dll
    class BaseClass : MonoBehaviour { void Start() {} }

    AssemblyB.dll (compiled w/ reference to AssemblyA.dll)
    class ExtendedClass : BaseClass { void Start() {} }

    Import both DLL files into Unity.

    AssemblyA shows up in the inspector with a dropdown which includes BaseClass. BaseClass can be dragged onto gameobjects, the component on the gameobject shows up in the inspector as BaseClass.

    AssemblyB shows up in the inspector with no dropdown.
    The only way to add the contained class to a gameobject is with a script: gameObject.AddComponent(ExtendedClass)

    However, the component on the gameobject shows up only as Script. Furthermore, when saved, the .scene file contents shows no GUID or type recorded for the MonoBehaviour:
    m_Script: {fileID: 956415588} // GUID and type are missing

    Then when you try to make a Prefab out of the object, the inspector shows "Type Mismatch" in the field next to Script. When viewing contents of the .prefab file, the line is now: m_Script: {fileID: 0}. It completely lost the reference. Finally, when you hit play in the editor, it crashes. (There are Debug.Log() calls in Start() in both classes.)

    Fatal Error
    The file "Memory Stream" is corrupted! Remove it and launch Unity again.
    [Position out of bounds!]

    2) How can we reproduce it using the example you attached

    1) Open the Unity project included and open TestScene.unity inside Unity.

    2) Look in the Project window and look under Assemblies at Base and Extended. Base has a dropdown showing BaseClass inside it. Extended has no dropdown even though there is a class inside called ExtendedClass which inherits from BaseClass (and therefore ultimately inherits from MonoBehaviour).

    3) Add ExtendedClass to a gameobject using a script:
    Go to the menu bar and run Debug -> Add ExtendedClass component to MonoTestObject
    It should add the component to MonoTestObject and select the object for you.
    Look at the inspector and you will see a component was added, but it only shows up as Script.

    4) Save the scene and open TestScene.unity in a text editor. Look for the MonoBehaviour entry that has a serialized string "variable in ExtendedClass!", then look for the m_Script: {} field. You will see there is no GUID or type recorded.

    5) Go back to the editor and create a prefab out of MonoTestObject. Look at MonoTestObject in the scene in the inspector and you will see the Script component now shows "Type Mismatch" in the field next to Script (near the little paper icon).

    6) Open the new prefab file in a text editor and you will notice the field m_Script: {fileID: 0} now says 0 and it has lost any connection to the original class.

    7) Press play in the editor and Unity will crash.


    Additional information:

    My use case for this scheme:
    (in compile order)
    Utility classes
    Base MB classes (extend MonoBehaviour, contain all field/property definitions, stub methods, reference to Utility)
    Extended MB classes (extensions of base classes, contain all methods [game code], attached to game objects, separated into various dll's by category, reference to Utility and Base)

    I separated the MonoBehaviours into base and extended so I could leave my existing code and pass around the base classes without any of the extended classes being dependent on each other. I could then split up my extended mb classes (where most of the game code is) into smaller chunks so compiling would be quick for whatever I'm working on at the moment.

    Without Unity being able to handle these extended MonoBehaviours, I cannot use this structure. Once a project reaches a certain size, recompiling everything with every little change is a huge time sink. The segmented DLL scheme would allow one to recompile only a few classes at a time.
     
    Last edited: Dec 17, 2012
  42. Maxii

    Maxii

    Joined:
    May 28, 2012
    Posts:
    45
    Thank you for bringing me up to speed. This is very vexing, especially the part about Unity's lack of responsiveness to fixing bugs for serious developers. I'm wondering:

    Has anyone invested in creating a single standin MonoBehaviour pattern (the classes of which would show in the .dll) that simply maps ALL calls to the real meaty class in the .dll that inherits from a MonoBehaviourBase class (that wouldn't show in the .dll)? Each standin would hold a single MonoBehaviour type reference to it's meaty cousin? The standin could then be dragged using the editor... I'm not at all sure this is feasible, but I imagine I'm not the first person to think of it, at least conceptually.
     
  43. C4MProdDev

    C4MProdDev

    Joined:
    Jul 9, 2009
    Posts:
    107
  44. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    6,221
    Compile time is 2X faster on a SSD.
     
  45. User340

    User340

    Joined:
    Feb 28, 2007
    Posts:
    3,001
    All you have to do is put as many scripts as possible into Plugins folder, that's it!
     
    laurentlavigne likes this.
  46. JamesLeeNZ

    JamesLeeNZ

    Joined:
    Nov 15, 2011
    Posts:
    5,616
    Deleting your cache folder helps.
     
  47. User340

    User340

    Joined:
    Feb 28, 2007
    Posts:
    3,001
    What is "cache folder"?
     
  48. Blastom

    Blastom

    Joined:
    Aug 13, 2013
    Posts:
    24
    Amazing :) I'm converting a project to be hot upgradable on Android. And this saved my life by denying a total reconstruction on prefabs :)
     
  49. DrKucho

    DrKucho

    Joined:
    Oct 14, 2013
    Posts:
    140
    hey i would like to say thanks to @guavaman for this post and all the detailed info, i was about to get a new computer but after reading this post i decided to convert all my scripts to C# as my compiling time was of 50 seconds, my computer is a pretty decent i7 but i guess i had too many scripts and also i was pushing javascript to the limits, using corrutines all the way and a lot of stuff that was making the compiler to work so hard to know exactly how to compile, as the more i went trough that conversion to C the more i was realising how forgiving javascript is compared to C and how much the compiler has to figure out what to do , i guess all that forgiving language characteristics is forcing the compiler to read the code several times till it understands what is what.

    conversion is done and now compile times is about 2 seconds ( it was 50!!!) the only thing i could not understand is the steps guavaman describes to don't loose serialisation and avoid your scenes and prefabs to break , i read it a few times and i was imposible to understand the steps, what i did , and i think it works perfectly is this.

    1.- backup your project completely and make sure you don't touch this one!
    2.- move all javascript files to a different folder out of the project( so unity won't try to convert them every time you edit the new C code)
    3.- do C conversion , let unity to help giving you all errors etc, once done your scenes and prefabs will be broken
    4.- use your backup (which has all code in javascript working alright) to create another copy of the whole project (backup2)
    5.- open unity and open backup2 project
    6.- go Edit-ProjectSettings-Editor and activate visible meta file only, wait for unity to complete the task
    7.- close unity
    8.- use your operating system go to your scripts folder of backup2 and delete all javascript files , but keep their meta files that are now visible
    9.- copy to the same folder all C# files from the project you did the conversion in
    10.- rename all *.js.meta files to *.cs.meta so what was MyScript.js.meta now is MyScript.cs.meta (use a renaming tool recommended)
    11.- open unity and let it recognise the new files, they will be associated to the renamed meta files cause they share name now

    everything should work , unless i am making mistake ... it seems to work here.
     
  50. JLJac

    JLJac

    Joined:
    Feb 18, 2014
    Posts:
    36
    Hi! I have a C# project that's compiling too slow, and am considering solution #2 here.

    My project is extremely interdependent and tangled, meaning there is no possible division I can make where not every part is referring to classes and calls in every other part.

    I have a hunch that under such circumstances, it's not even worth it to try doing the multiple-dll thing. Could someone who knows a bit more about dll's tell me if that assumption is correct?

    Thank you!