System.Xml in the web player

Hi everyone, new Unity pro user here, just started a new project.

I'm having some trouble with the usage of System.Xml on the web player.
For my project, I need to read a big XML file (over 4MB in size!) from an online server, parse it and use the data to build geometries.
I tried first using some of the XML Parser scripts other people posted on this forum, and they work fine, however the parsing is amazingly slow (minutes for such huge XML files).
Then I tried importing System.XML and it's amazingly fast! Parses in less than a second the same files! The code works great on standalone Windows, Mac and Android (didn't try iPhone yet). However the web player seems to ignore LoadXml()

So if I do something like

Code (csharp):

1.  
2.  
3. import System.Xml;
4.  
5.  
6. class XmlDataLoader
7. {  
8.     var GuiData:String;
9.     var www:WWW;
10.     var activeXml:XmlDocument; 
11.     var xmlData:String = "";
12.     var guiText:String = "";   
13.     var XmlIsLoaded:boolean = false;
14.  
15.    
16.    
17.     function LoadObjectXML(xml:String)
18.     {          
19.         XmlIsLoaded = false;
20.                
21.         www = new WWW(xml);    
22.     }
23.    
24.     function OnGUI ()
25.     {
26.         var textrect = Rect (10, 10, 220, 150);
27.         if (GuiData) GUI.TextArea( textrect, GuiData );    
28.     }
29.    
30.     function Update()
31.     {
32.         if (this.www  !XmlIsLoaded)
33.         {          
34.             GuiData = "Downloading XML... "+(Mathf.Round(this.getProgress()*1000)/10).ToString()+"% done"+"\n"+evLoader.guiText;       
35.             this.Update();
36.         }
37.         if (!XmlIsLoaded  www.isDone )
38.         {
39.             XmlIsLoaded = true;
40.             xmlData = www.text;            
41.            
42.             buildObjectsFromData();
43.         }
44.        
45.     }
46.        
47.     function getProgress()
48.     {
49.         if(www)
50.         {
51.             return www.progress;
52.         }
53.         else
54.         {
55.             return -1;
56.         }
57.     }
58.    
59.        
60.     function buildObjectsFromData()
61.     {          
62.         guiText+="Parsing...";  
63.         activeXml = new XmlDocument();
64.        
65.         if (xmlData)
66.         {          
67.             guiText+= "\nThe file exists and has "+xmlData.length+" characters\n";         
68.             activeXml.LoadXml(xmlData);
69.         }
70.         else
71.         {
72.             guiText+="\nNo XML data!!!!";
73.         }
74.        
75.         guiText+="\nParsing complete";
76.        
77.         //Then use the data for whatever
78.    
79.        
80.     }
81. }
82.    
83.  
84.  

(This is based on my actual code, but not all of it nor *exactly* like it. Serves for illustration of the problem only)

If then I create a XmlDataLoader object and make it load an XML posted online trough LoadObjectXML(url), it should:
-Load it, displaying a GUI text indicating the progress
-Call buildObjectsFromData(). Display "Parsing...". Validate the existence of the string correspondent to the XML, and display its number of characters (this is for debug purposes, for me to be sure the XML was really loaded). Then parse the XML. Then display "Parsing complete". Then use the data

All of this works fine in every build platform *except* the web player. It gets to verify the existence of the XML string data, displaying its number of characters (in test versions I even made it display the first 200 characters, just to make sure it was really reading the file correctly!)... And then... Nothing. It never gets to display "Parsing complete".
However I have a few objects in the scene, and a camera controller, and it's possible to keep flying around. So it's not frozen! Only it never gets to parse.


Anyone has a clue on what may be happening?

Thanks!

 

The reason why people are using the amazingly slow parser is that including System.Xml in the web player makes it amazingly big. You're seeing these results because we quite simply removed System.Xml from the webplayer.

If you need to run data loading fast, why then use XML in the first place?

 

AngryAnt,

Including System.Xml in the web player increased its size by ~320Kb, which is fine (even at dial-up speeds it would take much less time to download it than the extra parsing time with the alternative parser). I understand you removed it from the player, that's fine. But if I include it in my build, should't it be able to parse my files?
It just happens that I'm developing a scientific visualization tool, and the raw data is compiled to XML. No way around that, sorry. I wish.

So let me ask the question again, in another way. Can I use System.Xml in the webplayer at all?

 

Yes, you can use System.Xml in the web player. I do at work with version 2.6.1.

Although, I do not use the XmlDocument.LoadXml method; I use XmlTextReader and XmlSerializer. Perhaps its an issue with XmlDocument and that particular method?


EDIT: Maybe you can use its Load method instead and pass in a TextReader or XmlReader or a the XML as a stream:

http://msdn.microsoft.com/en-us/library/system.xml.xmldocument.load(VS.90).aspx

 

Hey FizixMan,

Thanks for your input! Just tried your suggestion of using the Load method with a XmlTextReader

Code (csharp):

1.  
2. var textReader = new XmlTextReader(new StringReader(xmlData));
3. activeXml.Load(textReader);

Again it works great on the standalone player, but not on the web. I guess there's some problem with XmlDocument on the web player.
Will try XmlSerializer next

 

its not xmldocument but what it relies on for load and thats system.net httprequest which is only present on the standalone windows / osx build and not usable anywhere else, neither webplayer nor iOS has it.

WWW + XMLSerialize will work fine on all platforms.

 

This sucks! I guess with XmlSerializer I can't simply parse an XML file and retrieve the nodes relevant for the application...
It would be great to have the choice of using System.Xml in the web player if we wish so, even if it "bloats" the application.

I never used XmlSerializer before (n00b, I know) but from the examples read and simple tests done over the last few hours it seems inappropriate for what I want. How do I get it to ignore tags not defined in my type class? I keep getting an error message.

 

You can use the various attributes such as XmlIgnore, XmlAttribute, or XmlElement, etc.

Here's a class I used in Unity a while ago (and would be very happy to go back and rewrite), it's probably more complicated than it needs to be, but serves as an example:

Code (csharp):

1. using System;
2. using System.Xml.Serialization;
3.  
4. namespace Genexis.Domain.Commands
5. {
6.     [XmlType("item")]
7.     [XmlRoot("item")]
8.     public class ItemCommand
9.     {
10.         private int m_ID = 0;
11.         [XmlAttribute("id")]
12.         public int ID { get { return m_ID; } set { m_ID = value; } }
13.  
14.         private ItemCommandAction m_Action = ItemCommandAction.Add;
15.         [XmlIgnore]
16.         public ItemCommandAction Action { get { return m_Action; } set { m_Action = value; } }
17.         [XmlAttribute("action")]
18.         public string ActionString
19.         {
20.             get
21.             {
22.                 return m_Action.ToString().ToUpper();
23.             }
24.             set
25.             {
26.                 if (!String.IsNullOrEmpty(value))
27.                 {
28.                     this.m_Action = (ItemCommandAction)Enum.Parse(typeof(ItemCommandAction), value, true);
29.                 }
30.             }
31.         }
32.  
33.         private int m_ModelConfigID = 0;
34.         [XmlAttribute("mcid")]
35.         public int ModelConfigID { get { return m_ModelConfigID; } set { m_ModelConfigID = value; } }
36.  
37.         private double m_X = 0;
38.         [XmlAttribute("x")]
39.         public double X { get { return m_X; } set { m_X = value; } }
40.  
41.         private double m_Y = 0;
42.         [XmlAttribute("y")]
43.         public double Y { get { return m_Y; } set { m_Y = value; } }
44.  
45.         private double m_Z = 0;
46.         [XmlAttribute("z")]
47.         public double Z { get { return m_Z; } set { m_Z = value; } }
48.  
49.         //private double m_RotationX = 0;
50.         //private double m_RotationY = 0;
51.         private double m_RotationZ = 0;
52.  
53.         [XmlAttribute("rotation")]
54.         public double Rotation { get { return m_RotationZ; } set { m_RotationZ = value; } }
55.  
56.         private bool m_Cache = false;
57.         [XmlIgnore]
58.         public bool Cache { get { return m_Cache; } set { m_Cache = value; } }
59.         [XmlAttribute("cache")]
60.         public string CacheString
61.         {
62.             get
63.             {
64.                 return m_Cache.ToString();
65.             }
66.             set
67.             {
68.                 m_Cache = (value == null ? false : value.ToLower() == Boolean.TrueString.ToLower());
69.             }
70.         }
71.  
72.         private bool m_Flip = false;
73.         [XmlIgnore]
74.         public bool Flip { get { return m_Flip; } set { m_Flip = value; } }
75.         [XmlAttribute("flip")]
76.         public string FlipString
77.         {
78.             get
79.             {
80.                 return m_Flip.ToString();
81.             }
82.             set
83.             {
84.                 m_Flip = (value == "1");
85.             }
86.         }
87.  
88.         private BoundingType m_BoundsMode = BoundingType.Box;
89.         [XmlIgnore]
90.         public BoundingType BoundsMode { get { return m_BoundsMode; } set { m_BoundsMode = value; } }
91.         [XmlAttribute("boundsmode")]
92.         public string BoundsModeString
93.         {
94.             get
95.             {
96.                 return m_BoundsMode.ToString();
97.             }
98.             set
99.             {
100.                 try
101.                 {
102.                     m_BoundsMode = (BoundingType)Enum.Parse(typeof(BoundingType), value, true);
103.                 }
104.                 catch
105.                 {
106.                     //TODO log this
107.                     //probably don't want to throw an exception as this will occur on deserialization
108.                 }
109.             }
110.         }
111.  
112.         private bool m_Selectable = true;
113.         [XmlIgnore]
114.         public bool Selectable { get { return m_Selectable; } set { m_Selectable = value; } }
115.         [XmlAttribute("selectable")]
116.         public string SelectableString
117.         {
118.             get
119.             {
120.                 return m_Selectable.ToString();
121.             }
122.             set
123.             {
124.                 m_Selectable = (value == null ? false : value.ToLower() == Boolean.TrueString.ToLower());
125.             }
126.         }
127.  
128.         private PropertyTransformCollection m_Data = new PropertyTransformCollection();
129.         [XmlElement("data")]
130.         public PropertyTransformCollection Data
131.         {
132.             get
133.             {
134.                 return m_Data;
135.             }
136.             set
137.             {
138.                 m_Data = value ?? new PropertyTransformCollection();
139.             }
140.         }
141.  
142.     }
143. }
144.  

You can also see in the last property "Data" it uses a PropertyTransformCollection which is another similarly adorned class that contains other serializable properties. In the end, it serializes/deserializes XML that looks like:

Code (csharp):

1. <?xml version="1.0" encoding="utf-8"?>
2. <item xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" id="1" action="CHANGE" mcid="28" x="0" y="0" z="0" rotation="0" cache="True" flip="False" boundsmode="None" selectable="True">
3.   <data>
4.     <properties>
5.       <p n="Colour" v="#FFFF0000" />
6.     </properties>
7.     <transformations>
8.       <t t="Scale" v="0.75" />
9.     </transformations>
10.   </data>
11. </item>

So you can create classes in Unity that with the correct design and attributes can likely match whatever the XML schema is that you're trying to read/write.


EDIT: you can also see in the class an example of how I use the XmlIgnore attribute for the Flip property. With this, my code can use the boolean but I can apply special formatting for serialization/deserialization with the "FlipString" property instead which is what's included in the XML. The XML specification was "1" or "0", but I wanted my code to treat that as a boolean true/false.

EDITx2: actually, there's probably a bug in that Flip getter property when sending data out (which we don't use which is why we never noticed it), it should look like this instead:

Code (csharp):

1. public string FlipString
2.         {
3.             get
4.             {
5.                 return m_Flip ? "1" : "0";
6.             }
7.             set
8.             {
9.                 m_Flip = (value == "1");
10.             }
11.         }

EDITx3: Also, when you go to try and create the classes and format them to match the XML specification, I find it easier to serialize those classes and manually compare the XML to see what needs tweaking/changing instead of trying to deserialize the incoming XML. Reason being, if way you setup the attributes/classes isn't exactly correct, the deserialization will throw not-very-helpful exceptions. Better to just create a dummy instance of those classes, serialize it and manually compare the resultant XML.

 

FizixMan,

I really appreciate your help. But unfortunately the XML files I need to read aren't always formatted the same way. Some tags may exist in certain files and not in others. So if I deserialize an XML which does not correspond to the format of the class, there's an error and the program will not proceed. However I know very well what tags/attributes to look for, and a parser may handle that.

I guess I'll resort again to a script-based parser. Dang! So frustrated!

 

I think you can use the XmlDefaultValue attribute to handle cases where tags/attributes are missing in the XML.

http://msdn.microsoft.com/en-us/library/system.xml.serialization.xmlattributes.xmldefaultvalue.aspx

Additionally, if you implement the IXmlSerializable interface, then you can perform the xml node reading/writing manually:

http://msdn.microsoft.com/en-us/library/system.xml.serialization.ixmlserializable.aspx

 

Hey Fizixman, IXmlSerializable is nice indeed, but its XmlReader() method needs a System.Xml.XmlReader, which won't work on the web player.
You say above you use XmlTextReader, but can you really make it work in the browser? It also depends on System.Xml...

 

Yup, here, I'll post the code from our old Serialization class. Again, probably not the best example of best code (and it's even been completely scrapped/overhauled for our Silverlight application) but you'll get the idea. In particular, note the FromXml(Stream, Type) method as that's the meat potatoes of deserializing.

In fact, looking at this now, we didn't actually need the XmlTextReader initially. We used to just convert the xml string into a MemoryStream and pass that into the XmlSerializer. The only reason we added the XmlTextReader was to handle some crap about how whitespace/newlines were being handled/translated between our various mediums.

Code (csharp):

1. using System;
2. using System.IO;
3. using System.Xml.Serialization;
4. using System.Xml;
5.  
6. namespace Genexis.Common.Utilities
7. {
8.     public class Serialization
9.     {
10.  
11.         private static ILog LOG = LogManager.GetLog(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
12.  
13.         public static T FromXml<T>(string xmlString)
14.         {
15.  
16.             try
17.             {
18.                 using (MemoryStream stream = new MemoryStream(StringEncoding.StringToUTF8ByteArray(xmlString)))
19.                 {
20.                     return FromXml<T>(stream);
21.                 }
22.             }
23.             catch (Exception ex)
24.             {
25.                 LOG.Error(ex);
26.             }
27.  
28.             return default(T);
29.  
30.         }
31.  
32.         public static T FromXml<T>(Stream xmlStream)
33.         {
34.             try
35.             {
36.                 return (T)FromXml(xmlStream, typeof(T));
37.             }
38.             catch (Exception ex)
39.             {
40.                 LOG.Error(ex);
41.                 while (ex.InnerException != null)
42.                 {
43.                     ex = ex.InnerException;
44.                     LOG.Error("INNER: " + ex.Message + ", TRACE: " + ex.StackTrace);
45.                 }
46.             }
47.  
48.             return default(T);
49.         }
50.  
51.         /// <summary>
52.         /// Deserializes the given xml string
53.         /// </summary>
54.         /// <param name="xmlString"></param>
55.         /// <returns></returns>
56.         public static object FromXml(string xmlString, Type type)
57.         {
58.             try
59.             {
60.                 using (MemoryStream stream = new MemoryStream(StringEncoding.StringToUTF8ByteArray(xmlString)))
61.                 {
62.                     return FromXml(stream, type);
63.                 }
64.             }
65.             catch (Exception ex)
66.             {
67.                 LOG.Error(ex);
68.                 while (ex.InnerException != null)
69.                 {
70.                     ex = ex.InnerException;
71.                     LOG.Error("INNER: " + ex.Message + ", TRACE: " + ex.StackTrace);
72.                 }
73.                 throw;
74.             }
75.         }
76.  
77.         /// <summary>
78.         /// Deserializes the given xml stream
79.         /// </summary>
80.         /// <param name="xmlStream"></param>
81.         /// <returns></returns>
82.         public static object FromXml(Stream xmlStream, Type type)
83.         {
84.             try
85.             {
86.                 //needed to add this wrapper because it was normalizing newlines from \r\n to just \n
87.                 XmlTextReader reader = new XmlTextReader(xmlStream);
88.                 reader.WhitespaceHandling = System.Xml.WhitespaceHandling.Significant;
89.                 reader.Normalization = false;
90.                 reader.XmlResolver = null;
91.  
92.                 XmlSerializer deserializer = new XmlSerializer(type);
93.                 return deserializer.Deserialize(reader);
94.             }
95.             catch (Exception ex)
96.             {
97.                 LOG.Error(ex);
98.                 while (ex.InnerException != null)
99.                 {
100.                     ex = ex.InnerException;
101.                     LOG.Error("INNER: " + ex.Message + ", TRACE: " + ex.StackTrace);
102.                 }
103.                 throw;
104.             }
105.         }
106.         /// <summary>
107.         /// Serializes the given object into an xml string
108.         /// </summary>
109.         /// <param name="obj"></param>
110.         /// <returns></returns>
111.         public static string ToXml(object obj)
112.         {
113.             if (obj == null)
114.             {
115.                 throw new ArgumentNullException("obj");
116.             }
117.  
118.             try
119.             {
120.                 using (MemoryStream stream = new MemoryStream())
121.                 {
122.                     XmlSerializer serializer = new XmlSerializer(obj.GetType());
123.                     serializer.Serialize(stream, obj);
124.  
125.                     string serializedData = StringEncoding.UTF8ByteArrayToString(stream.ToArray());
126.  
127.                     //we need to remove all characters before the first '<'.  
128.                     //These are characters indicating the formatting and they screw up some applications trying to parse the XML.
129.                     serializedData = serializedData.Substring(serializedData.IndexOf('<'));
130.  
131.                     return serializedData;
132.                 }
133.             }
134.             catch (Exception ex)
135.             {
136.  
137.                 LOG.Error(ex);
138.                 while (ex.InnerException != null)
139.                 {
140.                     ex = ex.InnerException;
141.                     LOG.Error("INNER: " + ex.Message + ", TRACE: " + ex.StackTrace);
142.                 }
143.                 throw;
144.             }
145.         }
146.     }
147. }

This is all running in 2.6.1. Perhaps there's a bug with all this in the current version 3?


EDIT: There's usages in there of "StringEncoding" which is another class of ours. All it does is convert to/from strings and UTF8 byte arrays.

 

maybe external call and parse xml with javascript on webpage then send data back to player?

 

Hi,

This answer is probably too late already, but if someone is interested. I got XmlDocument.Load() to work in the webplayer.

First of all I use Resources.Load() to load my textasset dynamically and then I use a XmlTextReader for the XmlDocument.Load() call. At first it did not work at all and I seemed to get the following error in the player log:

Code (csharp):

1. MethodAccessException: Attempt to access a private/protected method failed.

After inspecting the logs I noticed that the error always occurred when the following methods were called (note that they are in reverse order):

Code (csharp):

1.   at System.Security.SecurityManager.ThrowException (System.Exception ex) [0x00000] in <filename unknown>:0
2.   at System.Xml.XmlResolver.ResolveUri (System.Uri baseUri, System.String relativeUri) [0x00000] in <filename unknown>:0
3.   at System.Xml.XmlUrlResolver.ResolveUri (System.Uri baseUri, System.String relativeUri) [0x00000] in <filename unknown>:0
4.   at System.Xml.DTDReader.PushParserInput (System.String url) [0x00000] in <filename unknown>:0
5.   at System.Xml.DTDReader.GenerateDTDObjectModel () [0x00000] in <filename unknown>:0
6.   at Mono.Xml2.XmlTextReader.GenerateDTDObjectModel (System.String name, System.String publicId, System.String systemId, System.String internalSubset, Int32 intSubsetStartLine, Int32 intSubsetStartColumn) [0x00000] in <filename unknown>:0

So I thought that it could have something to do with System.Xml checking the xml file with the dtd and calling a method that was inaccessible by the security of the web player. Therefore I just commented out the reference to the DTD file in my xml files and suddenly everything seemed to work again.

So if you have the same issue, try removing your DTD reference in your xml file if it is not necessary. In my case it wasn't because the xml-files are used for some level logic designed by our game designer in an xml framework. He checks the xml vs. the DTD and it doesn't change afterwards.