Why does reading XML take so long?

[smercurio_fc]

I have dabbled with trying to use .NET from LabVIEW to read XML. Don't bother. Unless you feel like going to watch a movie while you wait for the file to be read. The problem isn't .NET per se, but rather the fact that you have to go though layers of interfaces to actually get to the .NET stuff. Doing the same thing in, say, a C# program, is no problem.

I have also used LabXML, which uses the Microsoft XML parser. It's reasonably fast, though it comes at the cost of ActiveX versioning headaches. There's a version of it that uses the open-source libXML, but I have found that to be slower than the one using MS XML.

[Bob_Schor]

---

@Steve Chandler wrote:

Out of curiosity can you check the performance of EasyXML?

---

A few replies ago, I briefly mentioned that I did test EasyXML, at least enough to know that it exhibits the same behavior (linear write, quadratic read) as the NI XML code.

I think I see "what's wrong" with the NI implementation -- I'm going to take a crack at "fixing" it, and will post the results (unless they are just too embarassing, of course).

BS

[Bob_Schor]

---

@TailOfGon wrote:

Which Version of LabVIEW are you using?

---

I did this with LabVIEW 2011.  I haven't tried earlier releases with this code.

[Bob_Schor]

---

Bob Schor wrote:

I think I see "what's wrong" with the NI implementation -- I'm going to take a crack at "fixing" it, and will post the results (unless they are just too embarassing, of course).

BS

---

Boy, that certainly is embarassing.  I completely missed it.  To summarize the Read XML from File routine, it first reads the entire file into a (very long) string.  It then hands the string off to Parse XML Fragments, which looks for XML tags and builds an array of strings with the contents of the tags.

What I noticed is that this function starts with an empty array of strings passed into a shift register, and as XML Elements are found, they are added to the end of the array with "Build Array".  "Aha!", I thought, "very inefficient, they should have used an Indexed Array Tunnel" (ignoring, for the moment, how to end the process ...).  Nope, "fixing" this made basically no difference.

Indeed, eliminating the "output" step entirely, that is, not building the output array, but returning a (constant) empty array ran just about as fast, or I should say, slow.  The problem is all the string handling, breaking strings up into sub-strings, passing the substrings around, etc.  A moment's thought shows that this has the potential for exponential growth, much worse than quadratic, as each pass through the loop generates more and more "string pieces".  I expect a smart "string-walking" routine could be built that would parse the string in linear time, but now I better appreciate the hoops that NI and LabVIEW are trying to jump through here.

I consider this "Question" answered, but don't think I can "Accept as Answer".  If someone else agrees that this is the likely answer, I'll let them "click the button".

Bob Schor

[SteveChandler]

---

@BOB Schor wrote:

---

@Steve Chandler wrote:

Out of curiosity can you check the performance of EasyXML?

---

A few replies ago, I briefly mentioned that I did test EasyXML, at least enough to know that it exhibits the same behavior (linear write, quadratic read) as the NI XML code.

BS

---

That's what I get for not reloading first!

[smercurio_fc]

Can you post the XML file that was created? (Please zip it since you said it's 1.7MB)

Also, the data structure you are trying to read it into.

[Bob_Schor]

---

@smercurio_fc wrote:

Can you post the XML file that was created? (Please zip it since you said it's 1.7MB)

Also, the data structure you are trying to read it into.

---

Instead of sending the biggest XML file, I'm sending the smallest.  Since I'm zipping it, I'm also including the TypeDef for the cluster into which the data are read.  The actual data structure is an array of these clusters.  The file is created by passing the array through a FOR loop with indexing -- the cluster comes in, is passed through Flatten into XML, and comes out as an XML string, which is then turned into an array of strings.  This goes directly to Write to XML File.

For reading, I reverse the process.  I Read from XML File to produce an array of strings, which (inside an indexing FOR loop) I turn each individual string back into a Folder Info cluster using Unflatten from XML, recovering the array of clusters.  Incidently, I compared the original array (even the one with 5000 elements) with the "recovered" array after writing the XML file and reading it back into a different array -- they are, as expected, identical.

[Bob_Schor]

Sorry -- I always forget to attach the file ...

[smercurio_fc]

I took the XML file that you provided and copied the contents a few times to get a large file. I ended up with a file that was about 2.2 MB in size (3800 "folders"). Using the MXXML library (ActiveX) and the LabXML wrapper VIs I wrote up a little VI to read the contents of that file and extract out the pieces of information to put into your data structure. With the file I created the VI took about 50 seconds to read in the file. That's not too bad. Note that I was running this in a virtual machine that was running Windows XP with LV 2011. My computer itself is a quad-core i7-860, so the virtual machine was not making full use of the processing capabilities of the computer. I also was using MSXML 4.0 SP2.

I've attached the VI so you can play around with it.

I haven't tried the XML Parser VIs, and that was a previous suggestion, so that's another route to test.

[TailOfGon]

Hi, inspired by smercurio_fc, I made a VI which make use of LabVIEW XML Parser.

Here is the result:

"20 folders" took 14 ms

"3927 folders" took 1.6 sec

"38000 folders" took 130 sec

My Environment:

Windows 7 32 bit, 2GB RAM, Intel Core 2 Quad CPU 2.66GHz / LabVIEW 2011 SP1