Combining Multiple XML Files into One File

[dlovell]

Hey Everyone,

So I am creating a program that takes in a bunch of arrays and stores them into a single xml file. I am finding out that my lag time to add these arrays to an xml file is getting worse with each array I put into it. I am placing over 5000 arrays into this file. This isn't working how I wanted it to so I have decided I will have to make multiple arrays to be able to obtain all of my arrays without having to sit here all day. I would like to have a VI that will take all of these xml files and put them together into one xml file. Does anyone have any ideas how this could be done?

Thanks,

Dustin

[Darin.K]

I like XML files, I really, really do, but are you certain they are the right fit for this application.  There is a lot of overhead as opposed to say a binary file or something along the lines of TDMS or HDF5.

If your heart is set on XML, I would try to split the difference.  One file/array is a bit of a nightmare, so I would find the "right" number based on your application and break it into chunks.  It could be 100 arrays/file or 50 or 200.  I'd add one more file which serves as an index.  To retrieve an array you look it up in the index file to determine which data file it is located in.

Of course I am making an assumption here, and that assumption is that you are using an efficient method in your current program to add the arrays to the XML file.  Perhaps that is not the case, so how exactly are you adding these arrays to the file?

[dlovell]

Thanks for the quick reply. XML might not be the best but the xml file was a sort of requirement given to me...  I will have 21 xml files with about 250 arrays per file and I would like to have the contents of these 21 xml files combine into one xml file. I have a VI that you choose an xml file and you choose array name you are looking for and the array appears on your output screen so you can see the specified value. Attached is my array to xml vi, reading from an xml file vi and a sample of one of my xml files.

[Darin.K]

We all know that the information content of XML is pretty low, but the LV schema is particularly inefficient. You could consider rolling your own XML if you are so inclined, you could probably reduce the file size by 60% at least. At any rate, you are constantly rewriting the entire file instead of appending which is probably the root of much of your problem. You should open up the write to XML file vi to see the steps: write header, write data, write close tag. You should break up the steps in your vi. Write the header, append the arrays inside the loop, and then write the close tag after the loop. In this way you are only appending new data instead of constant rewriting.

[dlovell]

Will this work even if I don't have all of my arrays at once? I am gathering my arrays throughout a whole testing process. How do you append an array?

[Darin.K]

We need to set a few groundrules for your data:

1. XML, maybe or maybe not LV schema

2. You will be generating multiple tests, in your example file it is always a 2D array with 2 rows and 1 column, is this for real or just an artifact of this example.  I was suspecting that you would have multiple length arrays.

3.  What is the timescale of your data taking?  If I am doing a relatively short, easy to repeat test, I take my chances with failures and gather all the data and write it at the end.  If it is a long, difficult measurement I tend to write in intervals (not necessarily every test, but typically every 5-10 minutes).

If your data is fixed size, like the example file, you can append your individual arrays into one large 2D array in a couple of different ways and write the single array to the XML file.  This is more compact.  If it is not the same size, I was suggesting that you append only the new XML text to the end of the file as you go along.  At the end you finish up by writing the closing tag.  If there is a problem before that, the XML file is not readable, but easily fixed by simply adding the closing tag.  You can even get fancy by keeping track of the offset of the start of the closing tag and start overwriting the new XML+close tag at that point.  This means the file is readable even during the intermediate steps, but you are not writing the whole thing.

I can help with an example or two once you specify the data requirements a little more.

[dlovell]

It has to be XML. The array should be 2 columns 1 row, but this isn't a big deal right now. The array use to be 2 columns and 46 rows. My reason for the array become so smaller is because I need each array to have a specfic name so I can use my Find Array.vi file to find a specfic measurement. The first measurement is a power level in dBm and the second is the frequency it was taken at. So I have to have a whole bunch of small arrays. I origninally had a huge array and didn't have this problem but I had no way to search the whole array for a specfic value I could only pull up the entire array. The test took me about two hours to complete, I don't really have a timescale for taking data. Hope this answers all of your questions. Thanks