04-24-2012 09:39 AM - edited 04-24-2012 09:40 AM
I've been trying to debug a problem with a DataPlugIn with no success. It was originally provided here in response to my request last November.
The Plugin works fine so long as the number of columns in the header is less than the number of columns of data (Channels) in the text file.
However, I have recently expanded the header information, and sometimes the Header now has more columns than the Data does.
When this is the case, the DataPlugin either generates an unspecified error, or it simply creates multiple "NoName" Data channels at the end of each group.
I'm guessing that somehow the Loop counter on line 33 is not being set.
For some reason the loop counter "i" is retaining the value set on line 18.
A sample input datafile that demonstrates the problem and the DataPlugIn URI are zipped and attached.
04-25-2012 08:14 AM
Hi Greg -
Good to hear from you again! It's been a few months - how's your DIAdem use going? Anything else we can help with? Now that you've added more properties to your header, you'll have even more flexibility for searching and mining through your data using the DataFinder. If you haven't seen this yet, let me know.
It seems you've already fixed the issue yourself, but just for kicks, I took a look and you're spot on. The number of items in the header has nothing to do with the issue you were seeing - Line 18 is written to dynamically account for a fewer or greater number of properties (by looping until the size of the array of parsed property names).
Instead, since your data (and channel name) rows have multiple delimeters at the end of the row, the Split method was seeing each one. The only parameter of the Split method that could help is the third one - which is optional - as it allows us to specify the maximum number of splits made. To utilize this, though, you'd have to know ahead of time how many legitimate channels to expect.
Because it seems likely you may have a varying number of channels, the solution you've identified is a great one - perform a check to see whether the [channel name / channel data] is empty, and if so, skip.
By the way, the TDMS file format was built to specifically account for situations like this. It allows you to dynamically scale the number of properties, channel groups, and channels in your file. I'm not sure if you have any control over the output format of the data, but if you intend to make future changes, TDMS may be something to consider.
04-25-2012 09:12 AM
I was baffled at first because I kept looking at the datafiles in Excel - which hid the comma delimiters from me.
Once I opened the files in a text editor and saw the commas, I knew that was the problem.
I intend to check my Labview code (which generates these datafiles) to eliminate these extra delimiters when I get time. But since I have already generated quite a bit of data with them in there, at least I can handle them properly now.
We do occasionally use TDMS for recording raw waveform data and then convert it to matlab files for those who want it.
However, in most cases we process everything in realtime and then output a text/csv file to keep our Excel and MatLab users happy. I know there are tools for both of those applications that allow them to work directly with TDM and TDMS, but getting people to use them is a battle. For now the csv text files are a necessary bridge.
Thank you for the reply!