TDMS files: Programatically adding channels

[bockdoug]

I am aquiring data from an optical spectrum analyser and write this data to a TDMS file. Currently the program analyses the spectrum to find peaks and logs each peak. This works fine, however, occasionally additional peaks pop up suddenly, and might disappear again. My program does log these changes, but in a very unreadable manner. Please see the attached image for an overview. The project is also attached. The Main VI is "OSAandDC.vi", and the VI doing the writing to the file is "WriteSpectrumToFile.vi".

I personally do not think there is a solution, as any peaks that appear infront of already logged peaks would have to be inserted infront of those previous peaks. But this would mean that the entire column would have to be shifted over before writing the new peak, and that doesnt take care of inserteing the value in the correct row yet either...
Any ideas. 
Cheers in advance
Doug

[mcduff]

You could try this. Write a set number of peaks, for example 10, instead of just how many you find. If there is only two peaks, the positions/amplitudes of the other peaks would be NaN or -1. In this manner you always have a place holder for empty peaks, and everything is aligned. If you were writing to a text file you would use an empty string as a placeholder.

Cheers,

mcduff

[bockdoug]

Ok but how does that affect file size? I have upto 10,000 rows of data for upto 20 peaks. If theres only 1 peak i would have 10,000*(20-1)=190,000 entries of NaN... 

[Hooovahh]

In these cases I'd suggest having a seperate time scale for the different DAQ rates.  Have an AbsoluteTime, and then an AbsoluteTime_Peak2.

Often times you'll see people break up their TDMS file into seperate groups, where each group is a different logging rate.  That could be used here too, so that each group only has one Time channel, and all other channels will be taken at the same rate and share the same time.

I do wish TDMS could do what you described easily.  But based on how TDMS is structured there really isn't an easy way to say skip a row.  TDMS doesn't know about rows, it knows about raw data, usually streamed from some hardware.

[c-RADAE]

So I'm not sure exactly how that will affect your file size, but if you'd like to check the file size during the test you can use the function File/Directory Info. from the Advanced File I/O pallete. This way you can determine if you will run into any file size constraints. 

Conrad S.

National Instruments

[mcduff]

Basically, it would increase your file size.

There is another possible option. Keep your data sorted by the greatest number of peaks, ie, data with 5 peaks would come before data with 1 peak. In this way, all your empty rows would be okay.

Cheers,

mcduff

[bockdoug]

@Hoovahh: My problem is not one of acquisition rates. Some peaks just appear and might just as well disappear again, randomly, in between sweeps. I sweep a spectrum at a constant sweep rate and then perform peak analysis. Say one sweep shows three peaks, then the analysis gives me an array of the wavelengths of these three peaks and I write them to the file. Then, in the next sweep, say the first of the previuos peaks splits and there are now 4 peaks, with the additional peak occuring between the "original" peak one and two. Now the analysis correctly returns four peaks, but logging in the TDMS file is wrecked...

[Hooovahh]

Could you create a time channel for every peak?

[mcduff]

Ok but how does that affect file size? I have upto 10,000 rows of data for upto 20 peaks. If theres only 1 peak i would have 10,000*(20-1)=190,000 entries of NaN... 

After saving all of your data but before closing your file you could check each column. If a column contained ALL NaNs then delete that channel. Obviously is only one row had 20 peaks then you still would have a large file size.

Or you can write separate files, write all of the data with 1 peak to FilePeak_1, write all of the data with 2 peaks to FilePeak_2, etc.

mcduff

[bockdoug]

@mcduff I am not entirely sure what you mean... Do you mean sort the wavelengths from smallest to largest before writing to file. If so, thats what the code is doing at the moment and it is messy when peaks appear in between previous peaks.

My idea now is keep a type def somewhere which only stores the last array of wavelengths. Then, after I analyse a spectrum, the result of this analysis is compared to the array in the type def. If I allow for small changes (I expect only small drifts of the peaks) I can then determine which peak is which. For example, if I find peaks at 1550, 1560 and 1570, then in the next sweep at 1550.2, 1555.3, 1560.4 and 1570.5 I know that peak 1550 and 1550.2 are the same (so that goes to the same column), 1560 and 1560.4 are the same and 1570 and 1570.5 are the same, while 1555.3 is not within range of the previous peaks, so it must be a new peak, so I write this into a new column as the 4th peak.
Only problem then would still be writing it to the correct row so that it is associated with the correct time. Are there any end of line characters for TDMS that make sure that any additional peaks don't just get written to the top of the file?

Cheers
Doug