How to use TDMS Advanced functions properly

[MemoryGrowth]

Hello everyone!

There is a known memory growth issue with TDMS. Our application basically writes to TDMS files 24/7. Memory growth caused by TDMS is very critical for us. Our program even has a routine to auto-restart itself when the RAM usage gets too high, which always happened over time. Here is the relevant NI article:

Memory Growth with TDMS Write

As suggested in the article, we began using the TDMS Advanced Synchronous Write along with other Advanced TDMS functions to deal with the memory growth problem. With the Advanced TDMS, we can separately set up the channel info before entering the data-writing loop. We have confirmed that the memory growth problem is almost completely gone! The RAM usage has become very flat indeed. Now there are two new issues though.

1. Waveform as input data type for TDMS Advanced

TDMS Advanced Write does not accept waveform as its data type. The input to the original TDMS Write was waveform data (Y, dt and t0). We need that complete information to recontruct waveforms. For a work-around, we are trying to separately input each component of waveforms in the TDMS files.

It's easy to deal with Y (data values) but how can we put dt (time interval between data points) and t0 (time stamp) in a TDMS file? In other words, when we open a TDMS file with Excel, we want to be able to see wf_increment (dt, Waveform x-start time in DIAdem) and wf_start_time (t0, Waveform x-step width in DIAdem) values.

2. How to read TDMS files while writing

With the old TDMS methods, we can open the TDMS files anytime we want while data is still being written to them. Advaned TDMS does not seem to allow this. Is there any proper way to open TDMS files written by Advanced Write while the program is running?

Please take a look at the attached simple VI. I put some comments and questions in it. If you find anything wrong or how to improve it, please help us.

Many thanks!

[PiDi]

Hello... Well, very problem-oriented nick you have, MemoryGrowth 🙂

1. What is the problem with using Set Properties for saving waveform dt and t0? You can even name them using default names.

2. What exactly is wrong with Adv Synch Read? I've noticed that it works fine when I create new TDMS file, but when I open existing TDMS, values return by Adv Synch Read are very random.

Please take a look at attached VI, I've modified your VI and placed comments at places of modifications.

[deppSu]

1. "TDMS Set Properties" is supported to work with TDMS Advanced APIs. In your case, you can try using a non-empty value as property value input.

2. For TDMS Advanced APIs, data is not cached in memory for each write call. Thus, it's proper to use read while logging. You can use "TDMS Set Next Read Position" to locate to the position to read from and use the "count" terminal of "TDMS Advanced Synchronous Read" to specify the number of samples to read out. It will read all the data by default.

[MemoryGrowth]

Thank you folks! PiDi's modified VI was very helpful. I tweaked it a bit so it can reflect our real application better. I will probably have to work on TDMS Advanced Synchronous Read later. As PiDi commented, it reads data fine if the TDMS file was newly created. If an existing TDMS file is used, however, I get very weird numbers on the data indicators.

 

Here is another question. If I want to use 2 channels, do I just have to add another TDMS Set Properties in series for the second channel since it does not accept a string array for channel name?

Here is a relevant example from applications engineer Zach, in case someone else may wanna take a look at it. In this code, timestamp for each data point in a waveform is recorded side by side with the data values.

[MemoryGrowth]

Quick question: Is there any way to save numbers I want in the Waveform Constants on the front panel? Data Operations > Make Current Value Default doesn't seem to work. If I save and re-open the VI, the numbers are back to the default values (Y: 1,2,3,4,5). Please consider the input below for the following example.

I simply added two more TDMS Set Properties for two additional channels. In this example, there are only three channels (Ch0, Ch1 and Ch2) but if there are a lot more channels, a For Loop may be used to set up the channel names efficiently.

 

In this case, Y0, Y1 and Y2 nicely correspond to Ch0, Ch1 and Ch2. Each second, it records 5 data values with 0.2 second intervals for each channel. Data is nicely saved in the TDMS file with corresponding channel columns.

Now, let's consider another situation. In this case, each 1D array of data (Y) contains values for every channel. Consider the following input setup.

What is the best way to rearrange the data above so it can be nicely saved in the TDMS file just like in the first example? Since Transpose 2D Array didn't change anything, matrix functions were used. It would work well if the matrix was square but due to the uneven dimension sizes, it seems the transposed matrix retains junk columns. I can probably use Resize Matrix but this is getting non-generic and messy. Is there any nice way to deal with this data arrangement problem?

Last question: Is it possible to open a TDMS file that is being written by TDMS Advanced Write on an RT target (ex. PXI-8106)? By opening, here I mean using Excel or DIAdem. When we tried it, we got an error message that says it is not allowed. It doesn't let us drag any TDMS file from the FTP folder. If we try to open any TDMS while it's running on a PC, not the RT target, it works fine. When we used the old TDMS functions, we could open TDMS files on the PXI through the FTP server anytime with no trouble. 

Any tip is appreciated!

[deppSu]

If all the channel share same properties in one group, you can set these properties on the group instead of each channel.

[deppSu]

You can just set data to be interleaved instead of re-arrage your data as follows:

I have modifed your original VI for this case in the attachments.

Besides, you said "TDMS Advanced Sync Read" would return "crazy" data for an existing TDMS file. However, it should work. Would you please post an example VI of the "crazy" case. I have made such a example of using read for an existing file in the attachments and it works.

[MemoryGrowth]

Thanks, deppSu. The problem was something silly. For each 1D array Y, there were 5 elements, not 3. 2 last elements were just hidden. If you run the iteration once, it doesn't show up but if you run it more than once, you would see data is messed up. Since you added the 2D array indicator, I could see that easily.

I don't think the data layout setting (non-interleaved or interleaved) on TDMS Set Channel Information makes a difference for 2D array data. If the input data was a 1D array, filling sequence in 2D output format would certainly matter. In my example, it's 2D to 2D though.

I notice a Diagram Disable Structure was inserted in the code but the matrix manipulation part is on the Enabled side so it didn't change anything. If I replace the matrix part with a straight wire, data is messed up. It seems we do need transpose to re-arrange the data.

As for the 'crazy' numbers, it's the numbers displayed on the Waveform Graph. When you first create a TDMS file (Operation > create or replace), the Waveform Graph displays it fine. But if you choose Operation > open to write to the same file, it shows a weird peak.

For using Data Operations > Make Current Value Default on the Waveform Graph I figured I was right-clicking on the wrong spot.

[MemoryGrowth]

^ In the last line, I meant Waveform Constant, not Graph.

Now, the remaining problem is, how to open a TDMS file while data is still being written to it in a loop (in other words, before its reference is closed) on a RT target. 

When I was using the old TDMS functions, I was able to open TDMS files on the RT target anytime with Excel or DIAdem on this host computer. With the Advanced TDMS, I can still open a TDMS file anytime as long as the VI is just running on this computer. However, I have to use a FTP server to see the files created by a VI deployed to the RT target. In this case, I can't open a TDMS file untill its reference is closed in the VI. I get a file system error (12003) if I try to open a TDMS file in the FTP folder of the RT target while data is still being written to it. Is this an inevitable downside of using the Advanced TDMS or is there any workaround?

[Eric_NI]

Hi MemoryGrowth, actually the problem is he same as if you would have another file open in your computer and you try to move it; in this case you can check the example at the link below, it is about limiting the file size and everytime that size (or condition) is reached the file will be closed and you can move it as a single finle to analyze its data.

https://decibel.ni.com/content/docs/DOC-3916

Eric NI