Instrument Control (GPIB, Serial, VISA, IVI)

cancel
Showing results for 
Search instead for 
Did you mean: 

What if TDMS file writing speed can't keep up with data acquisition speed?

Solved!
Go to solution

When I do data collection and data storage, I use producer-consumer architecture for data collection and storage. Once the program runs, there will be more and more elements in the queue. I guess this is caused by TDMS writing speed is too slow and data collection speed is too fast. Is there a good way to solve this problem, please help me!

5.png

0 Kudos
Message 1 of 7
(1,109 Views)
Solution
Accepted by topic author JHUN-KULI

When you are calling the TDMS write how many data points are you typically writing at a time?  Lets say you are only writing one sample every time you call the write.  When the TDMS Write flushes to disk it will write a bunch of header information, then write your data.  If you are writing lots of header information, but not much data, your TDMS file will become fragmented.  It will still contain all the same data but will take much longer to open, have a large index file, and the file will be larger than necessary.

 

For this I'd suggest creating some kind of buffer where you only call the TDMS Write once you have some large amount of data first.  You can dequeue your data, but build it into an array until you have say 50,000 samples, then call the TDMS Write once.  This will take a decent amount of time to write, but it will be much faster than calling the Write function 50,000 times, each with one sample.  You might also not even need to dequeue the data.  You can probably use the Preview Queue to wait until you have that many samples, then flush the queue and write.

 

One quick test you can do is to just disable the TDMS write function and see if your program behave properly.  If it still doesn't then the issue isn't with the TDMS Write at all.

0 Kudos
Message 2 of 7
(1,092 Views)

Thank you so much brother, you said exactly what I wanted to do, you helped me a lot, thank you so much!!

0 Kudos
Message 3 of 7
(1,089 Views)

I just solved this problem with the "empty6.png queue" function, but the idea is the same as yours. I'll try to solve this problem with your method later, and I think that's a good solution too!

0 Kudos
Message 4 of 7
(1,081 Views)
Solution
Accepted by topic author JHUN-KULI

Instead of just flushing the queue, I prefer to use the Dequeue Element inside of a FOR loop.  This way you can check for the "stop" condition inside of the FOR loop and abort it (enable the termination terminal).  The FOR loop also limits the number of samples you will accumulate.  You can then write the data that accumulated at the autoindexing output tunnel of the FOR loop to the TDMS file.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 5 of 7
(1,071 Views)
Solution
Accepted by topic author JHUN-KULI

That's a great suggestion.  I made tried to make that into a VIM.  I think it does what you are suggesting, where it will continually dequeue elements until you have the specified number.  It does use a while loop instead of For just because I didn't know any other way to know the number of elements it has read.

0 Kudos
Message 6 of 7
(1,060 Views)

I think I'm seeing a break in understanding on the specification here.  Are we dealing with a single channel or multiple?  My initial thoughts were based on multiple channels and/or set number of samples per queue element.  But if there is just 1 channel and we do not know how many samples, Hooovahh's VIM is the way to go.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 7 of 7
(1,045 Views)