04-14-2022 03:18 AM
When I do data collection and data storage, I use producer-consumer architecture for data collection and storage. Once the program runs, there will be more and more elements in the queue. I guess this is caused by TDMS writing speed is too slow and data collection speed is too fast. Is there a good way to solve this problem, please help me!
Solved! Go to Solution.
04-14-2022 07:53 AM
When you are calling the TDMS write how many data points are you typically writing at a time? Lets say you are only writing one sample every time you call the write. When the TDMS Write flushes to disk it will write a bunch of header information, then write your data. If you are writing lots of header information, but not much data, your TDMS file will become fragmented. It will still contain all the same data but will take much longer to open, have a large index file, and the file will be larger than necessary.
For this I'd suggest creating some kind of buffer where you only call the TDMS Write once you have some large amount of data first. You can dequeue your data, but build it into an array until you have say 50,000 samples, then call the TDMS Write once. This will take a decent amount of time to write, but it will be much faster than calling the Write function 50,000 times, each with one sample. You might also not even need to dequeue the data. You can probably use the Preview Queue to wait until you have that many samples, then flush the queue and write.
One quick test you can do is to just disable the TDMS write function and see if your program behave properly. If it still doesn't then the issue isn't with the TDMS Write at all.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
04-14-2022 07:59 AM
Thank you so much brother, you said exactly what I wanted to do, you helped me a lot, thank you so much!!
04-14-2022 08:15 AM
I just solved this problem with the "empty queue" function, but the idea is the same as yours. I'll try to solve this problem with your method later, and I think that's a good solution too!
04-14-2022 09:02 AM
Instead of just flushing the queue, I prefer to use the Dequeue Element inside of a FOR loop. This way you can check for the "stop" condition inside of the FOR loop and abort it (enable the termination terminal). The FOR loop also limits the number of samples you will accumulate. You can then write the data that accumulated at the autoindexing output tunnel of the FOR loop to the TDMS file.
04-14-2022 09:31 AM - edited 04-14-2022 09:34 AM
That's a great suggestion. I made tried to make that into a VIM. I think it does what you are suggesting, where it will continually dequeue elements until you have the specified number. It does use a while loop instead of For just because I didn't know any other way to know the number of elements it has read.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
04-14-2022 09:54 AM
I think I'm seeing a break in understanding on the specification here. Are we dealing with a single channel or multiple? My initial thoughts were based on multiple channels and/or set number of samples per queue element. But if there is just 1 channel and we do not know how many samples, Hooovahh's VIM is the way to go.