Help Optimizing Code: Reaching 100% buffer when saving data

[Joaquinh]

The following VI collects data from 3 sensors. Data from the 3 sensors goes into 3 separate "Write To Measurement" file within the While loop. This is causing the VI to reach 100% buffer and therefore crucial data is lost. If the "Write To Measurement" is removed, buffer stays around 10%. 

Also, I am using TDMS to save faster, but it's still not fast enough. What is a logical approach to saving data fast enough? Typically the files are of 1GB for 5 minute collection for all three sensors.

Below is an image of the location of the "Write To Measurement"

[altenbach · Accepted solution]

Maybe you want to append instead of creating a new file with every write operation if the file already exists (which it does). Look at your file IO express configurations.

Saving to file should be done in a parallel asynchronous loop. Use a queue based producer/consumer architecture.

(I also don't understand why you have an event structure, but that's a different discussion). Also your various "read/timeout cnts" belong in a feedback nodes or shift registers instead of ping-ponging via value property nodes. Where are their values initialized?

[Valentin87]

How about use Merge signals and use only one Write to measurement.vi?

[Joaquinh]

altenbach,
- I tried to append, but the value for the buffer was the same. It does help when starting the VI when the file already exists. I will look into the parallel asynchronous loop with a queue. Would it need three different queues? Or would one handle all three signals? The data size is 1x800 for each sensor.

- Event structure is more for the commercialization of the VI.  
- "read/timeout cnts" are initialized at the beginning when the sensors start monitoring.

Valentin87,
The merging was the first thing I did, but there was no difference in performance. Buffer was identical.

[altenbach]

It probably would also help to use low-level file IO. Open the files once before the loop, append inside the loop, and only close the files when done. Still, I would do a parallel loop for the file IO as already suggested.

---

Joaquinh wrote:

- "read/timeout cnts" are initialized at the beginning when the sensors start monitoring.

---

In any case, you should not use value property nodes here.

Also, how many times do you think you need to unbundle the same value? Wouldn't once be enough? (see picture)

[mcduff]

Another optimization tip:

Why do you convert to I32 to do a division by 1M? Why not just convert to a double so the buffer can be reused. Right now you are making an I32 buffer and a Double Buffer, just make it a double initially.

mcduff

[altenbach]

---

mcduff wrote:

Why do you convert to I32 to do a division by 1M?

---

The data is originally U32 and some values might represent negative I32 values. Need to be careful here...

[mcduff]

@Altenbach

Thanks did not see that.

[Joaquinh]

altenbach,

Thank you for that observation, it has been fixed.

The array from each sensor is 800 x 1200, would the queue handle this?
I am attempting to do the asynchronous while loop using the queue but I am not sure how to handle the large array as the queue has output of 'element'

[ZKami]

Are you still seeing the error mentioned above? The queue should be able to handle the array you describe. 

If you have LabVIEW 2016 or above you can also use channel wires. 

Channel Wires
http://www.ni.com/white-paper/53423/en/