Write one line of numbers to a text-file or write multiple lines each iteration?

[GiovanniV]

Hi,

I'm currently working on an application that has a few different while-loops running at the same time. One loop acquires the data (DAQloop), another shows the data (raw and processed data) on graphs (Graphsloop) and yet another stores the data in an open ascii text file (Storageloop).

The data is passed on from the DAQloop to the Storageloop through a queue. At this moment I let the Storageloop take up the first element of the queue (the oldest) to store it in the textfile each iteration. I was wondering however whether it would be better to let the Storageloop take up all the elements currently available in the queue to store it in one go.

Does anybody have any ideas on this? Which would be better performancewise? The textfile is already opened and is just passed on as a refnum.

Thx!

[GerdW]

Hi Giovanni,

you could poll the queue with GetQueueStatus (to check number of elements in queue) and FlushQueue to get all remaining elements in the queue...

You can also prevent polling by using GetQueueElement like this:

[GiovanniV]

Hey GerdW,

Thanks for the reply. The problem is not that I don't know how to store all the queued elements at once, the problem is that I don't know whether it's more efficient to store it line by line or store it array-of-lines by array-of-lines ...

[GerdW]

Hi Giovanni,

usually it's more efficient to store larger chunks of data at once...

(Modern OS use a lot of buffers internally, so Win7 might buffer your text data before writing them into the file actually. Your harddisc might buffer several write accesses before it actually does write the data.)

[DFGray · Accepted solution]

With the normal LabVIEW file primitives, you will get your best disk speed when you write about 65,000 bytes to disk at once (or read, for that matter).  When I write similar code, I include a string buffer in the write loop.  Every time I get write data, I add it to the string.  When the string hits 65,000 bytes, I write to disk.  You could break this into three loops, as well - a read, buffer, and write.  Depending upon you current chunk size, you could see over an order of magnitude change in disk write speed.  This write operation is often throttled by the conversion from binary to ASCII, not the disk I/O.  If you do this, be careful with your buffer.  You can get some serious slowdowns depending upon how you implement it.  This post has some details and examples.

[ThiCop]

Make sure to read DFGray's linked post, it really is a very interesting one and could prove to be extremely useful.

[Yamaeda]

With the improvements made in the 2010 compiler a build array in a for loop is very fast, there was some performance tests done and it was as fast as the preallocate/replace version. With a while loop it's a bit not quite true since the system cant preallocate with certainty.

/Y

[ThiCop]

Quick tip:
Also note the difference between using a Build Array function that adds an element at the end of an array and a Build Array function that adds an element to the beginning of the array.

Adding at the end is alot better regarding the amount of memory allocations.

There can however still be a significant difference in speed (in 2012) between a for loop that uses indexing to build the array (also a form of pre-allocation of resources) and a for loop that uses the Build Array function (even when adding at the end).