Showing results for 
Search instead for 
Did you mean: 

How to increase write speed using queues

Go to solution



    I am using a vi to read in two sets of data over a serial port.  I then write the data to a spreadsheet.  I do this by sending the data to a queue when I depress a record data button on the vi.  The queue then  writes the data to a file in another loop.  I based this off of a producer consumer architecture.  The setup works great but I have noticed that when record data for long periods of time (2 minutes or more) it takes an extremely long time after I have stopped recording data for the information to be written to a file.  Is there anyway to improve my write to file speed?  Thanks for all of your help and I am attatching my VI.

0 Kudos
Message 1 of 18

How fast is the data being generated and placed in the queue?


I would dequeue the elements in your consumer loop.  Accumulate them into an array in a shift register.  Once the array grows large enough (or the loop is forced to end) write the now larger array out with the Write to spreadsheet file and clear the array that feeds into the shift register.  By bunching your data together into larger blocks to write out, it will run faster.  Why?  Most of the time is spend just opening and closing the file, not much time with actually writing the data if it is a small dataset.  By minimizing the number of opens and closes, the loop will run faster.

Message 2 of 18

Along the same lines, you could open the file just once then use the other file-writing VI's to replicate the behavior of the 'write to spreadsheet' VI (but without all the file openings and closings).


How much data are you generating in an average 2-minute run?

Message 3 of 18

I had originally thought along the lines of adding a shift register within the consumer loop.  I am not sure how to implement this and have it write the data I set to record to a file then clear the array without eliminating the loop so I can record new data while running the program.  Could you provide an example?  That would be very helpful. 


In response to the other question.  I am collecting data from one source at a rate of 200 Hz and another at about 100 Hz.  I had also thought of using a different vi to write the data but again I am unsure of which VI to use.  


Thanks for all of your help.

0 Kudos
Message 4 of 18
Here is an example of storing your data into an array, and periodically writing that array to a file.  I set it up so that it collects 100 1-D arrays into a 2-D array before writing it out.  You could change that number to something else.
Message 5 of 18
I looked at your example and it is exactly what I need.  I still have a few questions about it though.  Instead of writing to the file every time the array reaches 100, is there a way to use shift registers to create an array with all of the data I recorded when I pressed the record button then write all of it to the file at once?  How would I determine when all my data is in the array to signal that I should write it all to a file?  Also, using the example Ravens Fan provided, as the loop runs it writes to a file when the array reaches 100.  On the last itereation of the loop, there may not be enough data for the array to reach a length of 100.  How would this last bit of data be written to the file?
0 Kudos
Message 6 of 18



It seems to me that Ravens Fan's code takes care of both those things.  That code will replace your existing consumer loop for writing to a file.  It will only accumulate data if there is data in the queue, and (according to the VI you posted earlier) there is only data in the queue if you have pressed the record button on the front panel. Hence data is only recorded when you press the record button.


As for the remaining 0<x<100 lines - when the queue is destroyed (which, in your code, happens after the STOP button is pushed on the front panel), whatever data is left in that array will be written to file before the consumer loop exits.


Perhaps I'm not understanding your questions correctly?

0 Kudos
Message 7 of 18
I use the vi to make multiple recordings to different files while the labview vi is running continuously.  When I am finished recording one set of data and want to record another set to a different file, it seems to me that I could have in the consumer loop array, data from my prevoius recording that was not written to the old file.  Is that a correct interpretation of the example code?
0 Kudos
Message 8 of 18

ah, I didn't think about writing to multiple files in one run.  Yeah that's a problem.


The thing is, it's difficult to clear stuff out of that accumulator because that loop's execution is gonna block once the queue empties.  I guess you could have some logic in there that clears the accumulator every time the filename changes... I don't think that's the right way to go about this, though.


Seems to me like you should make the "unit" of the queue a 2D array, and do the accumulating in the producer loop?  Then when you're ready to move on to a new file, "if (recording is done) and (still stuff in the accumulator), then output it to the queue," even though it's not a full 100 lines.  Once everything's out of the queue, change the file name and start a new recording.


I dunno, I'm not a queue expert so let's wait and see what the old farts say


0 Kudos
Message 9 of 18

Yes. you could definitely accumulate the data in a 2-D array in the producer.


What I would do, since now you are adding more requirements to the code, would be to turn to add an enum along with the data you send through the queue to the consumer loop.  The enum would be commands such as Open File, close file, log data, write to file, exit loop.  Attached is an example of one I wrote that I encapsulated in a subVI that runs parallel to other code.

0 Kudos
Message 10 of 18