I am using a vi to read in two sets of data over a serial port. I then write the data to a spreadsheet. I do this by sending the data to a queue when I depress a record data button on the vi. The queue then writes the data to a file in another loop. I based this off of a producer consumer architecture. The setup works great but I have noticed that when record data for long periods of time (2 minutes or more) it takes an extremely long time after I have stopped recording data for the information to be written to a file. Is there anyway to improve my write to file speed? Thanks for all of your help and I am attatching my VI.
Solved! Go to Solution.
How fast is the data being generated and placed in the queue?
I would dequeue the elements in your consumer loop. Accumulate them into an array in a shift register. Once the array grows large enough (or the loop is forced to end) write the now larger array out with the Write to spreadsheet file and clear the array that feeds into the shift register. By bunching your data together into larger blocks to write out, it will run faster. Why? Most of the time is spend just opening and closing the file, not much time with actually writing the data if it is a small dataset. By minimizing the number of opens and closes, the loop will run faster.
Along the same lines, you could open the file just once then use the other file-writing VI's to replicate the behavior of the 'write to spreadsheet' VI (but without all the file openings and closings).
How much data are you generating in an average 2-minute run?
I had originally thought along the lines of adding a shift register within the consumer loop. I am not sure how to implement this and have it write the data I set to record to a file then clear the array without eliminating the loop so I can record new data while running the program. Could you provide an example? That would be very helpful.
In response to the other question. I am collecting data from one source at a rate of 200 Hz and another at about 100 Hz. I had also thought of using a different vi to write the data but again I am unsure of which VI to use.
Thanks for all of your help.
It seems to me that Ravens Fan's code takes care of both those things. That code will replace your existing consumer loop for writing to a file. It will only accumulate data if there is data in the queue, and (according to the VI you posted earlier) there is only data in the queue if you have pressed the record button on the front panel. Hence data is only recorded when you press the record button.
As for the remaining 0<x<100 lines - when the queue is destroyed (which, in your code, happens after the STOP button is pushed on the front panel), whatever data is left in that array will be written to file before the consumer loop exits.
Perhaps I'm not understanding your questions correctly?
ah, I didn't think about writing to multiple files in one run. Yeah that's a problem.
The thing is, it's difficult to clear stuff out of that accumulator because that loop's execution is gonna block once the queue empties. I guess you could have some logic in there that clears the accumulator every time the filename changes... I don't think that's the right way to go about this, though.
Seems to me like you should make the "unit" of the queue a 2D array, and do the accumulating in the producer loop? Then when you're ready to move on to a new file, "if (recording is done) and (still stuff in the accumulator), then output it to the queue," even though it's not a full 100 lines. Once everything's out of the queue, change the file name and start a new recording.
I dunno, I'm not a queue expert so let's wait and see what the old farts say
Yes. you could definitely accumulate the data in a 2-D array in the producer.
What I would do, since now you are adding more requirements to the code, would be to turn to add an enum along with the data you send through the queue to the consumer loop. The enum would be commands such as Open File, close file, log data, write to file, exit loop. Attached is an example of one I wrote that I encapsulated in a subVI that runs parallel to other code.