Write to CSV Buffering

[mnchem]

Hi everyone,

Hopefully this will be my last post for this project. Disclaimer in advance: I am brand new to LabView/coding in general so please be patient with mistakes, limited vocab, messy code, etc.

I had a VI written that pulled up an Excel sheet and wrote data to that in real time, but I was getting (what I will call) buffering by one second every few seconds near the beginning and up to a few seconds every other second after about thirty minutes. I anticipate I will need to run this code around 30 minutes at a time simultaneously with another instrument which properly collects data every second without buffering, so I can't have that. Someone told me the Excel solution I used was causing it to be inefficient, so I edited a LabView tutorial (Getting Started: Write Continuous Data to Spreadsheet (CSV) with Headers) to get it to write to a CSV instead. I think I got all of the actual data to work out finally, but I am still having the same issue that it skips a second every now and then, which adds up by the end. Does anyone have any solutions for this? The only thing I can think of is to fiddle with the while loop and/or for loop to keep the data collection running once a second but do the data writing later so it doesn't get backed up, but I haven't found a way to make that work yet.

Any advice would be appreciated! And again, beginner here so the simpler the terms you use the better- thanks!

[RTSLVU]

You should consider using a Producer/Consumer architecture.

A Producer/ Consumer does this:

1. The Producer loop acquires data (usually a lot of data at at a high speed) and puts it in a Queue (a FIFO buffer)
   
   1. Modern Prod/Con may use a Channel Wire instead of a Queue
2. The Consumer loop removes data from the Queue then processes and/or saves it to disk
   1. This allows the Consumer to takes all the time it needs to process the data without effecting the data acquisition.

I know it looks complicated from a beginners perspective, but it's really quite straight forward.  

[johntrich1971]

---

@RTSLVU wrote:

You should consider using a Producer/Consumer architecture.

A Producer/ Consumer does this:

1. The Producer loop acquires data (usually a lot of data at at a high speed) and puts it in a Queue (a FIFO buffer)
   
   1. Modern Prod/Con may use a Channel Wire instead of a Queue
2. The Consumer loop removes data from the Queue then processes and/or saves it to disk
   1. This allows the Consumer to takes all the time it needs to process the data without effecting the data acquisition.

I know it looks complicated from a beginners perspective, but it's really quite straight forward.  

---

I would tend to agree with this, but 1 second is a LONG time so this may not be necessary. One glaring problem that I see is the use of "Write Delimited Spreadsheet." This is opening and closing the file on every iteration of the loop. It would be better to use primitives top open the file outside the loop, write to the file using the file reference inside the loop, and then close the file after the loop has completed. This is true whether or not you use the Producer/Consumer approach (I highly recommend learning about it as it may be necessary in the future). 

[mnchem]

---

It would be better to use primitives top open the file outside the loop, write to the file using the file reference inside the loop, and then close the file after the loop has completed. 

---

---

Could you elaborate on this part please? I think I follow what you're saying conceptually and would rather try this first before making a whole producer/consumer loop, but I am getting a little confused on what I do to do this. Thank you!

[crossrulz]

---

@mnchem wrote:

---

It would be better to use primitives top open the file outside the loop, write to the file using the file reference inside the loop, and then close the file after the loop has completed. 

---

---

Could you elaborate on this part please? I think I follow what you're saying conceptually and would rather try this first before making a whole producer/consumer loop, but I am getting a little confused on what I do to do this. Thank you!

---

I would also agree that you are running at such a low data rate I would not worry about the Producer/Consumer.  Just opening the file before the loop, closing after the loop, and doing the writing inside the loop will really help as you will not be constantly opening and closing the file (which can get really slow).