Data logging to csv file

[lee810]

I am new to Labview and trying to help out with a project.  One of the objectives is to take around 20 to 25 channels, some of which are displayed on various gages on the Labview front panel screen, as well as other which are are arrays of values that are not displayed and log them to a csv file for later analysis with Excel.  I've played around with the Write to Measurement File.vi (both tdms and lvm formats) but it didn't put the variables in an easy-to-use format to use in excel.  The tdms format worked OK with a single variable, but when I tried to make it work with more than one variable, it arranged them in a single column and lost the timestamps.  I've also been playing around with Write to Spreadsheet File.vi, which appears to be better for what I need, but it's still taking a lot more effort than I would have imagined, and I'm unable to figure out how to add the header row, or add timestamps at the beginning of each row, or have a good way to overwrite the file, because it just keeps appending to it.  When I set it up to overwrite the file, it overwrote the file with each row of data it sent to it.  It also insists on using tabs as delimiters even though I've changed it to a comma in the delimiter field.

There are a number of loops in the program because it's consolidating data from several CAN buses along with analog data.  We just want to take one second slices and arrange them like this:

TimeStamp, RPM,  Temp1,  Temp2, Pressure1,  Pressure2,....

2011-09-15 10:11:00, 2390, 74.3, 77.9, 25.1, 33.2, ......

2011-09-15 10:11:01, 2385, 74.6, 78.1, 25.0, 33.1, ......

....

I wrote a C program to do this prior to using Labview and it was quite simple and took virtually no time but, perhaps due to my lack of LV experience, I'm not finding an easy way to do it in Labview.

Even getting the parameters out of the various loops and into this separate 1-second data logging loop is not obvious.  Based on an example I found, 've been setting up waveform charts for each variable, and it feels like overkill because the waveform charts then appear on the front panel and have to be hidden out of the way.

Any tips would be appreciated.

Thanks,

Lee

[johnsold]

Lee,

Before the Write to Measurement File.vi and Write to Spreadsheet File.vi existed, a lot of data was saved in text files and you can still do it the same way today.

Since you know how to do it in C, it should be fairly straightforward to do the same thing in LV.  Use the string functions to format the headers. Write that to the file with the Write to Text File function.  Then accumulate your data in one or more shift registers.  Use the Number to String conversion and Format Date/Time to String functions to create a formatted string representing your data.  Write that to the file.  When done with all the data you can write a footer if needed and close the file.  You might need to use the Set File Position function to avoid overwriting, or to intentionally overwrite when you want to do that.  Read the detailed help and write some simple test programs until you get it worked out.

I suspect that you are using the charts for their history data.  As you have commented it seems like and is an unnecessary use fo a control.  Put an array of data in a chift register.  Initialize it outside the loop to the size of the maximum amount of data you need to accumulate.  Inside the loop use Replace Array Subset to put data into the array as you acquire it.

Lynn

[lee810]

Hi Lynn,

Thanks for your reply.  I gave the more generic 'Write to File' a try, and found it to require a lot more work than the Write to Spreadsheet File.  And I have a lot to learn about using shift registers.  I admire those of you who have mastered the nuances of figuring out how to do these things with Labview using the old school techniques. 

I eventually got it working with the Write to Spreadsheet File.  My stumbling block was trying to figure out how to get data out of one loop and into another.  The trick was using the 'local variable' element and then setting it up for output and assigning it to a value that was represented by some previously named variable that was displayed as a gauge, output of an array, etc.   I didn't need to be setting up Waveform Charts on all the variables I was trying to capture, the local variable element was all I needed.  Then I converted the output of a large numeric array to a string array, added the timestamp to it with another build array block, and was able to write a few dozen comma separated  parameters to a spreadsheet file once a second. 

[johnsold]

Lee,

The use of local variables for transferring data between parallel loops will work, but it is not considered good practice.  Writing to a local creates an extra copy of the data, requires a switch to the User Interface thread, and can lead to race conditions. It also requires an indicator or control on the front panel even if display of the data is not needed or wanted. And it is slow compared to other techniques.

A better approach is to use LabVIEW classes, a queue, or an Action Engine.  I have not used classes as they are relatively new and the other methods have worked for me for a long time.  With a queue you create the queue with the array of numerics as its datatype.  You enqueue data in the loop(s) which read(s) the data and dequeue it in the file loop.  The queue can act like a buffer to store data if the file loop slows down due to disk operations or something.  Of course on average it needs to keep up or you eventually run out of memory.  If writing to file once a minute makes sense, you can accumulate that much data and write it all at one time. The Action Engine is a subVI which uses a while loop and an uninitialized shift register to retain the data.  Search for Ben's Nugget on Action Engines.

Lynn