LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Add data to csv input file

Solved!
Go to solution

Im using a csv file as my data input for a test. The test data is simply recorded in an array. How do I add this array data back to the original csv file, as an additional column, once the stop button is clicked?

0 Kudos
Message 1 of 4
(1,266 Views)
Solution
Accepted by topic author HePlayGame

The only way to append a new column is to read the entire file, convert to a 2D array, append the new column, and then write the new 2D array to file. This is a limitation of the way windows writes files and has nothing to do with LabVIEW.

Message 2 of 4
(1,243 Views)

Ahh, ok. Thank you for your help!

0 Kudos
Message 3 of 4
(1,233 Views)

Actually, you do not have to read the whole file at once, although that does make the programming easier.  You can read and write data in chunks to keep memory usage down and file write efficiency up (65,000 byte file chunks give best read and write performance on Windows systems).  You will end up with a steadily increasing memory buffer as you read a chunk from disk, add to it, then write what you can to disk before reading another chunk.  At the end, you will have a buffer the same size as your added data that you can then stream to disk at the end to finish the file.  The algorithm would go something like this:

 

  1. Read N bytes from disk
  2. Modify as needed by adding data at the end of each row.  Your data is now of size N + M
  3. Write N bytes to disk, keeping the remaining M in a memory buffer
  4. Read the next N bytes from disk
  5. Modify and append to the current M bytes from the previous iteration.
  6. Write N bytes to disk, keeping the remaining in a memory buffer
  7. Repeat steps  4 - 6 until the entire file has been read.  You now have a buffer of left over data.
  8. Stream leftover data to end of file in 65,000 byte chunks.

In theory, this would give better performance, if done well, since you do not need to convert the entire file to numbers, then back again, but only need to convert the numbers going into the file into text.  In practice, if you are not very careful, all the appending and reshaping of strings and arrays can cause memory issues if you don't reuse and manage the buffers intelligently, causing slowdowns.  This can be made disk speed limited, if done correctly.

This account is no longer active. Contact ShadesOfGray for current posts and information.
Message 4 of 4
(1,182 Views)