Im using a csv file as my data input for a test. The test data is simply recorded in an array. How do I add this array data back to the original csv file, as an additional column, once the stop button is clicked?
Solved! Go to Solution.
The only way to append a new column is to read the entire file, convert to a 2D array, append the new column, and then write the new 2D array to file. This is a limitation of the way windows writes files and has nothing to do with LabVIEW.
Actually, you do not have to read the whole file at once, although that does make the programming easier. You can read and write data in chunks to keep memory usage down and file write efficiency up (65,000 byte file chunks give best read and write performance on Windows systems). You will end up with a steadily increasing memory buffer as you read a chunk from disk, add to it, then write what you can to disk before reading another chunk. At the end, you will have a buffer the same size as your added data that you can then stream to disk at the end to finish the file. The algorithm would go something like this:
In theory, this would give better performance, if done well, since you do not need to convert the entire file to numbers, then back again, but only need to convert the numbers going into the file into text. In practice, if you are not very careful, all the appending and reshaping of strings and arrays can cause memory issues if you don't reuse and manage the buffers intelligently, causing slowdowns. This can be made disk speed limited, if done correctly.