Actually, you do not have to read the whole file at once, although that does make the programming easier. You can read and write data in chunks to keep memory usage down and file write efficiency up (65,000 byte file chunks give best read and write performance on Windows systems). You will end up with a steadily increasing memory buffer as you read a chunk from disk, add to it, then write what you can to disk before reading another chunk. At the end, you will have a buffer the same size as your added data that you can then stream to disk at the end to finish the file. The algorithm would go something like this:
- Read N bytes from disk
- Modify as needed by adding data at the end of each row. Your data is now of size N + M
- Write N bytes to disk, keeping the remaining M in a memory buffer
- Read the next N bytes from disk
- Modify and append to the current M bytes from the previous iteration.
- Write N bytes to disk, keeping the remaining in a memory buffer
- Repeat steps 4 - 6 until the entire file has been read. You now have a buffer of left over data.
- Stream leftover data to end of file in 65,000 byte chunks.
In theory, this would give better performance, if done well, since you do not need to convert the entire file to numbers, then back again, but only need to convert the numbers going into the file into text. In practice, if you are not very careful, all the appending and reshaping of strings and arrays can cause memory issues if you don't reuse and manage the buffers intelligently, causing slowdowns. This can be made disk speed limited, if done correctly.
