Re : How to read very large csv File

[Napier]

I am a new fish for the Labview...

I made a DAQ acquisition system. And sampling rate is 100k samples per seconds, I captured 60 seconds..

And the CSV file is too big to read it back into the labview(Memory is full).

How to read a very big csv file in labview(alomost 350MB)?

Thanks Guys!

[ramses64]

Hi,
you can

use the offset and length to read a part of your large file : then you divide your laarge csv in several small csv which can be opened with LabVIEW...

Best Regards,

[Napier]

Thanks for your reply,

I want to plot the csv in the labview using the waveformgraph.

Is there a specific vi can set the offset and length?

[Freelance_LV]

How are you reading your csv file?

did you try using the Read Text File function and Set File Position function? you will have to set the number of characters to read in the count input for Read Text File. You can set the offset in Set File Position

[Napier]

I'm using the read drom spreadsheet file function..

[Norbert_B]

Several points:

a) a file of 300+MB IS readable in one piece, if you make sure that your application creates no or a very limited number of copies of that data. I would expect your application to create more than one copy of that data. Improving code for performance could solve the issue (maybe unlikely).

b) Obviously, you are running LV 32bit on a 32bit OS. You can swap to 64bit OS and use LV 64bit. This should solve the issue (still, code might be unclean).

c) "Read from Spreadsheet" reads by default the whole file at once. You can use the input "max characters/row' (optional input) to limit the number of values to be read. You of course have to iterate on the function and move the "start of read offset" (also optional input) to the next junk to read.

hope this helps,

Norbert

[johnsold]

As Norbert said, it is possible to read large files.

Read from Spreadsheet File brings the file in as a string and then converts it to a numeric array.  As a consequence it makes several copies of the data.  That, and the way it handles errors makes it less suitable in my opinion for use with large arrays or in robust applications.

If you are going to read in chunks anyway, go to the low level file and string to number conversion functions and control your data copies yourself.

Lynn