reading 5 million points from csv

[georgeofthejungle]

Hi,

I am looking to read a .csv file that contains 5 million numeric points. I want to read the data and put it into an array. I then I want to read another (different) 5 million point csv file and put this data into column 2 of the array. - then read a third to give me column 3 and a 4th to give me column 4. I have a bit of code to do this (using a for loop), but when i try and get the 4th array i get an out of memory error message.

I initialise a 2D array 4 x 5million before reading the data and then put my read data into this array using a build array. - I use a shift register at either side of my for loop which takes the initialsed array around the loop.

I wondered if i needed to be reading the csv file say 500k points at a time? - i started off down this road but i am struggling to read 500k points, i think because my csv file consists of 1 row of data, of varying length - for example 0,0,0,0,0,5,5,5,5,2.345,2.829,0,0,0 and so on until the 5 millionth point.

I hope this makes some sense to someone and that you might be able to help me.

I can do nothing about the data format and file type - it is what it is i am afraid.

Fairly new to labview and never had to do anything with large csv files before. - I might just be going about this all the wrong way. - I am afraid i am prohibited from uploading code or screenshots. - Sorry, i know this can't help.

[GerdW]

Hi george,

instead of BuildArray I would use ReplaceArraySubset...

[johnsold]

Five million points should not be a big problem with the  memory sizes available today.

Without code or images it will be difficult to determine what is going on.  Can you possibly create a simple VI showing how you are trying to do this but whic odes not reveal anything proprietary and post that?

You second paragraph may contain a clue.  Use of Build Array inside a loop will cause frequent memory re-allocations.  Since LV requires that arrays be stored in contiguous locations, you may have memory fragmented to the point where no blcok large enough for the next allocation exists.  The proper way is to initialize the array outside the loop and wire the initialized array to the left terminal of a shift register.  Inside the loop use Replace Array Subset to put the data from your files into the array. This does not create any extra copies and does not require any re-allocation of memory.

Search the NI site for a white paper on handling large amounts of data.  It was written several years ago but is still relevant.

Lynn

[georgeofthejungle]

Hi,

Thank you for the suggestions. I have used replace array subset, but still i run out of memory. - Also i don't believe my code is working as i would like it to. The idea is that the massive array is initialised and is then big enough to contain all my data. Then bit by bit i fill up this massive array. Each pass of the loop reading and then adding 1000 characters to the array in column 0. - But, each time i read the spreadsheet i see (in highlight code execution mode) something like #1x5000001, whereas i'd expect to see something like #1x1000? - Also, 1000 characters, does this mean 1000data valves from the csv entries, or just literally 1000 characters (0,0,0,0,5,5,5,9.3,43 etc) from my spreadsheet? - Does the vi automatically fill my massive array correctly, sorting each csv file entry into an array element?

Attached is the screen shot of the code.

Sorry for all the questions but i think i am now confusing myself and would like a pro to take a look!

[Yamaeda]

You dont have any real stop condition for the loop.

/Y

[georgeofthejungle]

Hi Yameda,

Not worried about stop condition, this was just a quick 'sketch' to demonstate what i am trying to do with another program that i am not allowed to post. So it was for demsonstration purposes if you like. Thanks for looking though.

Regards,

G.

[johnsold]

G.,

Look carefully at the Detailed Help file for Read From Spreadsheet File.vi.  From you image it appears that you have 1000*i wired to "start of read offset" and 1000 wired to "max characters/row ..."   You have not wired  "number of rows," for which the default is -1 = read the entire file.

Try wiring the 1000 to "number of rows."

Next. Read From Spreadsheet File.vi converts the characters to numeric arrays. To find out how many elements are in the column you extract, use Array Size.  You probably need another shift register to keep track of the index in the output array. Add the size on each iteration.

Create a small file and test the VI you posted.  Add indicators to the 2D array and the column extracted so you can see what is going on. Add a delay if it runs too fast to watch.  Little test VIs are extremely good ways to learn how things work.

Lynn

[georgeofthejungle]

Hi Lynn,

The read number of rows input is of limited use to me, becasue my csv file contains just one row. - 1 row of 5 million values, each separated by a comma. So if i read 1 row, i  read the whole file.

I have done some experimenting with small vis, and continue to do so, but i am running out of ideas.

Thank you for your help, -please don't think i am seeking the easy life here, i am trying to work out a solution at my end, but i am appreciating the help.

Regards,

Oliver.