transforming large data arrays

[charliedawkes]

Hi,


I believe this is quite a simple question but I am trying to find the most efficient way of doing this, currently I have acquired multi channel binary data in files that that can be upto and above 1GB. The data is stored in one file in the order (say when acquiring 3 channels) channel 1, channel 2, channel 3, channel 1..... and so on. I need to then convert this data into a spreadsheet file, .txt is fine and also transform it into voltages and reorientate it so the new file would be a 2d array in the form:

channel 1 channel 2 channel 3
channel 1 channel 2 channel 3.......... and so on

Currently I do this very simply by reading the I16 binary data, changing it to voltages by multiply by 10/32768 (I work with the range of 10v to -10v and the binary is 16bit) decimating the 1d array and building it to a 2d array and saving this.

The problem is when doing this to large files the system runs out of memory, I was wondering if there is a way just to part of the file at a time instead of all at the same time and just appending it to the saved file?

Thanks

Charlie

[GerdW]

Hi Charlie,

read just portions of the file - you can always specify the amount of data to read in!
The same applies for saving the data, you just have to append it (standard setting when using "Write to Text file")...

[charliedawkes]

Yeah that sounds perfect, how would you do that?

Thanks

[GerdW]

Hi Charlie,

it's as simple as the attachment...

[charliedawkes]

I'm very grateful for your help but I'm running 7.1, is there any chance of changing it?

Thanks

[GerdW]

Hi raj,

here's a picture (worth a thousand words 😞

Functions look different in LV7.1 (from left to right): Open file, read text file, write text file, close file.

And you should note your LabView version when you ask for example code!


Message Edited by GerdW on 01-24-2008 01:30 PM

[charliedawkes]

Hi,

I couldn't find the read icon that you had on the picture and the one that is similar did not seem to read binary? So I tried doing it another way, I've got something that works.... nearly... the problem is when I try and convert large binary files, say over 2GB. I believe this is because of the read subvi I'm using but I don't know how or where to change it so it can accept larger files.

Charlie

Message Edited by charliedawkes on 01-26-2008 10:48 AM

[charliedawkes]

...Also the conversion itself seems quite slow, converting a 36mb file takes about 30 seconds, then a 2GB file would take about half an hour! Is there a better way to make it faster as well?

Thanks

[GerdW]

Hi Charlie,

to make it faster you may use lower level file functions, that means opening and closing the files just once outside the conversion loop. But this will not help much, as the main work here is calculation and conversion to string... To speed it up I would try to read in blocks of roughly 65kB in size (~10500 rows of 3 columns of I16).

[GerdW]

Hi Charlie,

to speed up conversion for your big datafiles I can suggest also this: Try to work with lookup-tables!

1) Make up a lookup-table for every (2^16) I16-value possible. Create result strings as needed (format, length)
2) initialize a string with required length to keep all values from point 3-5
3) read in a chunk of data
4) for every I16-word get the result string from your table, use it to replace (!) a substring in a pre-initialized string
5) write the resulting string to a file
6) repeat with 3 until EOF

This scheme will work best when you used fixed-length result strings. (Atleast in older versions of) Labview may be faster when you work with U8-arrays instead of strings - they are also easier to control in length