topic Re: Fast processing of mixed representation binary files in LabVIEW

Fast processing of mixed representation binary files

nemi — Wed, 18 Dec 2013 20:57:39 GMT

Hi Everyone,

I see multiple ways of tackling this but I'm looking for the fastest approach as my data set is very large....

The issue:

I have a binary data file holding 2D data.

It encodes 200+ differnet "columns" that are repeated in time (sampled)

The data contains mixed data representaitons: a mxiture of U8, I8, U16, I16 etc.

They are all regullarly repeated in a known file structure (660 bytes per "line")

I'd like to gnereate the 200+ differnet 1-D arrays form the file each using the correct data representaiton (or a sub-set of the columns).

I can load the file in using binary file read and I specify U8 as the data type . I can then rediension to the correct 2D array.

I'm now stuck on the fastest method to process the columns of data (1-2 bytes wide) into the corect numeric representaiton 1-D arrays (2x U8 to i16 etc.).

Scanning byte by byte would be very slow.

any suggesitons?

Re: Fast processing of mixed representation binary files

nathand — Wed, 18 Dec 2013 21:11:28 GMT

I would create a cluster that matches the format of a single line. Use that cluster as the data type input to Read from Binary File, and set the count to as many lines as you have in the file. Then you'll have an array of clusters, one element per line. If you want to extract a single column, loop through the array of clusters and unbundle the desired element.

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 14:19:22 GMT

That is an interesting solution. I will give it a go.

One (addtional) issue is that the structure of the data could change from different sources so I would ideally like to progrmaiatically import the data. Is there any way to programatically build a cluster of mixed data types to use as the import data type?

I'm also looking into getting the data in as a U8, converting to 2D array of U8. Then slicing out 1,2 ,4 wide colums of data corresponding to u8 /i8, u16/i6, u32/i32 .

I'm wondering if i can then cast the data to the correct type. .... Do I flatten the 2d arrays of U8 to string and then onwards to the new data type (1D) arrays?

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 15:19:00 GMT

I have a working draft solution that can be made programatic to cope with different (mixed representaiton) binary file formats:

1) Import the mixed rpresentaiton binary data as U8 1-D array using binary file read

2) Redimension the 1D U8 array to the correct sized 2D array to represent columns and rows (any data represented by >8 bits will now span 2 or more colums)

3) Itterate through the 2D U8 array at the correct column index places extracting 2D arrays with all rows (samples) and N width columns (N=1 for i8/U8, N=2 for i16/u16 etc.)

4) for each extracted array flatten the data to string (prepend array size = FALSE).

5) then unflatten string to the data type you need by inputing a empty 1D array of the correct data type (i8, u8 , i16 etc.) and choosing correct big / little edian format for the conversion.

6) the output is a 1D array of the correct representaiton data.

I;ve got some array transposing goign on that I think I can eliminate....

I'll try and post some tidy simplified code soon.

Re: Fast processing of mixed representation binary files

Taki1999 — Thu, 19 Dec 2013 15:23:13 GMT

I don't get why you would flatten to string.

Won't Join Numbers on your N=2 columns work just as well?

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 15:31:41 GMT

Here is an exmaple:

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 15:38:50 GMT

>>suggesting to use number join approach

Yes, just tested that and it also works, here is vi code picture for comparison.

I will have to do further testing to see which approach is faster. However both methods are (probably) much faster than using a FOR loop so should "good enough".

Re: Fast processing of mixed representation binary files

Taki1999 — Thu, 19 Dec 2013 15:45:12 GMT

Your sequence structures are unnecessary.

I'd bet on Join Number being faster than any string functions.

What sort of mechanism are you going to use for column definitions?

Do you care if 8 bit datatypes get upcast to 16 bit?

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 15:57:31 GMT

@Taking wrote:

Your sequence structures are unnecessary.

I'd bet on Join Number being faster than any string functions.

What sort of mechanism are you going to use for column definitions?

Do you care if 8 bit datatypes get upcast to 16 bit?

Agreed, the sequence structures only present to aid illustration.

I'm not sure flatten to string is a "true" (slow) string function. I'm viewing it more as a container of bytes. I'm going to run some speed tests. The array massaging that has to go on to use the join function may be a large overhead.

Column definitions will be sourced from a secondary text file that describes the file structure. The example conversion of 2 x u8 to i16 would be replaced by a for loop (over all columns of 1-N bytes) and case structure (representation of the column) that processes the data. Ultimately each 1D array of correctly converted data will be saved off to it's own binary data file in a appropriate numeric representation within the case statement.

Next I will be looking what for loop paralization I can achieve vs. source array memory copies. (again it is a very big source data file).

Re: Fast processing of mixed representation binary files

Taki1999 — Thu, 19 Dec 2013 16:03:37 GMT

Sounds like you've got a pretty good handle on it.

Here's how I'd do it, assuming that I'd downcast back to U8 when necessary at a later point.

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 16:11:38 GMT

@Taki1999 wrote:

Sounds like you've got a pretty good handle on it.

Here's how I'd do it, assuming that I'd downcast back to U8 when necessary at a later point.

That's a very concise looking method to join the numbers. I'll check that out , thank you 🙂

Aside: yes, your definition cluster in very similar to what I have already in my main program. I already have such a procesor working with delimited text files of arbiary data types (encoded as text) , implementing binary data types is the next (requested) step.

What are your thoughts on paralizing that loop in your example? will there be a large overhead of (multiple) memory copies of the U82DArray ?

Re: Fast processing of mixed representation binary files

Taki1999 — Thu, 19 Dec 2013 16:21:52 GMT

@nemi wrote:

What are your thoughts on paralizing that loop in your example? will there be a large overhead of (multiple) memory copies of the U82DArray ?

The profiler says that the loop isn't parallelizable due to the shift register.

We could make it parallelizable by adding the column index to the column definition cluster and using that instead of the shift register.

How large of data sets are we talking about? What sort of machines are you running on?

It's been a long time since I've worked with large data sets that had to be chunked on multiple processors in parallel.

I'd see what the performance of the standard algorithm is before expending the additional effort to optimize for parallel operation.

Re: Fast processing of mixed representation binary files

nemi — Thu, 19 Dec 2013 16:32:31 GMT

@Taki1999 wrote:

@nemi wrote:

What are your thoughts on paralizing that loop in your example? will there be a large overhead of (multiple) memory copies of the U82DArray ?

The profiler says that the loop isn't parallelizable due to the shift register.

We could make it parallelizable by adding the column index to the column definition cluster and using that instead of the shift register.

How large of data sets are we talking about? What sort of machines are you running on?

It's been a long time since I've worked with large data sets that had to be chunked on multiple processors in parallel.

I'd see what the performance of the standard algorithm is before expending the additional effort to optimize for parallel operation.

Gigbytes of 300+ column data .

I already load my files in chunks to parse them and have sucessfully used parallel loops on the text file based data ( 2D string array parsing).

The major spead up was using paralel for loops for multi-threaded data mining of the 100+ individual 1D binary array files that result from the 1st step parsing. (we parse once to 1D binary files and analize them multiple times).

BTW, to answer my own quesiton:

by the looks of it there shoudl be no memory overhead for reading from a source array in multiple palces:

http://zone.ni.com/reference/en-XX/help/371361J-01/lvconcepts/vi_memory_usage/

"...However, in this case the Index Array function does not modify the input array. If you pass data to multiple locations, all of which read the data without modifying it, LabVIEW does not make a copy of the data. As a result, all the data is in-place....."