LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Process FFT on large data files

Hi!
In my application I'm working with large binary data files (TDMS). When I'm reading whole data and perform FFT on it, labVIEW crashes and says that it goes out of memory (which was predictable because i'm performing FFT over 27*5000000000 U32 array).
The idea is to read the TDMS file 10.000 pts by 10.000pts and perform the FFT on this smaller array... But how to get the real FFT result (like if processed the FFT algorithm over the entire array) using the partials FFTs ?


CLA, CTA, LV Champion
View Cyril Gambini's profile on LinkedIn
This post is made under CC BY 4.0 DEED licensing
0 Kudos
Message 1 of 5
(4,019 Views)
What does your data represent?  Have you considered a STFT?  I have done this to analyze 500KHz dat that was streamed over a long time period (>1hr) which had many 100s of GB.  Here you can see your signals freq response over time.
 
Paul
Paul Falkenstein
Coleman Technologies Inc.
CLA, CPI, AIA-Vision
Labview 4.0- 2013, RT, Vision, FPGA
0 Kudos
Message 2 of 5
(4,014 Views)
The data have been acquired previously using a PXI module and it represents vibrations (sensor : accelerometer). The TDMS files are already written on the disk and I can't do on-line FFT analysis. The files are too big to be read at once, so I must read them parts by parts.
Is the global FFT something like the average of the partial FFTs ? Or does it involves other mathematical process such as convolution, etc... ??
CLA, CTA, LV Champion
View Cyril Gambini's profile on LinkedIn
This post is made under CC BY 4.0 DEED licensing
0 Kudos
Message 3 of 5
(4,006 Views)
Nobody has a solution ?
CLA, CTA, LV Champion
View Cyril Gambini's profile on LinkedIn
This post is made under CC BY 4.0 DEED licensing
0 Kudos
Message 4 of 5
(3,981 Views)
My solution worked on my large peacwuired data sets. You can do ffts on shorter time samples (In my case I implemented  a sliding window STFT) where I used a state machine to read data from file in chunks and run the fft on the chunks, and then repeted the process until the data was fully processed.  You can use the averaging built into the FFT algorithm or can use a shift register to keep track of the data. 
Paul Falkenstein
Coleman Technologies Inc.
CLA, CPI, AIA-Vision
Labview 4.0- 2013, RT, Vision, FPGA
0 Kudos
Message 5 of 5
(3,966 Views)