Process FFT on large data files

[CyGa]

Hi!
In my application I'm working with large binary data files (TDMS). When I'm reading whole data and perform FFT on it, labVIEW crashes and says that it goes out of memory (which was predictable because i'm performing FFT over 27*5000000000 U32 array).
The idea is to read the TDMS file 10.000 pts by 10.000pts and perform the FFT on this smaller array... But how to get the real FFT result (like if processed the FFT algorithm over the entire array) using the partials FFTs ?

[falkpl]

What does your data represent?  Have you considered a STFT?  I have done this to analyze 500KHz dat that was streamed over a long time period (>1hr) which had many 100s of GB.  Here you can see your signals freq response over time.

Paul

[CyGa]

The data have been acquired previously using a PXI module and it represents vibrations (sensor : accelerometer). The TDMS files are already written on the disk and I can't do on-line FFT analysis. The files are too big to be read at once, so I must read them parts by parts.
Is the global FFT something like the average of the partial FFTs ? Or does it involves other mathematical process such as convolution, etc... ??

[CyGa]

Nobody has a solution ?

[falkpl]

My solution worked on my large peacwuired data sets. You can do ffts on shorter time samples (In my case I implemented  a sliding window STFT) where I used a state machine to read data from file in chunks and run the fft on the chunks, and then repeted the process until the data was fully processed.  You can use the averaging built into the FFT algorithm or can use a shift register to keep track of the data.