08-18-2020 11:41 AM
Hi NI Community,
I'm looking for advice on how to manage data collected from PXI chassis.
Currently, we are taking multi-channel data at a very high sampling rate (up to 1MS per second) and each run can last anywhere from 1 to 5 minutes. The output TDMS file can end up being 10-20GBs which is quite large! I've had no problems opening a 20GB TDMS file with DIAdem or FlexLogger to visually inspect the traces. However, I'm looking to see if there are ways I can avoid using DIAdem or FlexLogger to view the data. I found Scout (free version) online but it is not that great to open a large 20GB data file as it fills up your RAM quickly. Can you suggest to me how to deal with a large amount of data? And how to open and view large TDMS data files?
Thank you!
08-18-2020 12:02 PM
Hi Victor,
@victor55 wrote:
The output TDMS file can end up being 10-20GBs which is quite large! I've had no problems opening a 20GB TDMS file with DIAdem or FlexLogger to visually inspect the traces. However, I'm looking to see if there are ways I can avoid using DIAdem or FlexLogger to view the data. I found Scout (free version) online but it is not that great to open a large 20GB data file as it fills up your RAM quickly. Can you suggest to me how to deal with a large amount of data?
Use Diadem/FlexLogger as they are optimized for such large data files…
@victor55 wrote:
And how to open and view large TDMS data files?
There are TDMS functions in the file palette: you can write your own software to open/read TDMS files.
Think about a good decimation algorithm when starting with those 5min*60s/min*1MS/s=300MSamples long plots…
08-18-2020 01:42 PM
Hi Gerald,
Thank you for your reply as always. Do you have an example of a decimation algorithm or know where I can find some of that online?
08-19-2020 02:13 AM - edited 08-19-2020 02:13 AM
Hi Victor,
@victor55 wrote:
Hi Gerald,
Who is "Gerald"?
@victor55 wrote:
Do you have an example of a decimation algorithm or know where I can find some of that online?
LabVIEW has a DecimateArray function for a very simple decimation of an 1D array.
There are decimation functions in the signal processing palette. Read their help…
Use GoogleFoo to look up for other decimation algorithms like "Min/Max decimation"…