Manage and open large TDMS data files

[victor55]

Hi NI Community,

I'm looking for advice on how to manage data collected from PXI chassis.

Currently, we are taking multi-channel data at a very high sampling rate (up to 1MS per second) and each run can last anywhere from 1 to 5 minutes. The output TDMS file can end up being 10-20GBs which is quite large! I've had no problems opening a 20GB TDMS file with DIAdem or FlexLogger to visually inspect the traces. However, I'm looking to see if there are ways I can avoid using DIAdem or FlexLogger to view the data. I found Scout (free version) online but it is not that great to open a large 20GB data file as it fills up your RAM quickly. Can you suggest to me how to deal with a large amount of data? And how to open and view large TDMS data files?

Thank you!

[GerdW]

Hi Victor,

---

@victor55 wrote:

The output TDMS file can end up being 10-20GBs which is quite large! I've had no problems opening a 20GB TDMS file with DIAdem or FlexLogger to visually inspect the traces. However, I'm looking to see if there are ways I can avoid using DIAdem or FlexLogger to view the data. I found Scout (free version) online but it is not that great to open a large 20GB data file as it fills up your RAM quickly. Can you suggest to me how to deal with a large amount of data?

Use Diadem/FlexLogger as they are optimized for such large data files…

---

@victor55 wrote:

And how to open and view large TDMS data files?

---

There are TDMS functions in the file palette: you can write your own software to open/read TDMS files.

Think about a good decimation algorithm when starting with those 5min*60s/min*1MS/s=300MSamples long plots…

[victor55]

Hi Gerald,

Thank you for your reply as always. Do you have an example of a decimation algorithm or know where I can find some of that online?

[GerdW]

Hi Victor,

---

@victor55 wrote:

Hi Gerald,

---

Who is "Gerald"?

---

@victor55 wrote:

Do you have an example of a decimation algorithm or know where I can find some of that online?

---

LabVIEW has a DecimateArray function for a very simple decimation of an 1D array.

There are decimation functions in the signal processing palette. Read their help…

Use GoogleFoo to look up for other decimation algorithms like "Min/Max decimation"…