How can I sort the contents of a channel by timestamp?

[Mattrad]

I have data that's out of order and the time traces look awful because the line cuts across the figure horizontally. Basically what I want to do is that I have two channels: a signal and a timestamp.

[2.10,3.10,4.10,10.10,5.10]

[1:01,1:02,1:03,1:10,1:04] 

but I want A) put the time channel in order and B) put the signal in the same order. So the result should be

[1:01,1:02,1:03,1:04,1:10]

[2.10,3.10,4.10,5.10,10.10]

This is really simple in Matlab so I thought it would be do-able in Diadem, but I haven't been able to find anything in documentation.

[Brad_Turpin]

Hi mattrad,

You can use the ANALYSIS function "Sort Channel Values", found in the "Channel Functions" palette.  Here's a screenshot of using the time channel to sort the rows of both the time channel and the data (Engine_Speed) channel, so that the time channel values are ascending.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Mattrad]

so this will sort the time channel by ascending order and also the engine_Speed channel by the same order that time was sorted by? Or will it also sort the engine_speed channel in ascending order?

If I click the "include all channels" button, will that only apply to that group, or will it apply to ALL channels that I have loaded?

[Mattrad]

I'm attempting to use the "Sort Channel Values" analysis option with a large-ish file (~9GB) and it is very slow. 

Is Diadem designed to work with data like this? This seems to be a reoccurring problem with me.

[Brad_Turpin]

Hi Matt,

Yes, this ANALYSIS function will sort the time and data channels row by row, using the time values to determine order.  If you select the "Include all channels in sort" radio button, then that will include all channels in all groups in the sort.  If you want to sort just the channels in one group, you'll need to highlight those in the Data Portal and drag them into the selected channels field (or you can just drag the group itself) and choose the "Include the following channels in sort" option.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Brad_Turpin]

Hi Matt,

What type of data file is it that is 9GB large?  Is it a binary data file like a TDM or TDMS or MDF file?  Or is it an ASCII or Excel or XML file?  There are multiple ways of loading data files into DIAdem, and some will cause this analysis to run slower than it needs to.  How much RAM does your computer have available when you start the sort?  Are you running a 64bit version of DIAdem?

The short answer is that DIAdem is designed to work with 9GB data files.  Hopefully there will be some things we can do to improve your experience.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Mattrad]

These are TDMS files. My computer has 8 GB of RAM, and I am using 64-bit diadem.

[Brad_Turpin]

Hi Mattrad,

The good news is that the TDM file will give you fast, random-access to the data you want to load.  When DIAdem loads data from files into the Data Portal, it always converts the data values from whatever data type they were in before to a 64bit DBL data type.  So if your 9 GB data file stored data values as I32s or SGLs, then it would take up 18 GB to load it into the Data Portal.  Loading I16s or U16s takes up 4 times as much RAM, etc.

I suspect that your slow execution speed has to do with virtual (hard disk) memory use when exceeding the available RAM.  Have you tried loading only half or a fourth of the channels from that 9 GB TDM file and sorting them?  If that helps, you could save each sorted block of channels to a separate TDM or TDMS file and then merge them afterwards.  Or you could try the sorting calculation on a computer that has more than 9 GB of RAM available.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Mattrad]

I am now using a computer with significantly more RAM (and a better processor) and these functions work MUCH, MUCH, faster.