cancel
Showing results for 
Search instead for 
Did you mean: 
Reply

How to sort data in a TDMS file

Highlighted

How to sort data in a TDMS file

This is my first application utilizing TDMS. 

I need to read multiple CSV files of collected data and play them back syncronously. Fortunatly each csv file essentially represents one channel of streammed serial data.

First I would like to place the data into a datastore so that I ca better analyze the data and view the data.

 

I have a few problems. 

First, the data packets in the file are not in order. I would like to sort the data based on the "Time" data. Unfortunatly the time data is not periodic, so I can not use the delta atribute of TDMS.

 

If I was using SQL I would do a SQL querry with a sort by DSC, but I don't see any such option using TDMS.

Is there an equalivent to do this when reading back from a TDMS datastore?

 

I'm currently using LV2011 Professional Dev System.

Below is a sample of how I'm collecting the data so that you can better understand my TDMS data. (this is not my actual application)

 

When I view this data the time data can be all over the map. I really want to sort this, so that "Time" is descending. 

 

Before anybody asks, No, I can not change the format of the CSV files. I can probably do some pre processing of the data before I go to TDMS, but these CSV files are very big. In my actual application I'm doing a Read by line, because the CSV files can be in 10's or 100's of Megabytes, and I have hundereds of these CSV files.

 

(thus the traditional programatic sort will not work very well, because there would be tons of data that would need to be sorted. Somehow SQL does this very fast) 

 

Any ideas?

 

Regards,

 

 ExampleOfMyTDMS.png


Engineering - The art of applied creativity  ~Theo Sutton
0 Kudos
Message 1 of 15
(1,511 Views)

Re: How to sort data in a TDMS file

Have you seen NI's White Paper on TDMS?  I think a properly-formatted and organized TDMS file can be easily inspected and sorted using tools such as Diadem.  Of course, if you've already got tons of data written as simple .csv files, making them no "smarter" than comma-delimited text "Spreadsheet" files, you may have a problem.  You may find that some "massaging" of your data using some of the tools and ideas in the White Paper will help.

 

Bob Schor

0 Kudos
Message 2 of 15
(1,502 Views)

Re: How to sort data in a TDMS file

See https://decibel.ni.com/content/docs/DOC-11313 to write data events to a tdms file. 

0 Kudos
Message 3 of 15
(1,496 Views)

Re: How to sort data in a TDMS file

Events are not really what I'm looking for.

I don't think I'm fully explaining myself.

 

So lets say I have a csv file called exampleData.csv with the following data.

548.156,A45BCE,First Sample
548.160,123EEE,Second Sample
429.011,563EAC,Third Sample
549.111,ABC123,Fourth Sample
423.001,DDDEFF,Fifth Sample

 

Where the first column is time. Notice that time is not linear.

When I use the TDMS File Viewer I get the following. Notice how the TDMS File Viewer shows the data in a first come first serve basis. 

 

firstcomeFirstServe.png

 

 

I want the data sorted by the "Time" data where I would get something like this.

In this picture I am doing a 2D sort of the data before it goes to the table. I can't do this with my data because there is alot of data.

 

 

GoodSorting.png

 

In SQL world I could do something like this

~~~~~~~~~~~~~~~~

SELECT * FROM CSVFileTable
ORDER BY Time DESC;

~~~~~~~~~~~~~~~~

 

Is there no equlivent to TDMS??

 

 


Engineering - The art of applied creativity  ~Theo Sutton
0 Kudos
Message 4 of 15
(1,484 Views)

Re: How to sort data in a TDMS file

Your TDMS data is an array of data -  Create an array of time stamps from your data. Use the Index and Bundle function and the sort the array.  Your tdms data will be the broken line with the proper index for the time stamp. Sorry for the broken lines but I do not have your data structure.

Index and Bundle Array.png

0 Kudos
Message 5 of 15
(1,478 Views)

Re: How to sort data in a TDMS file

 

@MrQuestion wrote:

Events are not really what I'm looking for.

I don't think I'm fully explaining myself.

 

So lets say I have a csv file called exampleData.csv with the following data.

548.156,A45BCE,First Sample
548.160,123EEE,Second Sample
429.011,563EAC,Third Sample
549.111,ABC123,Fourth Sample
423.001,DDDEFF,Fifth Sample


 

 

I'm confused.  You are talking about writing TDMS files.  If you wrote exampleData as a TDMS file, you should have two files, exampleData.tdms and exampleData.tdms_index.  But you should not have a file with the .csv extension!  So what is going on?

 

I don't have Diadem installed on this PC, but I bet it could take a true TDMS file, with the index, and sort it for you ...

 

Bob Schor

 

0 Kudos
Message 6 of 15
(1,471 Views)

Re: How to sort data in a TDMS file

Actually he did clarify this Bob. He already has the data files, full with numbers and in the range of 100s of Mbs. The data files he already has (he's not writing new data)  have a .csv extension and are sorted in a "first come first served" basis, regardless of when the data was generated, he wants to sort them by timestamp, and write that to .tdms

 

So on that topic I'm certain some pre-processing is needed, I particularly like the sort aray idea.

0 Kudos
Message 7 of 15
(1,466 Views)

Re: How to sort data in a TDMS file

I guess I'm questioning the Title of this post -- if this is a TDMS file (presumably made by LabVIEW, as per the attachment the OP included), why are the Index files missing, and why is the Extension "wrong"?  If it truly is a TDMS file, I think there are "TDMS Tools" that will do the equivalent of his SQL question, namely insta-sort the data.  But since I'm not that familiar with TDMS, and have only played briefly with Diadem, I'll shrug and let others figure this out.

 

Bob Schor

0 Kudos
Message 8 of 15
(1,462 Views)

Re: How to sort data in a TDMS file

The TDMS Viewer is a VI that you could make a copy of and modify to your needs.

 

So for instance, in the pictures you show where you want to sort data in that one tab, you could modify the Viewer so that it sorts the table before it writes it to the table control.

0 Kudos
Message 9 of 15
(1,453 Views)

Re: How to sort data in a TDMS file

I am making the TDMS database myself.

The CSV files are given to me. 

Since I have so much data I am writing the data line by line to the TDMS datastore as I read the csv file. 

The reason that I'm wanting to use a datastore is because I will need to eventually perform analysis and playback of the data on real hardware. 

 

I'm not entirely sure how I could do pre-processing without buffering and storing all of the data in a mega array and performing a sort on the "Time" data.

 

In the future I will need to merge different datasets together. This will only make the sorting function that more tedious. 

 

 

It would appear that TDMS is optimized for continues channel samples like what you would measure from DAQ. In such a world why would you add a sort function when it is assumed that each sample is coming after the prevous sample. 

 

Perhaps TDMS is not the solution that I need.

I am impressed with the very fast write times, and the capability of asynchronous data writes, but I don't think it's really meant for storing data words from a serial bus from multiple sources.


Engineering - The art of applied creativity  ~Theo Sutton
0 Kudos
Message 10 of 15
(1,449 Views)