Formatting Data Using The Timestamps in an Array

[Bard]

Hey,

I'm having a problem coming up with a solution to the formatting problem I have been having. I will try to give a detailed explanation of what I'm trying to do and if you need any further information please feel free to ask.

I'm posting along with this, and example of what I'm looking for including what I want the result to look like in an excel file.

Basically this is my problem:

I have multiple 2D arrays containing data for various hardware devices. In the 2D array, column 1 is designated for just the timestamp (in seconds), and column 1 is the first channel, 2 is the second etc...

Each device has its own 2D array of timestamp and data. Each device as has its own sample rate. Say .2s, .1s, and 2s. So the timestamps are obviously not aligned correctly in each array. (see sample file I've attached, channels 1, 2 and 3)

What I want to do is figure out a way to aligned all of the timestamps in to one "master" timestamp and then copying that data into the new 2D "master" array which I can then write into a spreadsheet file. 

With the example I've given, there are 3 channels with 3 seperate sample rates. I want to combine as of these to look like the result I've given.

If you guys could give me any advice as currently the way I'm doing this is extremely buggy, doesn't work with all samplerates and ends up losing data almost 90% of the time.

Thanks in advanced!

Brad

[smercurio_fc]

---

Bard wrote:

If you guys could give me any advice as currently the way I'm doing this is extremely buggy, doesn't work with all samplerates and ends up losing data almost 90% of the time.

---

The obvious question is what are you currently doing? Off-hand it seems to me that using the Threshold 1D Array followed by the Round to -Infinity will give you the index of the element that you need to peel off. You would need to create an array based on your largest "X" value and your smallest sample rate. This array can be easily created using the Ramp Pattern function. This array provides your "lookup" values for the Threshold 1D Array.

Just out of curiosity: why do you need to do this? What are you doing with the data in LabVIEW?

[Bard]

Well what I'm currently doing is exactly what you said. I find the largest timestamp (smallest samplerate), then I check whether or not I need to modify it (say for instance there are two sample rates like .2s and .5s, where you would have to use .1s as the largest time stamp), then I use the threshold 2D array function to match up timestamps and try and insert the data.

Like I said this way does work with some errors, but I'm wondering if there is a better way of doing this. This way is very hard to maintain and find errors for missing data and such.

The reason for doing this is, I'm using some drivers that were created for the company I work for, for some of the hardware we sell. Currently our hardware stores its own buffers including the time stamp on the device and I have to manually pull each of them off and then formatt them myself. I really wish there was an easier way of doing this and sometimes the files are quite huge and the samplerates are quite small (e.g. .1ms). My current way of formatting makes the process of converting the files into a .CSV file take a extremely long time (1 hour +) which is rediculous. So any tips you have that can make this faster and less complicated please let me know. Thanks.

PS I apologize I really can't post the current VI as it is incorporated into my project and won't work as a stand-alone vi. I'm merelly summing up how I do it.

[smercurio_fc]

---

Bard wrote:

Well what I'm currently doing is exactly what you said. I find the largest timestamp (smallest samplerate), then I check whether or not I need to modify it (say for instance there are two sample rates like .2s and .5s, where you would have to use .1s as the largest time stamp), then I use the threshold 2D array function to match up timestamps and try and insert the data.

---

I'm sorry, but this makes no sense. What does timestamp have to do with sample rate? Also, if you have sample rate of 0.2 and 0.5 why would you need to use 0.1? Also, there is no Threshold 2D Array function. Are you referring to the Interpolate 2D VI?

I just wrote a quick VI to do this using the Threshold 1D Array and Round to -Infinity. The problem you run into is when the same rate that's being used to generate the lookup matches the sample rate of a specific channel. In this case the Threshold 1D Array sometimes returns values like 18.999999... instead of 19 due to way numbers are stored in computers. This happens with Channel 2 of your data. 

[Bard]

Here I took some time to make this a stand-alone VI. I also inputed a test case so you can just run it and it should produce the correct result. However, I guess what I'm really trying to ask is if there is a better way to do this. Obvisously you can see that this is a lot of code for what seems like it should be just a simple thing to do. With large amounts of data this function can take a very long time to complete. Any suggestions, please let me know if you have any problems with the VI or if you need me to explain anything further. The code is a bite messing a long so I apologize.

Just insert the LCD function into the Format Data Array function where needed.

Brad

[smercurio_fc]

Yeah, I don't really quite know what you're doing there. I also don't understand why you're dealing with the matrix datatype instead of simply using a 2D array. The matrix datatype requires you to convert to a 2D array. Also, there's no reason to use EXT. DBL is fine, and you can probably get away with SGL based on your numbers.

I don't know if the attached will suit your needs, but you're welcome to play with it. 

[Bard]

Thank you for all your help. I think I may have zoned in a deeper problem I was having.

Does "Threshold 1D Array" work for integer numbers?

I'm having a problem now where if I have two samplerates 2s and 5s, because I need to have a "continuous" timestamp I need to make one that is 1s. However, when I use my function to search through the array I have a feeling something is off and I believe it is the "Threshold 1D Array" function.

My reason for thinking this is becasue if I have two samplerates .2s and .5s the functions works fine after I create the new .1s timestamp. Finds all the correct timestamps and lines them up perfectly.

Any ideas?

Brad

[smercurio_fc]

The Threshold 1D Array's "threshold y" input is a DBL. If you wire an integer datatype (i.e., blue wire) to it then it will be coerced to a DBL. However, just having the value "2" in your array does not make it an integer. It's still a float. I can't say what your problem is since your code seemed more complicated than it needed to be so I did not inspect it in detail. Does the same thing happen if you use my code?

[Bakk_Diagram]

Hi Bard,  I think it would be more efficient for you to do this at the time you write the data to a file but the process is the same. 

Try this.  Create an array that is the same size as your largest array and make the column dimension the total number of signals you have.   For each timestamp of the largest array, subtract the first timestamp of the other arrays from this timestamp.  Use a conditional to check if the sign of the result is negative.  If it is, write that sample and the corresponding sample from the largest original array, to the larger agregate array in the appriopriate columns.  Otherwise add only the one from the large array.  This will ensureyou are in the closest spot to the true timestamp in the agrregate array.  Delete anysample you write from the original array.  Rinse and repeat until all samples are added to the agrregate array.