Time aligning multiple signals

[jeanlucpicard]

Hello,

I have some signals recorded with two separate programs (5-6 hours each) and would like to combine both time-aligned for analysing them together.

Currently I do have a txt file with 4 channels, 100 Hz sampling rate and about 5-6 hours length and I would like to merge it with another file of similar properties. Only problem is that these files are not completely sampled at the exact same timepoints, since they were recorded by separate programs. How can I match these so they could be imported back to my analysis program? (Unfortunately the analysis program that I'd use can't do this, I have to do it myself)

I tried so far changing the imported data to waveforms with defined t0 for both arrays of signals and then use "Align (single-shot)" vi but for some reason it gave back empty arrays. I do know, these files are huge, and I had to go to LabVIEW 2015 64-bit to be able to completely read them, so I can't use for example Advanced Signal Processing Toolkit (I could not find a 2015 version for it, only 2014 and 2016).

Thanks

[wiebe@CARYA]

Is the dT off or just the T0?

Correcting an offset\shift is a hole lot easier then aligning different dT.

[jeanlucpicard]

their dt is identical (0.01s - 100 Hz), they just have different t0-s

both their times are correct on their own, I'd like to align them so they'd be completely aligned and I could export them together (in a file where there is a timepoint and all channel data that corresponds to that timepoint, for each timepoint)

[Bob_Schor]

This is a somewhat complicated question, as there are multiple "unknowns" (to me, but probably not to you).  In this discussion, I'm going to assume you have two Files, A and B, to merge.

1. Do you "trust the clocks"?  In particular, are all of the A samples continuous with a reliable fixed sampling frequency, and are all of the B samples similarly continuous with a reliable fixed frequency?
2. Assuming the answer to #1 is "Yes", are the two clocks the same?  In this "best of all possible worlds", all you now have to do is align the two signals by using t0.  You'll need to either "trim" the overhanging samples (eliminate them) or "pad" the missing samples (I'd recommend the "trim" option to simplify the rest of the computations).  You will still have some small time offset, but if it is less than a clock tick, you can probably "know, but ignore" it.
3. If the clocks are not the same, then you have the problem of creating two signals with the same clock.  Look up "resampling", a technique for doing this.
4. When you are done, you will have an aligned 2D array of the A and B samples.  Now build your new Waveform using the common t0 and "compromise dt" (if you had to do resampling).
5. 6 hours at 100 Hz is a pretty big array (as you note) ...

Bob Schor

[wiebe@CARYA]

---

@Bob_Schor wrote:

1. Do you "trust the clocks"?  In particular, are all of the A samples continuous with a reliable fixed sampling frequency, and are all of the B samples similarly continuous with a reliable fixed frequency?

---

That was what I was trying to find out.

For instance, two cRIOs can sample at 100 Hz, that doesn't give them the same sample rate. If the PC is also sampling at 100 Hz, it will be even more off.

If you know absolutely sure the dTs are the same (or can live with the errors), I'd use the "Read Lines" option of the Read From Text File. Simply read each text file line by line, and do the processing line by line. If Ta>Tb, stall reading from Ta until Ta=Tb, and vice versa. This would prevent reading the entire file into memory.

[jeanlucpicard]

Thanks for all the comments, I am trying to answer all of them:

1. I trust the clocks enough. They are recorded at the same computer, so at least I don't have to also deal with basal clocks being slightly off. File A is recorded by a Powerlab (a commercial bio-medical DAQ with hardware sampling) and the File B is recorded from a serial port which spits out data points spaced 100 Hz. I'd say, from my point of view they can be considered good enough and I can probably live with the potential differences. (actually File A is sampled at 1 kHz, but I can live with a decimation to 100 Hz at export-time and won't lose probably anything with it)

2. They are recorded on the same computer, but have some differences based on the two clock ticks not happening simultaneously (e.g. A would sample at 0.000 sec, while for B it would be 0.004 s, then 0.010 s and 0.014 s, and so on), but they have same dt, so this difference is quasi constant (I can consider it to be constant)

Read by line doesn't sound bad, although at one point I'll also have to make a text file out of it, but I could also do that line-by-line (and probably should)

What I am trying to achieve is to have a "merged" text file (A+B) which has signals from both files time-aligned (so anything that happens at sort-of one time-point [within a clock] should be merged to one single line of the txt). Unfortunately my analysis program only eats text if I want to keep some metadata during the process (like event markers).

[Kevin_Price]

Several thoughts, not all mutually consistent.  Here goes:

1.  Your 2 data sources are very likely *not* using the same base clock.  You have a biomedical DAQ with its own hardware sample clock.  And you have some serial instrument that streams at what *it* thinks is 100 Hz based on some clock of its own.

   What I've personally observed when I've compared timing sources (typically a PC time-of-day system clock vs. a DAQ sampling clock) has typically been a time discrepancy of single digits worth of seconds per day.  Specs would allow even more discrepancy, but that doesn't seem typical.

   Consequently, over the course of 5-6 hours you shouldn't be surprised if your two timing sources disagree by a second or two.

2.  For future testing, you should come up with a scheme that helps you with this problem *before* writing data to file.  I don't know what options you get with "Powerlab" though.  The main idea is to capture a PC system time as close as possible to both the first and last sample taken.  You'd do this for both the DAQ and the serial stream.  You'd also know the actual # of samples taken from each device during those intervals.

   This won't be perfect.  You'll probably have at least 10's of msec uncertainty in the start and stop times.  But at least you'll be able to calculate the two devices' average sample rates as measured by the *same* clock.

   Further, I would expect the clocks to tend to maintain the same amount of discrepancy with one another over days and weeks.  (Note: this isn't a promise, just an anecdotal observation).  So you could conceivably measure this discrepancy for a day and then have some optimism that you could apply the result to your existing data files.

3.  I've had a number of apps where I combined live hardware-clocked data and serial data into a simple Excel-compatible ASCII .csv file.  Usually the serial data updates considerably slower.  In that situation, I usually just keep repeating the last-known-true serial data value until I get a new one.

  The same kind of idea could be another possible strategy for you to use while post-processing your 2 separate files into 1 merged one.

-Kevin P

[wiebe@CARYA]

---

@jeanlucpicard wrote:

Read by line doesn't sound bad, although at one point I'll also have to make a text file out of it, but I could also do that line-by-line (and probably should)

---

Indeed. No matter how the rest of the problems are fixed, that would probably be the basis of every solution.

[wiebe@CARYA]

---

@jeanlucpicard wrote:

Thanks for all the comments, I am trying to answer all of them:

2. They are recorded on the same computer, but have some differences based on the two clock ticks not happening simultaneously (e.g. A would sample at 0.000 sec, while for B it would be 0.004 s, then 0.010 s and 0.014 s, and so on), but they have same dt, so this difference is quasi constant (I can consider it to be constant)

---

If all hardware is software timed, or averaged based on a software timer, then you might be OK.

Software timing is no guarantee though. A DAQmx simulation channel for instance has a deviation of (I think) one sample per simulated trunk. So when  simulating 100 samples at the time, there is a 1% time deviation. Doesn't sound like much, but at the end of a day, that's 14 minutes. Good luck explaining that to a customer!

Ideally, all hardware should use the same clock. Expensive HW has 10 MHz clock inputs (and outputs). Use a (firefly) GPS clock reference or a Caesium atomic clock ref, and you're done!

I made a "universal logger", and used a PID regulator for each channel to tweak the dT of samples... Not easy, not perfect, but worked like a charm.