Fastest method to get the amplitude and phase in a signal with lots of mixed similar frequencies

[obarriel]

I am working on a really complicate signal that contains lots of mixed similar frequencies. I am interested in obtaining the  amplitude and phase of several particular frequencies, all of them in the KHz range (for example the 50000 Hz) component. But the signal contains many other frequencies that are very close to the interesting ones (for example 50005 Hz), and I don't want them to interfere the mesurement of the "interesting ones". I know that miracles doesn't exist and the basic approach here should be to increase the adquision time to have a better frequency resolutionn after the FFT. Howewer I wonder if there is some extra advice or alternative to the FFT applicable to this particular situation to help making the measurement as fast and accurate as possible, since time-efficiency is essential in the application that I am working. 

Thank you in advance.   

[obarriel]

Any advice? Now I wonder if there exists any kind of sharp filter could be added.

[johnsold]

As you mentioned in your original post, you cannot expect miracles.

I have more questions than answers at this point.  What is your sampling rate? How often do you need updated outputs?  How many signals (both desired and undesired) are in your data? Is the 1 part in 10000 you cited in your example the closest spacing of signals you need to resolve?  If not, what is the closest?  What is the highest frequency signal in your data? The lowest frequency? What is the greatest amplitude difference between two signals "adjacent" in frequency?

There are several things you can try, but the choice will be driven by the nature of the problem.

[obarriel]

I give some answers

What is your sampling rate?

Usually 1MHz or 2MHz (I can go up to 3MHz, the maximum permitted by my 16bits A/D board)

How often do you need updated outputs?

In a previous similar problem I worked but with many less "undesired" signals (i.e. with much cleaner adquisitions) I worked with datablocks of 16384 points. Ideally, I would like to keep similar updating times, but I know is going to be impossible since I need accurate measurements.

How many signals (both desired and undesired) are in your data?

Desired there are 15, ranging between 5kHz and 150kHz aproximately. Undesired there are many many more.

Is the 1 part in 10000 you cited in your example the closest spacing of signals you need to resolve?  If not, what it is the closest?

Yes, it is about the closest. But to be safe I would say that I need to resolve 1 part in 50000.

What is the highest frequency signal in your data? The lowest frequency? What is the greatest amplitude difference between two signals "adjacent" in frequency?

My highest desired frequency is 150kHz and the lowest 5kHz. The undesired ones extend over a broader frequency range and although many of them are highly attenuated (particularly those out of this frequency interval) some of them have amplitudes comparable  to the desired ones.  

Thank you!

[johnsold]

You did not get an easy problem!

I may have some time in several hours to think about this.  My initial reaction is that you may have some possiblities, but I need to work through some things before I make any suggestions.  One key point: Make sure you have a good anti-aliasing filter ahead of your A/D converter.  Since you have significant energy in the interfering signals outside the frequency range of the desired signals, you need to be very careful that you avoid any aliasing.  An aliased signal cannot be "fixed" by any type of processing after the conversion so you need to prevent it from happening.

Lynn

[johnsold]

Here are the results of some number crunching I did.

I am assuming a sampling rate of 1 MHz and a data set size of 16384 samples retrieved from the A/D converter at a time.

1. Update rate is approximately 1 data set every 17 ms. So this is the target time for processing the data.

2. The frequency resolution (df) of the FFT is df = fs/N, where fs is the sampling rate and N is the number of points in the data set.

2.a. df = 1E6/16384 = 61.04 Hz.  This is obviously much larger than the desired 1-5 Hz resolution.

2.b. If df = 1 Hz, then N = 1E6.

3. On my computer a 16384 point FFT takes ~ 0.5 ms.  A 1E6 point FFT takes ~105 ms.  The largest FFT it can perform in 17 ms is about 3E5 points. These numbers can vary strongly with small changes in N.  Check carefully whatever numbers you choose.

4. If N = 3E5, then df = 3.3 Hz.

5. Using the Size input to the FFT VI to zero pad is slower than feeding a large array to the FFT with Size set to -1.

This choice of N (= 3E5) would make it (almost) possible to get both the resolution and the speed you want.  Obviously you cannot collect 300000 points at a time or you would have to wait 300 ms.  You could use a circular buffer where you insert the 16384 new points at one end and remove the same number of old points from the other end every time a new acquisition occurs.

Do you have any knowledge about your signal source which might be used to advantage?  For example knowing the exact frequencies of the desired signals or that two signals always are correlated might be useful.  What generates the signals and what are you doing with the data after the analysis?

Lynn

[obarriel]

Thank you for your numbers Lynn

In fact the FFT aproach I am using is a little more complex.  I run 15 different FFTs, one for each desired component (I know this is slow!, but it seems the only way to be accurate).

I know with good precision (better than 1Hz) the frequencies of the desired components; so what I do is to adjust, for each frequency, the number of points of the data so that the desired frequency corresponds (approximately) to an integer number of times df. I use this simple program for this:

The reason for doing this is that I need very accurate mesurement of the amplitude of each component, and if I don't use this "adjustment" for each particular frequency the results given by the FFT are quite inacurate, ever for large

data blocks.

The signals are generated by a combination of photoelastic modulators, and the amplitude of each desired component contains different physical information.

I am not using now any anti-aliasing filter, could you recommend one?

[johnsold]

So what you really need is 15 lock-in amplifiers?

Even with your adjustments for the frequency bins, you probably have some unknown amount of spectral leakage. (That is what the effect of the signal frequency not being exactly in the middle of the frequency bin is called).  You may also be getting some leakage into your desired bin from the interfering signals because your adjustment will be wrong for them.  Thus the energy in those signals may spread over multiple bins, including the one with your desired signal.

Do you have access to a reference signal at the frequency (and phase) of each of the desired signals?  Ideally it would be the same signal which excites or modulates the component.  If you have these references it may be possible to get much better information without an FFT at all.  Are the frequencies rationally related, that is are the frequencies generated by digital techniques from a common reference frequency?

Lynn

[obarriel]

So what you really need is 15 lock-in amplifiers? Yes!

Yes, I have access to a reference signal at the frequency (and phase) of each of the desired signals.

Are the frequencies rationally related, that is are the frequencies generated by digital techniques from a common reference frequency? Yes, they are rationally related. All the frequencies (both desired and undesired) are generated from four common references that I have available.

I have also worked with software lock-in like techniques but although they are more time-efficient that the 15 FFTs they are not much better in terms of measurement accuracy, again due to the the tiny separation between frequencies.

[johnsold]

I am about to leave for the day, but I will try to post some additional information over the weekend.  What I am thinking about is synchronous filtering.  It has been a few years since I last worked on the technique, but I think it could be fast, depending on bandwidth effects.

Lynn