Noise Spike Detection Tool

[ATarman]

I would like a tool that would help identify noise spikes in my data, so that they may be set to NO VALUE or carefully interpolated.  In the attached example, I have two torque spikes in my data that are significantly higher than the mean.  If I were only looking at one file, I would simply "Set Flags" in VIEW, then "Flags: Remove Data Points". However, I am reviewing dozens of files with several channels of interest.  

Peak data points are the most damaging to the components under test, so it is important that we keep the real events and reject the noise.  

Concerns with other peak / spike removal options:

• Manual Set Flags -> Flags:  Remove Data Points; time required
• ANALYSIS -> Event Search -> Event = Window -> Upper Limit:  In other data files, I have real events with values higher than the noise in this data file.  
• ANALYSIS -> Digital Filters or Smoothing:  This will change all of the data - it will likely narrow the peak-to-peak of my other data and interpolate my noise spikes, adding damage that is not actually real.  I only want to remove a few data points in the channel data. 

nCode Glyphworks has a nice Spike Detection Tool that uses 6 available algorithms to detect spikes.  Once it identifies them, it allows the user to see them, prior to performing any actions - this is important, because it allows the user to identify if they are real or noise, prior to taking action.  

[AndreasH_Ha]

Hello,

would it be possible to share an example data set so that we can try out some more options. We have several ideas we would like to test...

Thank you

[ATarman]

Thank you for your reply.

I have a sample file, but in a comment, I can only upload photos or video, not a .zip or .tdms file.....

Where should I upload to?  

[Walter_Rick]

You can upload your data to the NI FTP server:

ftp://ftp.ni.com/incoming

Greetings

Walter

[ATarman]

Thank you, Walter - I've uploaded TorqueSpike.zip to ftp://ftp.ni.com/incoming

[AndreasH_Ha]

Hello,

thanks for sharing the data. I think the best you can do is use a simple algorithm to detect the outliers. Assuming your example data set is representative for your data in General you could use "z-score". 

Please see the attached example. It uses the algorithm to flag the data Points so that you then can do the next step in VIEW. If you decide that the algorithm is stable enough for your purse, you could simply delete teh outliers directly from the script

Dim     oChnY
                 
Call LogFileDel()
Call PrepareData(CurrentScriptPath & "TorqueSpike.tdms",oChnY)
Call FindOutliers(oChnY)
'-------------------------------------------------------------------------------
' Prepare/load data
'-------------------------------------------------------------------------------
Function  PrepareData(sgFilename,oChnY)
  Call Data.Root.Clear()
  Call DataFileLoad(sgFilename)
  Set oChnY = Data.GetChannel("TORQUE@Input_TQ.RN_1")
End Function
'-------------------------------------------------------------------------------
' Find Outliers
'-------------------------------------------------------------------------------
Function FindOutliers(oChnY)
  Dim     oaEvents,oaEventsUpper,oaEventsLower,K
  Dim     LowerLimit,UpperLimit
  Call ChnStatisticsChannelCalc(oChnY, eStatsArithmeticMean+eStatsStandardDeviation, 0, 0, False, False, False, "NameName")
  ' Define a window which represents the mean value +/- 3 times the Standard deviation. 
  ' This is based on teh algorithm called "Z-Score"
  LowerLimit = StatsResult(eStatsArithmeticMean) - 3*StatsResult(eStatsStandardDeviation)
  UpperLimit = StatsResult(eStatsArithmeticMean) + 3*StatsResult(eStatsStandardDeviation)
  ' Find value above the window ...
  oaEventsUpper = ChnEventDetectionWindow(,oChnY,UpperLimit,NULL)
  ' ... and below
  oaEventsLower = ChnEventDetectionWindow(,oChnY,NULL,LowerLimit)
  ' and combine the two results into one list
  oaEvents = ChnEventOperationOR(oaEventsUpper, oaEventsLower) 
  ' Flag the values to prepare interavtive inspection with view  
  For K = 0 To UBound(oaEvents,1)
    Call ChnFlagSet(oChnY, oaEvents(K,0), oaEvents(K,1)-oaEvents(K,0)+1, True) 
  Next
End Function

[markwkiehl]

Here is a full functional View embedded dialog to help manage spikes / outliers.  Please try and give me feedback.  Enjoy!

http://www.savvydiademsolutions.com/apps.php?topic=ana-spike-outlier-mgmt 

[ATarman]

markwkiehl - I like it.  Thank you very much!

[Walter_Rick]

Hello

Thank you for your request. R&D has read this suggestion and it will be included in feature planning discussions for future DIAdem versions.

Greetings

Walter