DIAdem

cancel
Showing results for 
Search instead for 
Did you mean: 

Removing Spikes

Hi all,

I've some data in 6 channels (3 time, 3 data) with a length of appr. 70000 (will be more soon). Due to some problem our LabVIEW software engineer will solve in the future I get sometimes values far to small or far two large.

So I'd like to delete them. The charactersitics are:
  • value nearly 0 with the surrounding two values being "resonable" larger, or
  • value almost 10000 with the surrounding beeing much smaller (ok here I could tell an expected maxiumum plus some safety).
Because of the amount of the data I'd like to use an internal DIAdem function. I wrote a quick and dirty VBS script, but it is far too slow.

Just a smoothing doesn't do the trick, esp. because of the large values which are too far out of spe
c.

The values I want to delete are anytime "alone".

Thanks for any ideas,
Carsten
0 Kudos
Message 1 of 8
(4,312 Views)
Hi Carsten,

I don't know the real course of our channel pairs, so it is a little bit difficult to provide a solution which works fine directly. But, maybe I can give you some ideas.

If there are really spikes you can use this formula in VBScript:

R1 = 9000 ' Threshold value
call formulacalc("ch('NewResult') := (ch('SpikeChn') > R1) * NoValue + (ch('SpikeChn') <= R1) * ch('SpikeChn')")

It will set all spike values to NOVALUE which marks not correct values. After that you can use the Novalue function in DIAdem to interpolate or eliminate the marked values.

But maybe that this doesn?t solve your specific problem and because of complexity you must use a small VBScript - therefore some tips in general:

Calculations with channels are much faster than loo
ps (see function above).
If a loop is necessary than use CHDX instead of CHD this is much faster, too. After working with CHDX it is necessary to set the channel characteristics (Call ChnCharacter(ChnName) or Call ChnCharacterAll)

I hope this will help you.

Greetings

Walter Rick
0 Kudos
Message 2 of 8
(4,312 Views)
Hi Walter,

I know that working on channels is much faster, that's why I'm looking for a solution without a VB loop.

The formula above will work well for the values beeing too large. But for the values beeing too small I need to do a test.

Right now I do something like

for i=2 to chnlength(mychannel) - 1
if (chd(i - 1, mychannel) > 100 * chd(i,mychannel)) and (chd(i+1,mychannel)>100*chd(i,mychannel)) then
chd(i, mychannel) = Nv
end if
next

For the low "spikes" I assume I can't work on the channel but I need to use a loop.

So I'll optimize a bit. Let's see...

Thank you very much,
Carsten
0 Kudos
Message 3 of 8
(4,312 Views)
Hi Carsten,

what do you think about this:

' copy MyChan
call formulacalc("ch('MyChan_1') := ch('MyChan')")
' delete first value to have MyChan minus 1
Call DATABLDEL(cno("MyChan_1"),1,1)

call formulacalc("ch('MyChan_2') := ch('MyChan_1')")
' delete first value to have MyChan minus 2
Call DATABLDEL(cno("MyChan_2"),1,1)

call formulacalc("ch('NewResult') := (ch('MyChan_2') > 100 * ch('MyChan_1') and ch('MyChan') > 100 * ch('MyChan_1')) * NoValue + ch('MyChan_1')")


Greetings

Walter Rick
0 Kudos
Message 4 of 8
(4,311 Views)
Hi Walter,

yeah that could do the trick! Nice idea -- I'm not very familar with the formulacalc syntax (much more with VBS as it is more like "normal" programming).

I'll test it. But in my experience so far with DIAdem the channel functions are that far compared to VBS loops, that any complex structure with channel operations will be faster.

Thanks again,
Carsten
0 Kudos
Message 5 of 8
(4,311 Views)
nb m
0 Kudos
Message 6 of 8
(4,311 Views)
Hi Carsten,

in 1999 i wrote a small AUT-Script "SpikeEx.aut" (German).

You have to select a pair of channels (XY) and a absolute jumpwidth of the spikes.
All Spikes (up and down) will be deleted.
In some cases there are Spikes longer than 1 Sample.
The Script is realized as loop to kill those Spikes.
In L8 a max of 5 Loops is defined.
At the end of the script a "linear interpolation" is included to expand the channels to original length.

Hoping you´ll enjoy this old hack

Martin
0 Kudos
Message 7 of 8
(4,311 Views)
Hi Martin,

thanks for your answer.

My solution right now is:


SudDefLoad("channels.sud")
Suddlgshow("Dlg1")

call formulacalc("ch('MyChan_left') := ch(L2)")
call databldel(cno("MyChan_Left"),1,2)

call formulacalc("ch('MyChan_mid') := ch(L2)")
call databldel(cno("MyChan_mid"),1,1)

call formulacalc("ch('MyChan_right') := ch(L2)")

Call formulacalc("ch('Mytime'):=ch(L1)")
Call DATABLDEL(cno("Mytime"),1,1)
Call DATABLDEL(cno("Mytime"),cl("mytime"),1)
ChnFormat(cno("Mytime"))=ChnFormat(L1)

call formulacalc("ch('NewResult') := ((ch('MyChan_mid') > 100 * ch('MyChan_left') and ch('Mychan_mid') > 100 * ch('MyChan_right')) or (100 * ch('MyChan_mid') < ch('MyChan_left') and 100 * ch('Mychan_mid') < ch('MyChan_right'))) * NoValue + ch('Mychan_mi
d')")


Call ChnNovHandle(ch("Mytime"), ch("NewResult"), "Delete", "XY", 1, 0)
Call ChnMove(ch("Mytime"),ChnGroup(L1))
ChnName(ch("Mytime")) = ChnName(L1) & " processed"
Call ChnMove(ch("NewResult"),ChnGroup(L2))
ChnName(ch("NewResult")) = ChnName(L2) & " processed"

ChnDel(cno("MyChan_left"))
ChnDel(cno("MyChan_mid"))
ChnDel(cno("MyChan_right"))


"channels.sud" lets you select two channels (just the standard NI solution). If you want to use this script you might have a look at some commands, because this one is written for DIAdem 9.0 Beta (I'm using a Citadel 5 database as source).

This works very well, because I have spikes of 1 datapoint width always and an interpolation is not needed for me. As all the work is done in "formulacalc" it is pretty fast...

For spikes longer than one datapoint a more flexible solution like yours is definitly needed.

Cheers,
Carsten
0 Kudos
Message 8 of 8
(4,312 Views)