I've some data in 6 channels (3 time, 3 data) with a length of appr. 70000 (will be more soon). Due to some problem our LabVIEW software engineer will solve in the future I get sometimes values far to small or far two large.
So I'd like to delete them. The charactersitics are:
value nearly 0 with the surrounding two values being "resonable" larger, or
value almost 10000 with the surrounding beeing much smaller (ok here I could tell an expected maxiumum plus some safety).
Because of the amount of the data I'd like to use an internal DIAdem function. I wrote a quick and dirty VBS script, but it is far too slow.
Just a smoothing doesn't do the trick, esp. because of the large values which are too far out of spe c.
It will set all spike values to NOVALUE which marks not correct values. After that you can use the Novalue function in DIAdem to interpolate or eliminate the marked values.
But maybe that this doesn?t solve your specific problem and because of complexity you must use a small VBScript - therefore some tips in general:
Calculations with channels are much faster than loo ps (see function above). If a loop is necessary than use CHDX instead of CHD this is much faster, too. After working with CHDX it is necessary to set the channel characteristics (Call ChnCharacter(ChnName) or Call ChnCharacterAll)
in 1999 i wrote a small AUT-Script "SpikeEx.aut" (German).
You have to select a pair of channels (XY) and a absolute jumpwidth of the spikes. All Spikes (up and down) will be deleted. In some cases there are Spikes longer than 1 Sample. The Script is realized as loop to kill those Spikes. In L8 a max of 5 Loops is defined. At the end of the script a "linear interpolation" is included to expand the channels to original length.
ChnDel(cno("MyChan_left")) ChnDel(cno("MyChan_mid")) ChnDel(cno("MyChan_right")) "channels.sud" lets you select two channels (just the standard NI solution). If you want to use this script you might have a look at some commands, because this one is written for DIAdem 9.0 Beta (I'm using a Citadel 5 database as source).
This works very well, because I have spikes of 1 datapoint width always and an interpolation is not needed for me. As all the work is done in "formulacalc" it is pretty fast...
For spikes longer than one datapoint a more flexible solution like yours is definitly needed.