remove outliers progrmatically

[SZArjun]

I am working on a project, in data analysis of test data. My program allows the user to select the parameter for which  he wants to run the program and the relevent data is retrieved from a file. The data points sometimes are absurd, for eg when user chooses KwHr usually values are around 4 to 5, but sometimes it is 600 and 500. Also, with temperature measurement, it should be aroung -20 to 40F, but sometimes it is 300 or 400. I want to get rid of these points cos they are affecting my mean and other parameters, which are required for analysis.

How do I do it programatically?

Arjun

[Serge67]

Arjun,

If you know the valid range for your data values you can do simple data validation using 'In Range and Coerce' function from Comparison palette or you can use a case structure and only process your data when the input is valid and don't do anything otherwise

Serge 

[SZArjun]

Serge, there more than 100 parameters, and I can't specify the range for each one of them, since each of them are unique. The user chooses one of these parameteres and the analysis is carried out for that particular parameter. The outliers affect my plot significantly. Refer the attachment.

I am going to go ahead and eliminate any value that is greater than 10 times the normal value

[jcarmody]

I'd guess you don't want to coerce a spurious value; this VI will remove them.  I'd write my own analysis VI to use Chauvenet's criterion to detect outliers rather than selecting a range (unless the values you want to discard are extremely obvious).  Removing outliers should involve some statistical analysis to justify the decision.  While it's ultimately subjective, it's doesn't have to be completely so.  

[Serge67]

Well, in this case it probably makes more sense to invest a little bit of time in understanding the root cause of the issue. Why do you get sometimes absurd values from the data file? Do you read proper data type? Do you have the right read offset? You need to know your data file format in order to verify all that. The only exception is if you have to work with corrupted files, then it is going to be a pain in the neck, otherwise you should be able to avoid getting absurd values from the file in the first place.

[SZArjun]

Serge, thats not under my control. The values are generated in a factory and the probe records these values which are stored automatically. Sometimes the probe breaks and ridiculous values get stored.

JimCLD, I didn't know about Chauvenet's criterion. Thanks for that.

I'll go ahead and use the VI u suggested or just apply the principles of the criterion directly.

Arjun

Message Edited by SZArjun on 01-28-2010 11:52 AM