Significant digits from Property Loader

[jchretien]

Hello guys,

I'd like to know if anyone already faced that issue, if I load a certain limit from property loader, for example Minimum "10.0", and the VI returns a measure of 9,999472894, the test should pass.

Is there some easy way to detect the significant digits from the property loader, and automatically apply that rounding to the measured values? Actually it would become heavy to program it test by test.

Perhaps in my .xls property fils I could put an integer for each limit that represents the number of significant digits...

Any better idea?

[dug9000]

Why should the test pass if the minimum is 10.0 and the value 9.999472894? TestStand already does comparisons within a particular tolerance value, generally the tolerance is within the precision of a double precision floating point number (approximately 15 significant digits). 9.999472894 is likely outside of this tolerance. However you could just make your limit something like 9.999 instead if you prefer a bigger tolerance.

Hope this helps,

-Doug

[jchretien]

 Thanks for your answer, I just noticed that we can customize the numeric format and format the number of fractional digits, that's likely what I needed to know.

Have a good day

[dug9000]

The numeric format is for display purposes only. It will not affect the results of a comparison.

-Doug

[jchretien]

Oh it doesnt help at all.

9.9999 should be equal to 10.0

9.999 is smaller than 10.0000.

TestStand is mainly purposed for numeric comparisons, I don't figure why it doesn't handle some rounding easily.

[Kurt-P]

You could try:

Val(Str(Step.Result.Numeric, "%.3f"))

where "%.3f" would be the units of precision you want. 

-KP

[jchretien]

Yeah that would work for one particular test, but if our product quality criteria changes from 10.0 to 10.00, I can't just change the precision in the property loader .xls file and apply the new precision in the measurement test.

I should load the limits as strings, and put some code to detect the number of significant digits in the string and use this number as a value that I'd put in the formula Val(Str(Step.Result.Numeric, "%.(Locals.Limits.String.len)f"))

it becomes quite heavy, while the property loader is supposed to simplify the whole thing.

[dug9000]

Just trying to understand this use case better. If you really want to support 9.999 why not set the lower limit as 9.999? If you are trying to check for exactly 10 within the 4 digits of precision why not specify a GELE comparison of:

9.999 <= x <= 10.001

-Doug

[jchretien]

The 10.0 example is just an example. My cases are several values, we don't need to enumerate every tested values.

It's heavy to hardcode every test limits, I don't want my value to be arround 10.0, it's a GE (=>) Greater or Equal test (in this example).

It's just a matter of significant digits and it seems that teststand cannot handle significant digits for comparison.

Actually for one very problematic test we hardcoded our test value to 9.95 instead of 10.0. So it works for this particular case but let's imagine we have 50 configurable test values, it becomes a nightmare to hardcode every significant digits, the property loader feature loses all it's benefits.

It appears to me that most of TestStand users mostly do values comparisons using that tool, the significant digit is a very important topic and it's not handled at all.

It's not because the DBL value returned can be expressed using 5 digits that they are all significant...

[dug9000]

Correct, TestStand does not base the signficant digits of the comparison off of the significant digits of the given limits, instead it assumes you want a comparison done within the full precision of a double-precision floating point number.

To fix existing limits, you could write a converter that converts limits like 10.0 to 9.999 since it is a mechanical process that could be done programmatically. Not sure why you would have to hard code limits to fix this problem. Is it not possible to change the limits in the property loader source to 9.999 or that is not an option? If that's not an option, another idea is that we ship the source code to the property loader, you could modify it to convert limits to their precision range values, for example, if a limit of 10.0 is loaded as a lower limit, it could automatically convert it to 9.999 before setting the limit value in the teststand step. That way you wouldn't have to change any sequence files, just the property loader itself.

Not to say it wouldn't be useful to add a new feature to TestStand to match the precison of the comparison to the precision specified by the limit (or perhaps in a separate setting on the step), but that is not a common request. You could post such an idea to the idea exchange though and see what people think.

-Doug