Formula Node Floor(x) Produces Different Result

[Code Toade]

Hi, A search didn't find anything about the Floor(x) function, so... I'm using LabVIEW 6.0.2, and the Floor(x)function in a Formula Node seems to be producing different results depending on previous calculations. I've got the following 2 lines in a Formula Node:

MSS = Ref / RefDiv;
MDN = floor(RF / MSS);

Ref is always 20.0M, RefDiv always 500.0, and for this calcualtion RF is always 1539.4, all numbers Double with 6 digits of precision. I generate an array of frequencies given a start, step, and frequency count. These frequencies then go to a subVI with a Formula Node that calculates the byte values to send to a couple PLLs.

If Start = 70.1, Step = .025, and Count = 20, at frequency 70.2 the Floor function gives 38.485.

If Start = 70.0, Step = .025, and Count = 20, at frequency 70.2 the Floor function gives 38.484.

I've omitted some calc steps here, but I've verified the starting values in the subVI are the same in both cases. Why the result changes I'm hoping someone can tell me...

Thanks,
Steve

[altenbach]

The floor function always gives an integer result, so how can it be 38.485?

Could you post the entire code? Obviously you've ommitted too many details to reproduce the problem. 🙂

And what do you mean by "DBL with 6 digits precision" are you just talking about the cosmetic decimal format in the controls/indicators or are you actually truncating the values in the code? Are you using plain numbers or special units.

[JeanPierre]

Did you check the intermediate results? Floor(1.9999999) and Floor(2.0000001) give different values for nearly identical inputs. You said "DBL with 6 digits precision": DBL always have 14-15 digits of precision. You display them with 6 digits then maybe what is displayed as 2.000000 is less than 2 and floors to 1. You should generate your series by scaling an integer series e.g. offset + int*step.

[Code Toade]

Sorry, 38.485 is really 38,485, I was displaying in SI Notation .

DBL with 6 digits is just 6 digits in the display.

The "series" is a generated sequence of MHz frequencies in 25 kHz steps.

There aren't really any intermediate steps.
MSS = Ref / RefDiv is always = 20M / 500 = 40k. These are default values stored in the VI.

MDN = floor(RF / MSS) always starts as floor((1539.4 * 1M) / 40k). RF is the varying frequency, but for this calculation it is always 1539.4, verified by break points.

I can't post the whole program. Maybe I could post the subVI? Or just the diagram? There are several VIs between the series generation and the Floor subVI calculation.

Everything works fine for a series of 20 freqs starting at 70.1 in 25k steps. But start at 70.0 and the PLL won't lock at 70.2, 70.4, 70.6, and 70.8 - traced to the result of floor(((1539.4 * 1M) / 40k)...

FWIW. Could be a "feature" of WinXP for all I know... 🙂

[JeanPierre]

Output the argument of the floor function in each case and display it with 15 digits of precision. My guess is that due to calculation imprecision, it will be 38485.000... in one case and 38484.999... in the other case. Your start value 70.1 can't be represented exactly in a double, it is binary rounded to 70,099999999999994.
That is probably the cause of your problem. Maybe the int(x) function is more appropriate.

Message Edité par Jean-Pierre Drolet le 06-07-2005 01:49 PM

[Code Toade]

You are right, the argument is 9994 instead of 0000.

Now this presents a real problem, because the calculations to get the correct PLL settings assume real number truncation. I vaguely remember initially using the int(x) function to start with and discovering that all numbers needed to be truncated. So I need something like the floor(x) function. It's just that in a very limited number of currently unpredictable calculations this imprecision becomes a factor.

Well, I'm stumped. Unless I manually check values to the nth decimal place and decide to round down or up...

Thanks for the insight!
Steve

[Ben]

Usually a strategic multiplication by a factor of 10 followed by a remainder quotient function will handle these issues.

Will that work for you?

Ben

[Code Toade]

---

@Ben wrote:
Usually a strategic multiplication by a factor of 10 followed by a remainder quotient function will handle these issues.

Will that work for you?

Ben

---

I'll have to do some testing. I don't want the "fix" to throw it off somewhere else. Don't know if the hardware guys want to take the time right now to do more extensive testing, since this is test code itself and we can work around it.

MDN = floor(RF / MSS); is the only place I could try it, since it has the only varying number...

Thanks for the help, Gents!

Steve

[Code Toade]

I want to thank those involved again for their help. With ideas and hints from others I found a solution without scaling.

In recap, what had bothered me was it *appeared* like the same subVI was giving correct results one time and incorrect results only randomly. While I understand binary fractional imprecision, I wasn't doing any looped calculations 100+ times or anything.

I did some more checking though. The problem was indeed introduced by cumulative fractional addition. In this case 10 additions were enought to cause the error. Apparently, floor(71.199_94) produces 72.0. However, using a shift register and constant fraction to add an offset to produce an array introduces enough error in under 10 iterations to be a problem. By the time the loop got to what was supposed to be 72.0, it was actually 71.199_84 or something, enough to throw the floor function. Now I understand why the error occurred, and why it wasn't a problem before.

I fixed this problem by changing the real frequency number to a I32 before introduction to the formula node. This corrected the error introduced by the fractional addition by forcing 71.199_84 to 72, instead of letting it propagate through the rest of the calculations. And it was a whole lot easier than changing all the VIs to allow scaling! Also, I prefer to know where and why the problem happened, instead of just scaling all my calcualtions. Maybe I can recoginse potential problems in the future.

My boss wants to go back and look at his program to see if HPVee somehow bypassed the problem or if he did the calculations differently.

Thanks again for the insight and help,
Steve