12-07-2007 04:29 PM
Hey George,
What are the values of the sacling coefficients returned from the 6008, and what terminal configuration (RSE, Differential) are you using? I'm trying to track down a 6008 to compare. Also, what are the binary values returned at those voltages? We seem to be close, but just off somewhere...
cheers,
Andrew S
12-07-2007 05:12 PM
12-07-2007 05:29 PM
Hey George,
You may want to check how you (or possibly my code) are doing calculations. When I do it by hand:
+1.62 V = 0x9200 = 10010010:00000000 = 37376 decimal.
37376 *0.0003180253475693 + -10.2685031890869 = 1.62
I did note that with such large numbers, the number of significant digits really matter - cutting the calc off at .000318 yields significant error.
Did you use my direct code for the calcuations or do something different?
Cheers,
Andrew S
12-07-2007 06:31 PM
12-08-2007 05:09 AM
12-10-2007 09:44 AM
I'm glad you go that figured out and thanks for posting back with your resolution.
Cheers,
Andrew S