error adding large numbers

[ronoh]

I am adding large numbers and getting the wrong result. there seems to be some rounding taking place in the sum but i am adding integers. I am using DASYLab 9.02, data is summed in the arithmetic module, example problem 331153408-31570 = 331121838 but the output is 331121824. I tried making the variable where the inputs are stored 20 digit with 10 decimals but that did not help and i also tried dividing first by 1000 and 10000 only to get different answers. is there a setting that needs to be configured differently?

[TomRizzo]

I think that your reached the maximum size of the number.

DASYLab number are processed as floting point so, even though I forgot the tollerance now, you reached that limit.

What are you trying to do?

[ronoh]

Hi Tom, Thanks for the reply. I am reading a hex value in from a serial port. the number is large and when i format it as hex on one chan it is off by a small amount. there is some rounding in the LSD. i then take another reading later and calculate the delta. since i dont have the right values to begin with my difference calculation is wrong. when i read as bytes through 8 channels, i can see the ascii for each digit and that they are correctly displayed. using a formula module i can convert from ascii to decimal so that i get the decimal equivalent of the hex character then in the next formula i do the math to find the value of each hex digit in place it holds. then using a sum arithmetic module i get the final value of the large number coming in. it is correct all the way upto the aritmetic sum. i tried cutting the large hex number into two parts and then adding up the weighted parts and still have the wrong ans in the display module. i also tried dividing the halves by 1000 prior to adding them so that i was working with smaller numbers in the summation but that didnt help.

so i did the math directly in the extended portion of the variables. the numbers add up properly there but when i try to bring the correct sum back into the work sheet to display it, it is wrong again. it seems that a value around 04000000 hex is the limit. below that i get the right value displayed that was calculated in the variable field, above it there is some degree of variation. I can set the limit of cycles to a value below where the addition becomes problematic or i can export the hex to a spreadsheet, do the math there and then bring it back in but i will still have the same issue displaying the answer.

the limitation doesnt seem to be in DASYLab in general but in the Read, Formula, Constant Generator modules that read the variable back into the worksheet. it is displayed properly in the contents window

[hw70]

It's the data lines fault: data is tranported from one module to another in float-sized containers. So, you get 6 to 7 significant digits, if the numbers have more digits, their value is... some arbitrary value.  (Hopefully the data type will be changed to double some day... 😉 )

If you have DAYLab 13 you can possibly circumvent the problem by using the script module: do not output the value from the RS232 module to the worksheet ( a\r ), instead write the received text into a global string ( e.g. into glostr 1: $1\r ).

Now you have to write some code for the script module, that reads the glostr 1, transforms its text content to a number to do your calculations. If the difference is small enough to fit into a float variable, you are now able to output this value to the worksheet safely.

[ghav]

À follow Up question On this.
I have set Up à timer to start our proces. I create a large nummer from all time components
Minuten+100*hours+10000day of Month+1000000month+[year-2014]*100000000
This I do for the actual time and for a startup time. I can compare these and stat when one passes the other.
Due to the large number problem, the values are not precise, which is no problem if my starts are let say ten minutes apart, but if they are e.g. Only 2mins apart it may generate the same number for both.

Any suggestions?
I have been unable to find time related functions I could use like eg excel has for converting full dates to numbers.
Any suggestion appreciated.
George

[TomRizzo]

George,

There are time modules like counter and test time but I am convinced that you need to download a V13 demo and play with the script module.

This module is a Python interpreter and would satisfy many of your needs.

Tom

[ghav]

Thanks Tom,   I looked at counters earlier, but they do not easily start /stop at exact dates/times.

I'll have a look at your other suggestions.

best regards

George