Are you sure you are getting complete messages every time you read the serial port?
Right now you loop runs 10 times per second (100 ms wait timer). It takes all the bytes available at the port and reads them, then proceeds to work on them. This could be 1 byte, it could be 100 bytes. I would bet that if you are getting incomplete data (such as only 1 byte when a complete data packet might be 10 bytes) your string data is not being converted to good numerical values.
What does one package of valid data look like? Does it have a termination character?
You need to set up your serial communication that ensures you have a complete package of valid data before your proceed to convert it and do the math on it.
