I mentioned a few posts back that it is possible to change the trigger level used for the serial port to decide when to send data back to the host. This could in effect change the latency you see in received data. The lowest latency will be observed by setting that trigger level to 1, but it will also increase your CPU utilization, although this may not be significant depending on your application.