Hi Vincent,
To answer your question regarding numbers of buffers, there is no upper limit besides available memory. The lower limit I'd have to check but is probably 3-5 for a continuous acquisition. However, the choice of number of buffers doesn't actually affect latency directly, unless you specifically are reading back images from further back in the queue. If you always retrieve the latest image (Buffer mode next or newest), then the number of buffers used won't affect your latency.
In trying to achieve the lowest latency, keep in mind there are a few items involved you have to consider. There is the trigger latency, the exposure time, the sensor readout time, the transfer time (might be overlapped with readout), and then finally the overhead to retrieve the image from the driver.
Without knowing the specifics of your camera, sensor ROI, etc, I can't really give you hard numbers of all the sources of latency you may be seeing. I have attached a presentation I gave a year ago at the AIA Vision Show that dives into some of these. Hopefully it helps.
Eric