Thanks for bringing this to our attention and for the VI. I've done some testing with it and have been able to reproduce the same error reliably so I've passed the information on to our R&D team. It generally takes some turn around time for bug fixes such as this to get released, so notifying our R&D department will not work as an immediate fix for you.
However, in my testing I found that the error only occurs when there is a variation in the slope that is around .01 to .03 pixels. Larger variations generally do not cause this error to occur. Since the IMAQ Bisecting Line VI is used primarily for machine vision functions it is possible that the algorithm does not work well with sub-pixel accuracies down to 1/100 of a pixel. That being said, is it possible in your application to round the values up to the nearest tenth of a pixel? The algorithm works well at this level and for most machine vision use cases sub pixel accuracy down to 1 tenth of a pixel should be sufficient. If that is not sufficient for your use case I would recommend adjusting the physical parameters of your system (such as the lens) to increase the resolution of your image, or some similar work around until we can release a fixed version of the IMAQ Visecting Line VI. I'm sorry for any inconvenience this has caused you.
Regards,
S. Arves S.
National Instruments
Applications Engineer