Frustrated,
A few questions for ya:
1) You mention modbus library but also a modbus I/O server. Which one are you using?
2) What type of serial cable are you using - null modem or straight through?
3) Can you open hyperterminal and do a loopback test to verify the serial port is working?
4) Are there any coils you can read from the device instead? These are a little easier to read and we don't have to worry about conversion codes
5) Are you getting any sort of error codes?
6) Do you have an extra computer that you can run the serial modbus slave and serial modbus master examples on?
7) Have you tried using any other software to communicate to verify everything is set up correctly? Modbus poll is a useful shareware tool to make sure everything is set up correctly. If this doesn't work, then we know the problem isn't with labview, but with hardware. If it does work, something is wrong on the software side.
I have also seen issues with 485/232 converters in the past causing issues.