It's difficult to determine the cause of the problem from your description. Here are a few suggestions:
1. Bundle the inputs to your complicated VI's into a cluster. Do the same with the outputs. You will technically then have only one input and output. This will keep you connector pane a lot cleaner and less confusing.
2. Have Highlight Execution turned on in your complicated VIs. This will allow you see the portions of your complicated VIs that might be slowing them down.
3. Run your complicated VI's without VI Server and with a set of input data that allows your VI to run quickly. Then, using VI Server, pass this same input data to your complicated VI's. This will allow you to determine if there is a problem with the way you are calling y
our complicated VI's.