Hi Johnny,
You are right that this module may not be able to supply a solid sustainable +5V output by setting the DO line to high. In the specifications document for the 9401 on page 13, it specifies that the MAX output Voltage is 5.25V, but when the module is sourcing current that voltage is going to be lower. For example, in the same document, it specifies that if you are sourcing 2mA of current the max output voltage will be 4.3V. That is good that you found that out already, I just figured I'd point you to where that is specified.
As far as connecting your quadrature encoder to your 9401, you must have that module in either slot 5 or 6 to be able to have external access to the counter on the backplane of the cDAQ chassis. Also, to figure out the right connections you should go to your 9401 in MAX and right click it and select "Device Pinouts." This is where you can see the connection pins to use for specific counters. One other way to see what pins you'll need to connect to when performing the angular encoder task, is to create a task in MAX and it will show you which pins need to be connected and how. I have attached an image of an angular encoder measurement task in MAX. This is basically the same thing that you'll want to use.
The other thing I forgot to mention about your specific encoder, is that it has those other connections, A', B', and Z', because it is a differential quadrature encoder which is better suited to provide noise immunity. However, NI counters do not support differential encoder measurements, so you simply need to use A, B and Z signals only and as long as your signal quality is acceptable, your measurements will work the same.
I would also take a look at the cDAQ 9172 manual for more information about counter operations. I hope this gets you started.
Chris W