Just to give an update, I've figured out my problem and will post the solution I found for others that may be trying something similar. When using the open workspace tool API in labview if the tool is not a native workspace tool you have to provide the absolute path of the tool as it pertains to the location on the host PC. Basically enter the same path location as if you were to add the custom tool to the Workspace Window Configuration from the system definition explorer.
Example: If in the Configure Tools dialog under the User Interface tree item in the System Definition file you enter the custom path "c:\Software\custom window.vi", You would have to input that same path into the API Path Control. Originally I though I would only have to enter the "custom window.vi" into the input similar to if I wanted to open the native calibration tool where you would input the "calibration tool" into the API.
