Lets make sure you are performing the correct actions:
1. Place a Call Library Function node on your block diagram.
2. Double click the node to open up the Call Library Function dialog box.
3. Click the Browse button and navigate to the DLL function. Select the proper DLL and click Open, or just double click on DLL.
4. On the Function Name box, click on down arrow. A list of functions found in the DLL should be displayed. Choose the function.
5. On the Calling Conventions box, click on down arrow and select C or WINAPI. In your case, it should probably be C.
6. The Parameter box should show return type. In the Type box, select the data type of the return value. Since you won't see HRESULT here, you need to find out what data type HRESULT is in the C code. I would guess it is a numeric so select numeric from the drop down list.
7. Click the Add a Parameter After button.
8. In the Parameter box, change the name arg1 to the actual name of your first parameter, which is byPortNumber.
9. In the Type box, choose Numeric, since a BYTE is a numeric in Labview.
10. In the Data Type box, choose Unsigned 8-bit Integer, since a BYTE is a U8 in Labview.
11. In the Pass box, choose Value. If the C parameter was a pointer (*BYTE), you would choose Pointer to Value.
12. Verify that the contents of the Function Prototype box contains the correct syntax according to the C declaration.
C: HRESULT RSPInitComm (BYTE byPortNumber) should be in Labview: long RSPInitComm(Unsigned char byPortNumber)
13. Click OK button.
Try it out.
If you get a crash, change the Calling Convention to WINAPI.