The NI 17xx Smart Cameras run LabVIEW RT on top of VxWorks on a PowerPC processor. It is possible to interact with compiled libraries through LabVIEW but you'd have to cross-compile the library using GCC for VxWorks and Power-PC. If you search around you'll see there are guides to setting up a compatible toolchain for targets such as the CompactRIO that will work for the Smart Camera as well....
However, this all rests on OpenCV being able to compile for PowerPC processors. From some quick research, the codebase seems to have a lot of x86 assembly and is not likely an easy port to PPC/VxWorks.
That said, you could always use the NI Vision Development Module for your algorithm. Are there any shortcomings in what is offered that you have chosen OpenCV instead?
If form factor is not an issue, you could use the CVS or the new Embedded Vision System with a seperate camera and try to use OpenCV with that instead. Both of these platforms are x86-based and you'd likely have a lot less trouble interfacing with the OpenCV DLL.
Correct. Visual C++ can only output x86 binaries. The Smart Camera and newer cRIO devices use a PPC processor. There are several instructional articles around (such as this one: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA03q000000x4puCAA&l=en-US) that explain how to compile the VxWorks equivalent of DLLs using the GCC toolchain. After building the binary you can link to it within your VI just like a DLL on other platforms.
