The trick of duplicating handles works in Windows because their underlying memory structures are part of common system memory in kernel address space, whereas user handles reference user memory areas. It is possible to write a user driver which keeps its variables in kernel space as well, so that sharing would work, but as soon as you need access to any third party code libraries which are not written this way (eg VISA), it all falls over.
How about this as a solution to your original problem: create a third, 'multiplex' application which is solely responsible for the VISA/hardware communication handling, and make both of your applications communicate to this common one (via shared memory) when they want access to the hardware. Obviously you would need a suitable command/status protocol in place and an elementary task locking arrangement to prevent simultaneous access, but its worth taking a look at this general approach.
JR
