One issue I see with a custom device is that it will have no way of querying host processes unless you are deploying the VeriStand engine to the local Windows environment.
I would recommend looking at a VeriStand Service as an alternative because it always runs on the host system. Services are similar to Workspace Tools (and the interface is identical) but they run as soon as VeriStand deploys and connects to the Engine.
You can find an example Workspace Tool at C:\Program Files (x86)\National Instruments\LabVIEW 20xx\examples\NI Veristand\Workspace.
You can create a Service from this VI by copying all of the Front Panel controls from Workspace Example Tool.vi to a new VI and then adding your custom code. The VeriStand help linked above describes how to add the new VI as a service. This service could then monitor the UI processes as you described.
An alternative to monitoring the UI processes would be to create a custom Workspace control in addition to a custom Service. The Workspace Control could act as a watchdog and send a message to the Service whenever the Workspace is closed. The Service could then take appropriate action - whether that is re-opening the Workspace or stopping the test and un-deploying the test engine. This has some extra moving parts but in general custom Services and Workspace controls are straightforward to implement.
A final option would be to create a custom user interface with LabVIEW or another .NET compatible language. This would give you full control over how the GUI can be used. As an example one of our partners implemented a custom LabVIEW GUI with user management handled directly by Windows libraries for much more granular control. You can read about it in their webcast and White Paper below.
Build a Test System for Maximum Re-Use White Paper
Build a Test System for Maximum Re-Use Webinar
I hope this helps!
