Custom devices need to be compiled for the target they're deployed to, so unless they are specifically written for VxWorks, they're not going to work. The custom device you're referring to (I assume it's from this page?) is just an example and not a full-featured custom device, and was not compiled or set up for VxWorks.
However, NI VeriStand 2010 ships with a better RT data logger called the "Embedded Data :Logger". Use this custom device instead since it has a lot more features and has both PharLap and VxWorks distributions.
