Looks like it's not straight-forward to do. Here's the documentation on the external digital interface of the 5644. Looks like the PFI0 line is LVTTL and so is incompatible with the generator that you have. The intention with the VST is to use the software from RFmx or the modulation toolkits to generate your BPSK signal and measure the received error rate using the VST by itself. You can use the modulation toolkit to create the waveform, play it through RFSG, and then analyze the BER in RFmx. Even better would be to use the RFmx Waveform Creator to generate that signal.
If you did want to give it a shot, I think this is the path you'd take: You either use your own level shifter or use a 6672 or 6674T to bring in the TTL signal from your generator and route it to the VST over the PXI backplane. You could then use the simple VSG sample project that comes with the VST Instrument Design Libraries to use that signal. You can modify the FPGA design to adjust the frequency shift block based on the input from the backplane to accomplish the shifting. Since this is BPSK, that should be pretty simple.
OK, I get it. The VST is not meant to be a general-purpose instrument that can be modulated via hardware. Makes sense, it's a unique thing for unique purposes.
This just went well beyond the scope of what I wanted to do, which was just replace a bench instrument. If I had copious free time I could build a BERT from LV or the FPGA and have fun doing it. Maybe in retirement.
Hi Ed, you're right--modulating the VST from another instrument is not supported. But just to be clear (maybe for someone reading this thread in the future), the VST could replace both instruments using the software from RFmx. Instead of creating the modulation from one instrument to another and back again, you create the waveform (either through the Modulation Toolkit or RFmx Waveform Creator), send and receive through the VST, and read the BER from RFmx. This would be some effort to move your test setup and get things working, but then you could free up two instruments instead of one.