the 5660 is RF signal analyzer hardware
the SM toolkit is found HERE
The toolkits are used to work with the hardware for "real-world" solutions. It sounds like you just want to simulate a q-FSK signal so the examples from the Modulation toolkit would only serve as a resource showing how it would be done in hardware- Yet, the modulation scheme is the same so you just need to "virtualize" the examples.
You may want to install the "bug fix" for 8.6 (8.6.1) though- I don't exactly recall the bugs that related to the MT but I do know I needed the patch to "fix" a third- party app I integrated into a system back when it came out.
"Should be" isn't "Is" -Jay