In the 'interactive' control panel of the MAX you can also enter speed in steps per second, sometimes this is more convenient than entering a rpm value. Of course, in the axis configuration, you have to indicate the steps per revolution rate of your drive system.
But possible motor shake does not only depend on the step rate. If you set your drive (in most cases this is done by configurating the motor driver unit) to full-step mode almost any stepper motor will run rather shaky. You should set it to a reasonable microstep rate (I would suggest to start with 5 or 10 microsteps per full step), and adjust the steps per revolution rate in MAX accordingly.
Even if you set all these values to their optimum there may be some step rates where the motor w
ill run shaky due to mechanical resonance. It is good practice to check the performance of a stepper motor just in the application, i.e. with the mechanical load applied. Mechanical resonance rather occurs at low rpm rates, for higher rpm rates the inertia of the moving parts may be sufficient to overcome resonance effects.
As mentioned earlier, shake of the motor may also be related to improper encoder settings. If you do not run the motor in the velocity mode, MAX will try to compensate an enventual following error when the motor movement is supposed to be finished. If there is still a following error due to improper encoder settings MAX will run through several sequences of trying to correct the following error which sometimes looks like irregular behaviour of the motor itself.