This has been discussed here as well as extensively on LAVA. Some of the definition is personal opinion as well, but here is mine.
Queued State Machine (QSM): Actions are put onto a queue and executed by the state machine. The state machine itself can enqueue new actions (state transitions) based upon state logic. This is kind of like a wind up toy, where you wind it up, put it on the table, and you expect it to move around for a little bit until it stops. However, it could move around forever, or it could fall off the table.
Queued Message Handler (QMH): Actions are put onto a queue and executed by the message handler. The message handler itself cannot enqueue new actions. This is like your loyal butler, Jeeves. If you ask Jeeves to do something, he will do that and only that.
Either of these could be abused by the programmer. Just try to stay in the mindset of having your processes be small, discrete, actions that can be easily interrupted if need be.
Note that NI, in their Tutorial on the QMH, specifically says that the Message Handler can send messages to itself, which as GregoryJ mentioned, can cause trouble. Good Rule of Thumb is to remember IBM's famous one-word (5 letter) Motto ...
What is Difference between QSM and QMH?
Kind of just reiterating what Greg already stated:
QSM - State Machine that holds a series of states to be ran in a queue. The JKI State Machine is a form of a QSM, it just uses an array of strings (or is it lines in a string? Been too long since I really examined it) to hold the states.
NOTE: It is really bad if other loops have access to the state queue. Use other means (another queue, user event, notifier, etc) to send messages to a QSM.
QMH - Loop that runs in parallel with the main loop that does one specific job when another loop tells it to do something. For me, these are commonly log or instrument handlers.
NOTE: It is really bad if a QMH enqueues states onto itself. Race conditions ensue.