There are two things that I can think of that could be causing the behavior that you're seeing. The first is that you are using the onboard Frequency Generator to generate the AO Sample Clock. If this is the case, you will notice that the Frequency Generator has limited capabilities in the frequencies it can generate. If this is the case, use the default AO Sample Clock as the clock for your AO application. M Series devices have a dedicated timing engine for AO applicatins, hence, no onboard subsystem (i.e. counters, PFI lines, etc.) will be used up if you use the AO Sample Clock default selection for timing your AO applications.
The second guess that I have is that you are not specifying your AO application correctly in order to get the generated waveform frequency you're expecting. Their are several parameters that you need to set in order to get an expected frequency with AO - rate and samples per channel. Have a look at
How Can I Calculate the Output Frequency of my Analog Output. This knowledgebase might currently be written with Traditional NI-DAQ terminology, but it should still help. Also, remember that even though you specify a certain update rate on your analog output, the actual update rate will be slightly different because of limited resolution of the divide down.
Jared Aho