I would expect the ideal SNR for a 12-bit ADC to be about 74dB, not 90dB. However, the 40dB figure you are seeing definitely seems very low.
The error you're seeing in your output may be related to the operating conditions of the PC and are likely related to the absolute accuracy that is achievable given the temperature conditions within your PC case. The M Series Help contains helpful tables for the 628X devices to calculate the absolute accuracy you should expect at full range. You can view the last calibration conditions and the current operating temperature for your device in MAX on the calibration tab under NI-DAQmx Devices>>[your PCI-6289]. You will need to make absolutely sure that the conditions are similar (the help file indicates that a change of +-10C is the operational limit for properly calibrated output values).
The PCI-62XX M Series cards do not feature onboard analog output filtering; the only filtering functionality available is the analog input LP filtering on the PCI-628X devices.
Hopefully this helps-