Thank you for the clarification. I do have the cable and SCXI-1349, and have been working with SCXI myself for about 15 years. I can state that we did not seek out the link I sent you and decide that it could not be done; we attempted to control the SCXI with the PXI-6254, and MAX would not recognize it. After repeated attempts, we called NI, and were told it would not work - they directed us to that article. This is when we decided to punt and use the 6254 as the input without the 1102 amplifier.
I apologize for essentially stating that the 6259 would not work in any PXIe; it is true that its incompatibility is unique to chassis like our 18-slot PXIe.
I hate to undermine the NI support you were given, but I believe that if they told you it should not work that they were mistaken. That article (mostly) correctly states that you cannot control the SCXI part of a PXI/SCXI combo chassis with a full backplane card. I say "mostly" because it is true that you cannot control the SCXI part of a PXI/SCXI chassis with a hybrid card over the PXI backplane local bus. However, you can (and I have) controlled the SCXI part of a PXI/SCXI combo chassis with a hybrid M Series card in a PXI slot in a PXI/SCXI chassis using the cable/SCXI-1349 method. It absolutely works. It's just not the most elegant method in that style of chassis, and it may not be officially tested/supported/recommended by NI, which is likely why they state things the way they do in the article. However, exotic combo chassis configurations aside, ANY M series hybrid board should absolutely work with the cable/SCXI-1349 method to control a free-standing SCXI chassis.
If your PXI-6254 could not identify your SCXI chassis through a cable and SCXI-1349, I am quite certain that it's unrelated to the fact that the PXI connector is hybrid. It is likely a bent pin (the most common failure point in my experience) on the 1349 or the SCXI backplane connector or just a bad part. (Perhaps the digital pins on the 6254 are dead, for example.)
Going back to your signal integrity issue, am I understanding your setup correctly when I state that your have an uncontrolled SCXI chassis with the SCXI-1581 module providing current to your RTD and are digitizing it directly (with no amplification) with your PXI-6254? I assume you're acquiring with a differential input configuration since you mentioned using bias resistors on AI+/- to AI GND. Are you leads especially long? Are they twisted? Is there a shield on your RTD? If so, connect it to AI GND as well. How many wires does your RTD have?
Are you against filtering in software? The thermal mass of an RTD means that temp changes are certainly not responsive at the acquisition rates the DAQ board supports. Could you slap a low pass filter on the data in software? If you do, do you get the expected DC temperature readings? Keep in mind that the 1102 has a very aggressive low pass filter (2 Hz cutoff if you have the standard 1102), so you may well have had all of this noise previously and just never seen it on account of that filter. I suspect that whatever signal conditioner you're using now also has a low pass filter.