Thermocouple measurements that after a period of time suddenly go bad

[LV_Pro]

Good Afternoon,
I originally posted this on the multifunction DAQ thread but haven't gotten any response. I don't think the problem is software but...
I'm making long term temperature measurements using a PCI6040E, SCXI1001, SCXI1102 32-channel thermocouple amp and a SCXI1303 isothermal block. These are making readings using J type t'couples. My customer's problem is that the measurements will be fine for a period of time (minutes, several hours) and then suddenly some channels will start giving really bogus temperatures, i.e.: -270 DegC vs the more realistic 35 DegC. I came in the last time it was doing this and when I went down into the code to my DAQ portion the DAQmx Read vi, set as analog, temperature, J type measurement was returning those numbers for the offending channels. Some were still giving valid numbers while others were giving the weird numbers and when a Fluke meter was used to measure the output of the "bad T'couple" it too was giving valid numbers. When I stop the LabVIEW program and go into DAQmx to try and troubleshoot it, there usually seems to be no problem. If I restart the LabVIEW program there also seems to be no problem, but when there is those numbers are coming out of the lowest level of code that I have programmed (the standard DAQmx read). I'm suspecting something "electromechanical" like connectors in/on the SCXI chassis, but the fact that it goes away when software is restarted is confusing me, as well as making troubleshooting difficult.


Thanks,

Putnam Monroe

[Brian_Beal]

I am not sure about the details of your setup, but with thermocouples you need to be aware of the common mode voltage. If your t/c inputs are isolated from each other, then this isn't the problem. If they are not, then electrical noise will do wild things to t/c readings. Are the thermocouples or extension wires near any power wires? One thing to do the next time it happens is to start disconnecting the thermocouples one at a time. If the problem lessens, then goes away when you get to one t/c connected, then you will need different hardware for this application. Looking at the specs for the 1102, there is no isolation.

[CoastalMaineBird]

Sheesh - you've got a bear of a problem.

You said: then suddenly some channels will start giving really bogus temperatures

Is it always the same channels, or different channels every time?

Does it happen on several channels at once, or one channel, then another, then another...

Are you storing these readings in RAM (memory filling up) ?

i.e.: -270 DegC vs the more realistic 35 DegC

Absolute zero is -273.15 deg. C - that sounds like an open channel, falling to the voltage limit.

when a Fluke meter was used to measure the output of the "bad T'couple" it too was giving valid numbers

Possibly the act of probing the terminals corrected the problem by re-connecting a loose connection.

Possibly your temp comp connection is bad?

Possibly the connection from your terminal board to the measurement board, rather than the TC connections themselves.

[johnsold]

I have not used your hardware, but a -270 reading sounds like an open thermocouple. Try disconnecting a thermocouple and see if the readings are similar to the bogus readings you have been getting. Some stray signal may be picked up by the long leads so there may be some differences between the real system and an open channel. If the poor connection is at one of the terminal blocks, the act of connecting troubleshooting equipment may be sufficient to "reconnect." Another possibility is ground loops causing currents to flow in the TCs. This can cause a variety of strange behaviors from small to large errors to destruction of equipment, depending on circumstances.

If an overload condition occurs, can the DAQ hardware recover without a reset? If not, that would explain your problem going away when the software is reset. You might try having the program reconfigure the DAQ hardware after seeing -270 temperatures as a test to see if recovery is possible without restarting the program.

Lynn

[LV_Pro]

Well I guess I learned an important lesson, if I need an answer fast, post it to the LabVIEW discussion group! I posted this yesterday on the multifunction group and had no responses as of late this morning. When I post it here, to quote Emeril "Bam!". I'm going to have to have my customer try a few things. First I guess is seeing what numbers come back with an open TC connection. The TC's are brought back (about 3 - 4 meters total) to the SCXI junction block. The CJC sensor is part of the SCXI arrangement and is internal, in this case. As it is different channels, and usually more than one (but so far never all), it seems unlikely that it would be a bad connection. I haven't been there when he used the Fluke to measure, but I think the problem remains when he is done. Hopefully the problem will crop up when I'm there, trying to resolve weird intermittents will drive you to drink, but then you need someone to drive you home!

thanks

Putnam Monroe

[LV_Pro]

Good Morning,
Well my intermittents are still there. When they occur tehy can be on different channels, at this point seem to remain bad until the SCXI-1001 chassis is reset, and are bad values at the lowest level I can access (in the LabVIEW program the actual DAQmx read, or running test panels in MAX). They aren't, as originally observed, always a temperature that goes to -270, the latest that my customer reported was a few channels that hung up reporting 25 DegC. When I went in they started working again once the SCXI was power cycled and reset. We had the local NI rep onsite this morning, but it is really hard to troubleshoot a problem that isn't there when you are! I've looked at a bunch of the suggestions, the numbers we were getting were pretty much the same as open TC channel, but now ... I'm not storing the values, so the chances of memory filling, or those memory locations being overwritten seems small. I'll have to look more closely to eliminate the possiblility of ground loops, but both sides of the TC's are being brought all the way back to the SCXI chassis.

[CoastalMaineBird]

Like I said, you have a bear of a problem.

bad values at the lowest level I can access (in the LabVIEW program the actual DAQmx read, or running test panels in MAX).

That pretty much eliminates your LabVIEW code.

seem to remain bad until the SCXI-1001 chassis is reset

What exactly do you mean by "chassis is reset"? (I'm not a SCXI expert, but I do have a SCXI-1000 chassis here). There is no reset button. It's not hooked up at the moment, so I can't tell if there's a software "reset" function.
Do you mean power Off & On ?
Do you mean a reset button (maybe the 1001 has one, I don't know).
Do you mean a software reset from LabVIEW / Max ?

I don't see how any channel could hang up at 25°C and stay, that's not likely the top or bottom end of your TC range.

Possibly some sample/hold or track/hold amplifier quits sampling, and stays holding.

It IS about room temperature, I wonder if your CJC channel is somehow being read instead?

Ground loops would not likely be the problem, unless there's something else happening when the channels fail (like a welder firing). Even then, it's not likely that a rest would fix that.

My first thought would be to disassemble the SCXI stuff, and look for the usual suspects: dirty connectors, chips not in sockets, loose screw terminals, broken wires. Sounds like hardware to me.

[LV_Pro]

When I said reset I was talking about the one finger reset (power cycling), but there is are chassis and module resets in MAX. I haven't found one yet in LabVIEW, but I haven't really looked. I have only seen the extreme readings, haven't see the 25 C ones, but my customer is taking notes on an hourly basis. This is a new system, not quite finished yet, and so I'm reluctant to tear too far into the hardware. We still have a few channels that are configured for other measurements (voltage, current and frequency) that are connected but not reading properly, but their causes are HW (the "DC" supplied to the units under test turns out to be a PWM signal, managed to clean that up with an RC circuit on the first "test case", waiting for our order for more caps to arrive to fix the others). I'm afraid that I will have to camp out onsite to try and pin down this TC issue. Knew I should have gone into food service!

Thanks,

[johnsold]

Whoa! You have thermocouples and PWM going to the same module? Cross talk, ground loops, high frequency noise and aliasing all pop into mind when that mixture occurs.

Are the TCs grounded or isolated? How is the PWM signal coupled to the DAQ hardware and to the rest of the system? What are the sampling rates? What is the PWM frequency?

Lynn

[LV_Pro]

The TC's and PWM signals are not going to the same module, they are going to different cards in the SCXI chassis. The PWM "signals" (as opposed to the actual, heavy current voltages going to the motors) are brought back as twisted shielded pairs to a different "slot" in the SCXI-1001 chassis. Looking at the signals (TC, etc.) isn't showing cross talk or noise being picked up, but of course if it is a transient event who knows ... I have gotten some info from NI regarding another issue I'm wrestling with on this chassis, and one of the recommendations is to move to NI-DAQ 7.4. I doubt if this will have any effect on this problem but ...