Instrument Control (GPIB, Serial, VISA, IVI)

cancel
Showing results for 
Search instead for 
Did you mean: 

crazy, my serial device confounds everything

I have an unconventional question, and I'm really grasping at straws here.


I have a test adapter that has been been reliable in service for several years.  It uses five (Kvaser) CAN channels to communicate with 30 devices using six different addresses on each CAN channel.  The devices are on a fixture in a thermal chamber controlled by a Watlow F4.  The adapter consists of a USB-6501 that controls a home-built 32-channel relay card.  There are three two-channel CAN transceivers, and all four devices are plugged into a powered USB hub.  The computer's serial port is connected directly to the Watlow F4 controlling my thermal chamber.  (The same behavior occurs when I replace the computer with my laptop and a USB-serial converter.)


Last week, the system began to fail to identify most of the devices on the test fixture.


I spent two days banging my head on the station before discovering that the adapter works if I unplug the serial cable.  I thought that the Watlow controller had gone bad and had Maintenance replace it.  The problem persists.

UUT power is controlled by an SSR triggered by the 6501.  Yesterday, I saw the adapter turn on UUT power when I unplugged/plugged the serial cable.  It happened twice, in the presence of a witness, but I couldn't cause it to happen a third time.  The failure of the adapter to identify all UUTs when I connect the serial cable is repeatable, as is the converse.

 

I've tried replacing the 6501, our relay cards, Kvaser transceivers... Do you have any ideas?

Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
For he does not know what will happen; So who can tell him when it will occur? Eccl. 8:7

0 Kudos
Message 1 of 9
(4,559 Views)

Jim,

 

I suspect that you have a (new) ground loop. Unplugging the serial cable breaks the loop. The current flowing in the loop introduces sufficient errors in the communications to prevent the identification.

 

The question is how to decide whether that is the problem and to identify how to fix it. The answers probably require more detective work similar to what you have already tried.

 

First I would use a portable DMM to measure differences in voltage between various grounded (or ungrounded) bare metal points throughout the system. I would pay particular attention to (1) the devices which are not working and (2) any device powered from the AC mains. The thermal chamber and the SSR power to the UUT should get checked carefully.  I would be thinking about a ground fault (possibly through sufficiently high impedance that fuses do not blow) or a broken/disconnected grounding conductor in a power cord.

 

Next, with all power disconnected measure resistance between power line conductors in the system (not the mains) and the grounds/exposed metal. Also repeat the ground to ground measurements but this time look at resistance rather than voltage.

 

It is difficult to say what the results should be, but look for inconsistencies between similar connections or readings of more than 10s of mV AC or DC or resistances between grounds of more than an ohm or resistances between power conductors and grounds of less than megohms.

 

Did you have a spectacular failure (lots of smoke and sparks) of an UUT in the week or so prior to the new behavior?

 

Lynn

Message 2 of 9
(4,553 Views)

Has there been any changes made in AC power wiring in the area?

 

Sea Story below:

 

I had a test system communicating with an autocollimator via RS-232 that had been working reliably for several months. Out of the blue, Production reported that the system kept reporting communication errors with the autocollimator. I verified the cabling, port and device settings, and could not even get communications established via a serial comm app.

 

Pulled out my DMM and eventually  found several volts (AC) of differential between the PC ground and the autocollimator ground when the two were disconnected. They happened to be on two different mains circuits, plugging an outlet tester into the two showed that one circuit had the ground and neutral lines swapped. Turns out that maintenace had done some rewiring of the mains to add another power drop over a weekend and got it wrong.

 

Luckily no equipment or personell experienced issues. Once the mains wiring was corrected, all returned to normal.

 

-AK2DM

 

~~~~~~~~~~~~~~~~~~~~~~~~~~
"It’s the questions that drive us.”
~~~~~~~~~~~~~~~~~~~~~~~~~~
Message 3 of 9
(4,549 Views)

@johnsold wrote:

Jim,

 

[...] Did you have a spectacular failure (lots of smoke and sparks) of an UUT in the week or so prior to the new behavior?

 

Lynn


No, nothing like that.  One of the technicians told me that he recently smelled smoke, though, but I didn't.

 

I went through many of the ideas you and AK2DM suggested and, a few hours ago, got to the point where I was at the limit of my skill-set (perhaps I over-estimate my abilities, but that's neither here nor there) and called in our Mfg. Eng. electronics technicians.  I explained what happens, what I've done and my observations and, within ten minutes, they had isolated my test fixture from the inside of the chamber and the symptoms disappeared.  The problem remains, but there's a work-around and production can resume.

Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
For he does not know what will happen; So who can tell him when it will occur? Eccl. 8:7

0 Kudos
Message 4 of 9
(4,509 Views)

@AnalogKid2DigitalMan wrote:

Has there been any changes made in AC power wiring in the area?

[...]


Nothing that our Facilities department will admit to...

Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
For he does not know what will happen; So who can tell him when it will occur? Eccl. 8:7

0 Kudos
Message 5 of 9
(4,508 Views)

Jim,

 

If isolating the fixture from the chamber eliminated the symtopms, I would say that the system has a ground fault.  If the problem had always existed, it might be a design issue. Becaues it was a recent change, some kind of failure seems more likely.

 

I would keep a close eye on that system. A ground fault in the thermal chamber (my guess as to where the probelm is) will not cure itself. Eventually it will get bad enough to hurt someone or destroy some equipment. If you are lucky a fuse will blow or a circuit breaker will trip.

 

Measure between the test fixture and the thermal chamber - voltages with power turned on or resistances with the power disconnected. I suspect you will see some ugly results.

 

Lynn

Message 6 of 9
(4,495 Views)

Hi,

Never used the CAN bus before but it (the hardware layer) looks like a RS-485 interface. A balanced line. What I have seen with RS-485 interface that it always went well with short distances. Mostly everything is at the same 0V (ground) level. I always recommend to use the 0V connection in a RS-48 connection. Some people say I am crazy but documentation from IC manufacturers tells me that there must be a 0V connection. One time when we used more isolated RS-485 interfaces it became clear that the 0V connection was necessary to get thing working.

The CAN bus does not have a 0V connection. I think that is because it is designed for automotive where everything is at the same level.

 

Maybe because of changes in earthing or something like that your signal shifted out of the common range of the transceiver's. A 0V connection could solve this.

 

Kees

0 Kudos
Message 7 of 9
(4,486 Views)

@johnsold wrote:

[...] 

Measure between the test fixture and the thermal chamber - voltages with power turned on or resistances with the power disconnected. I suspect you will see some ugly results.

 

Lynn


You're correct.  I was poking around with the technicians and we saw a voltage appear & decay when I applied UUT power.  The voltage started somewhere (autoranging DMM) above 15VDC and decayed to zero within a few seconds.

Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
For he does not know what will happen; So who can tell him when it will occur? Eccl. 8:7

0 Kudos
Message 8 of 9
(4,481 Views)

That same voltage or some portion of it is what is corrupting your communications. Now you just have to have enough persistence and patience to find out what is causing it and how to fix it.

 

Lynn

0 Kudos
Message 9 of 9
(4,471 Views)