serial port hang/freeze

[beuvink]

Hello Matt,

I wonder what the status is on this issue? I am experiencing a similar effect when using an FTDI 232 converter. It has taken me a year sofar to do all kinds of debugging as it happens unfrequent, sometimes after half a day, sometimes after a few weeks. I have been able to trace it back to EMC distortions that cause USB devices to act strangely and it seems that software drivers cannot handle it. Plugging/unplugging the device is the only way to restore functionality.

The tests done sofar have included many PC's, windows XP and 8. Updated drivers for the FTDI converter and VISA. All with no result. At this moment I am at the stage that I can generate logs and even introduce distortions into the USB signals themselves and I found that the USB bus is an Unreliable Shitty Bus and cannot be compared to the simple RS232 lines we had in the good old days. One of the simple experiments I did to get to this conclusion is to generate some noises at one USB device (using an piezo-EMC-noise generator) and I can make my mouse hang (eventough it is half a meter away).

It seems to me that USB itself cannot be trusted in noisy environments so I will improve the screening and isolation of the setup. This will include ferrite beads around the cable, USB isolators and if it is needed I will design my own USB watchdog to do the plugging/unplugging. I did find some tools to monitor plugging/unplugging as well at http://www.nirsoft.net/utils/usb_log_view.html

Or did you find another solution??

Kind regards,

Mark

[camtest]

Hello folks

I've been having some bother, intermittently, with USB-RS232 converters. We've built a test system for big transistors (up to 4500V, up to 7200A nominal, though we've been over 25kA on occasion). The whole rig is connected via a fibre-optic ethernet link to its control computer for isolation purposes - ie no direct connection between the control PC and the rig. It's in a room of its own; the only externally accessible control (other than the ethernet) is the power switch.

Amongst other things in there is an ethernet-USB hub, one of whose devices is an FPGA development board with an FTDI USB-RS232 converter. There's another to control the PID on a heater controller.

Occasionally one of the USB-RS232 converters stops responding, usually the one on the dev board. The software recognises this and attempts to shut down cleanly / safely and returns to its 'idle' state.

However VISA Close hangs forever when trying to shut down the dud USB-RS232 adapter, to the point where the VI calling it can't be stopped, and even LabView (2011.0.1f2) won't quit. Killing it in the task manager (end process, or end process tree) closes all its windows but the process remains.

Watching in I/O Trace, the VISA Close is issued, but the status column remains blank. If I power cycle the rig, then the VISA Close eventually succeeds, and things go back to normal(ish).

I'm assuming that either dV/dt or dI/dt is coupling into "something" and upsetting "something else". I've just stuck some big toroids around various USB and RS232 cables, and things have calmed down. It's all subject to a bit of speculation, as the problem can't always be produced on demand.

What amazes me is that it's possible for a VISA Close to stiff LabView completely, no matter what the device on the other end has done; it doesn't seem to time out? Is there any way of "gingerly" testing for such an eventuality and prompting the user to do, hm, something different?

thanks

John

[RTSLVU]

Beware counterfit FTDI and Prolific USB to serial chips have been flowing out of China the past few years.

[camtest]

Interesting, thanks: is there any straightforward way to spot them? Presumably Vendor ID, product ID, vendor name are all correct? And is there any supplier who can guarantee you get a "proper" one when you buy a USB-RS232 cable? (though, in this case, one of them is on a demo board, so I got what I got)

[beuvink]

Hello Camtest,

As described in a few posts above I did have the exact same problem: Having a electrically noisy environment and using the FTDI232 sometimes causes a complete hangup exactly the way you describe it. One thing you could try is instead of power cycling, unplug the USB device. Now it should also return to normal.

As the device hangs, the driver hangs. And as the driver hangs the thread hangs, and as the thread hangs Labview hangs. The thread (and thus labview) comes alive after unplugging.

I have now attached a lot of ferrities on all USB cables and the problem is gone (hopefully doesn't come back after making this statement ). It seems that something goes wrong within the FTDI device when the outside world is generating spikes.

I have also tested it on a lot of differenent PC's, hardware and different batches of the device and searched the internet. I am pretty sure it is the FTDI device.

I also tried to make this effect happen on purpose but this proved to be difficult. In one case I connected a high-speed clock signal to the Rx pin (12MHz) and this was causing problems with even my mouse pointer moving erretically (seemed like some mixup at the PC between the two USB devices...) but I wasn't able to reproduce it.

I was able to reproduce the issue exactly by putting a tiny drop of demineralised water near the USB pins of the device, and wait for a minute. Then when I dried the device using a hairdryer/paintstripper I could undo the effect. I could even cycle this (in my case I thought it was caused by humidity but now I have prove it isn't).

-- in conclusion --

Install and tun USBDeview and USBLogView and keep them running at all time. Do your tests and see if your device disconnects/connects. If so: you will run into unpredicatable issues like this. The only solution is then to improve EMC robustness of your setup until the issue is gone.

[camtest]

Hi

Thanks for the interesting info - nice to know "I'm not alone"!

Unplugging isn't an option as it's in a room that I can't go in whilst the test gear is running due to the high voltages around (3kV). The only physical control I have is the isolator (which also powers off the rest of the rig - two scopes, two power supplies, ethernet hub, a bunch of other stuff).

It's complicated by the fact that I'm only around one day a week to do development work. For the other four days, assorted people use the rig, aren't so aware of (or interested in) the issues of routing signal cables away from high current cables, and then I get requests for help along the lines of "the rig's bust again".

I tried USBLogView but nothing unexpected yet; however I've got a bunch of ferrites all over the place which seems to have calmed things down. As ever things never go wrong when I'm about. Given some time I'd take all the ferrites off and go looking for the problem - but there are a million other more pressing things to do and it seems to be working for now.

Perhaps I should look at ethernet - serial instead of the intermediate USB step. Are the Brainboxes ones any good in electrically noisy environments?

thanks

John

[beuvink]

Hello John,

I don't have any experience with other serial converters so I cannot help you on this one. Solving this problem took me more then one year as I first tried to solve it in SW, but in the end my conclusion was that you just shouldn't trust USB in situations like these... It may work fine for weeks on a row and then suddenly you run into freeze issues every few hours.

Of course USB has never been intended to be used like this and if it is so hard to reproduce, it is also difficult for the supplier to prevent it. So I would keep your eye open to more robust (isolated) interfaces.

[camtest]

So I thought I'd fixed it with a scattering of ferrites around the test system.

Next time somebody else uses it (and I'm not around) - it's back!

Two more ferrites later and it seems to be OK again ... for how long?

Nothing reported in USBLogView.

I need to get a proper AWG instead of this CPLD dev board that I was given, and dispense with the USB altogether!

[Solariseir]

Hi guys,

I have very same problem with my robots, the USB controller hangs and the robot does not receive the stop command, if I wasn't around last time it could make a mess.

The USB device hangs completely when I send command to USB serial converter, system freeze for 3 or 4 seconds, but when I try to close the controller program it hangs, even end process tree doesn't work.

The only solution is restarting the laptop board on the robot. I have no access to USB cable to unplug it, its under 3 levels PCB in the bottom layer of robot boards.

So you are not alone 🙂 its been a year since I built this robot and I couldn't use in in the Lab.

Jack

[camtest]

Well "my" system has been OK for a little while since the latest round of ferrites. From the description I can't figure how big or electrically noisy your robots might be; but in my case a few turns of the RS232 cable around a big ferrite have done the job.

Now I have a sick USB-ERB08 relay box which appears to have fried during a rig failure - and the manufacturer (MCC) reckons it takes 5-10 days to ship a new unit from the US! Haven't they heard of overnight shipping?! Anyway that's for another thread 🙂