Occasional freeze when reading serial port

[acepjp]

I've a labview application which is a gui for a remote device via RS422.  The host computer is a quad core running XP SP3.  Initially I was developing with labview 7.1.1.  At random intervals (anywhere between 10 minutes and 5 hours observed), labview will freeze.  Using a port monitor during this freeze, I'm able to observe repeating calls for IOCTL_SERIAL_GET_COMM_STATUS followed by IRP_MJ_READ.  VISA is configured for 0.5s timeout.  The serial card is a MOXA, CP-114EL (4port RS422).  During the freeze, Labview cannot be closed.  Sometimes it will try by displaying the "Resetting..." text but it never completes.  A hard reset is then required.  After the freeze, processor usage is 25% (on average, 1 processor working full time for labview, but bouncing between cores).  Before the freeze, its less than 2%.

At first I investigated the serial card driver via emails with MOXA.  After running a terminal emulation program and a loopback connector they gave me for several days without their program freezing, they're confident the problem isn't with their card or driver.  During discussions with them, I installed the evaluation of LabVIEW 2009.  The program still freezes except now I have less access to the VI (I used to be able to switch to the block diagram and via the highlighter tool observe it stuck in the read operation.  In either version though, the above mentioned port monitor output is the same.

My instinct is that the source of the problem is in the multi-core application, some kind of timeout race condition.  My VI is fairly simple in summary, write a command (read temps, read status, write new params, etc) and wait for reply timing out if there's a problem.  Repeat.  Serial port is configured for 115kbps and transfers are in binary.  Before the freeze, the communication works well.

I appreciate any suggestions.  I've looked around the net and this forum and didn't find anything similar.  Long time reader, first time poster.  I've a lot of experience in labview but can't solve this - the random interval doesn't help.  I think I'll start logging all serial transfers and see if anything always occurs before the freezing.

[tbob]

What sort of handshaking are you using?  I had a problem once where the handshaking was set to XON/XOFF.  When a binary file was being transferred, it just so happened that some of the bits formed an XOFF character, and the transfer would freeze.  No way to unfreeze it because the serial port is expecting XON to unfreeze it.  I had to change handshaking to RTS-CTS.  Maybe this is your problem.  If not, well I tried.

[acepjp]

Thanks for the reply, tbob.  I checked the VI to be sure but handshaking is set to None at the VISA Configure VI.  Communication is validated via checksums.  If it doesn't match, it gets ignored and a timeout occurs.  Its the same on the remote device side.  Parity is set to odd.  Transfers are in chunks no greater than 300 bytes.  You're right to suspect the XON/XOFF.  Being a binary transfer, all values from 00 to FF are sent.  But, I think if that was the case, I'd still have control over labview and not need a hard reset.

[tbob]

---

acepjp wrote:
Thanks for the reply, tbob.  I checked the VI to be sure but handshaking is set to None at the VISA Configure VI.  Communication is validated via checksums.  If it doesn't match, it gets ignored and a timeout occurs.  Its the same on the remote device side.  Parity is set to odd.  Transfers are in chunks no greater than 300 bytes.  You're right to suspect the XON/XOFF.  Being a binary transfer, all values from 00 to FF are sent.  But, I think if that was the case, I'd still have control over labview and not need a hard reset.

---

Hmmm... you are right.  XOFF would not cause LV to freeze.  This one is a mystery.

Can you use a serial sniffer to see if a certain pattern of characters cause the hang up?  Ooops, never mind.  I re-read your original post.  You already did that.  Somehow Labview is just losing its connection to the session.  Still a mystery.

Message Edited by tbob on 04-30-2010 02:47 PM

[Henrik_Volkers]

Since you run with LV2009 you also updated the VISA driver?

Do have NI spy installed? What does the logfile tell you? 

[acepjp2]

This morning I updated the labview drivers to the most recent online (I hadn't done that yet when I installed LV2009, download was large).  It was something I planned to try if the LV2009 change alone did not resolve things.  The update also did not solve the serial freezing problem.

I've attached two captures from NI Spy, one during normal operation and one of the last capture (up to the arrow).  I did a brief probing of a different VI that isn't a part of this afterward.  You can see the Read request of Com 2 for 266 bytes.  In the before image, red lines are timeouts.  So this read request is hung up waiting for bytes and never timing out.  That's at least my interpretation.

I appreciate your and tbob's suggestions.  I've a new serial card (Brainboxes PX-310 card) to try coming soon, not that I don't want to solve this, I just can't spend too much time on it.  If it still gets hung up on the new card, I'll look into the motherboard PCIe port, maybe a failing transaction.  I would've went with the NI card but I need electrical isolation and 8 ports is excessive.

[tbob]

It is very obivous from the time coulmn that the serial ports went on a lunch break at 12:00.   Sorry, I know your problem is serious but I just couldn't resist.

Have you noticed a pattern with timing?  Does it have up at 12:00 all the time?  Does it hang up after a certian number of hours and minutes of running?  Is the PC setup to perform an automated task around 12:00?

[acepjp2]

Well, it was 11:06a that it went awol.  I might be missing something wrt noon.  No, the interval between start and freeze is somewhat random.  It seem to occur most usually within an hour.  I couldn't find any scheduled tasks but its XP so all bets are off.  I've been reading into the multithreading of labview as well.  All my serial port executions are in series though, one after the other, but I'm covering my bases.

I read your WORM global link, interesting.  I've a followup question that I'm frankly embarrased to ask but what the hey.  Globals can be an enemy, I get it.  I get that when reading and writing at the same time, erroneous data can be read.  Is it more than bad data that can be the result?  Not that that's ok, but in some cases it just might not be a big deal (a string indicator for example that different loops write their status to, maybe sometimes it reads/writes garbage but mostly its updated with the two different loops state).  Can it cause deeper labview problems (crashing, references going absent, etc)?  I do have global variables acting as a status led and string status that have the potential to be accessed at the same time.

[tbob]

---

acepjp2 wrote:

Well, it was 11:06a that it went awol.  I might be missing something wrt noon. 

---

Actually, I meant 11 not noon.  Bad attempt at humor.

As for globals, the only problems caused by race conditons would be erroneous data.  No crashes and that sort of thing.  However, erroneous data MAY cause other problems depending on how they are used.  Status strings, for example, would not be a big deal.  But if some operation was dependent upon some previous data operation (like the input to a case structure), then your code would not execute in the manner that you intended.  You would't get any error message, just the wrong results.

Sorry I don't have any further suggestions as to what causes your port to hang.  It may be an XP problem.  Can you try it on a totally different computer?

[acepjp2]

In that case, we're alright as far as globals go, nothing besides use insight dependent on their result.  Can't imagine good programming to leave default case structures that crash things but point taken.

If that other card doesn't come in soon enough, in the meantime I will probably begin tests from a different computer like you suggest.  In the end, I have to try and use this problem computer since its currently my only rackmount unit.

As for noon, a belated lol.