cRIO serial read problem

[David XIII]

I have a device giving a serial output (8 data bytes, 1 stop, no parity) at 115200 baud to COM1 on my cRIO-9002 (LV 8.2).  It is a 72-byte packet of binary data, transmitted at 100Hz (thus 100 packets/sec), with no unique termination character.  It uses E7h as a sync byte at the end, but often E7h shows up elsewhere in the packet.

I have a program to read interpret this packet, which works great  -- IF! it starts correctly.  I know that the data is getting to the serial port, but sometimes the "Bytes at Port" returns zero.  If you look at the .png below, you'll see the heart of the matter.  I put the "Bytes at Port" in a While loop which stops as soon as any nonzero number is returned (as soon as any bytes show up in the buffer).  When the serial port is connected and receiving data, I can push the start button on this simple vi and five times out of ten the while loop will run forever saying that "Bytes at Port"=0.  Why??  The rest of the time the program will recognize the data in the buffer and exit the loop after one or two cycles.

I have tried different values for the buffer (150 - 16384).  I have tried a delay between 1 and 10ms in the while loop.  I tried *not* flushing the buffer before it enters the while loop, virtually ensuring that the entire buffer was full of junk, and it still returns "Bytes at Port"=0 roughly half the time.  Once it gets started, it does not seem to fail (though I've never let it go more than a minute so far).  The problem seems confined to the initial read of the serial port.  I don't think it is a cabling issue, either, because when my (larger) program is working, there's no effect when I wiggle the cables or connectors.

By the way, I ran the same program using my laptop (Win XP) as the target, using the laptop's serial port.  I ran the program 25 times, and it never failed to see bytes at the port (i.e., ran fine every time).

[NathanT]

Hello David,
       Is it possible for you to change the baud rate?

Thanks,

NathanT

[David XIII]

Unfortunately, no -- the instrument gives only the 115200 baud option for output.  By the way, my vi is a control application which requires fast feedback (from this instrument); even if I could slow it down I wouldn't want to.

My temporary solution is to put the whole serial port initialization + first read in a while loop.  If I still have zero bytes at port after three loops, I close the serial port and try initializing again.  As soon as any bytes are detected at the port, I leave the loop and proceed to the main program.  I'd really rather a cleaner solution, or an answer to why I might be experiencing this problem -- but for now this seems to work.

[Wayne.C]

Two thoughts and maybe you've already been there and done that.

1 -  Look at your COM port settings under the Windows Device Manager and compare the settings between the two PC's.  If the settings are different, try the two sets of settings on both PC's.  Even if it doesn't fix the problem, it will eliminate a potential cause.

2 - Does it make a difference if the instrument is already up and running when you fire up the PC vs. firing up the PC first and then starting the instrument?

[David XIII]

Thanks for the suggestions, centerbolt.  I don't think (1) applies, since I already know the instrument's serial output configuration (and it's not PC-based anyway), and CompactRIO's serial port is configured inside the vi.  As for point (2), the instrument requires a lengthy initialization period before it spits out meaningful data, so I don't really have that option either.  I need to be able to repeatedly start and stop the vi without stopping the instrument.

[Wayne.C]

My mistake on point 1.  Be sure and double check that all the COM port settings are identical between the cRIO and the PC.  I'd consider running it on a 2nd pc and see what happened.

Point 2 was never intended to be a workaround.  Running this test may help you isolate the problem and create the fix.

[Wayne.C]

The vi you posted has a race condition in it.  You are checking  "bytes at port"  then you do a VISA read.  If data shows up after "bytes at port" and before the VISA Read, the VISA Read will empty the buffer.  The timing of the "bytes at port" and the VISA read will vary platform to platform.  Try modifying your vi so that the VISA Read is only done if there are more that 0 bytes to be read.

Also, you are running your loop at the exact rate at which packets of data are arriving.  That means that the timing of when the vi starts running vs the incoming packets is very critical.  You need to add some sort of syncronization step to the vi.  You might be able to look for the dead time between packets.

[David XIII]

Thanks again for your persistence.  You suggested reversing the order as a troubleshooting method:  fire up the vi first, then turn the instrument on.  When I did this, my vi in cRIO recognized the start serial stream ten out of ten times.  I'm trying to figure out how to interpret that, though...

Are you sure that my vi has a race condition?  My VISA read is told to only read the number of bytes that "bytes at port" says are there.  When "bytes at port" says zero, VISA read is told to read zero bytes -- so it shouldn't empty the buffer.  To check this, I did two things.  (1) I put an indicator on "return count" so the VISA read would tell how many bytes it actually read.  I never saw it read any bytes when "bytes at port" was zero.  In fact, "bytes at port" always matched "return count".  (2) As you suggested, I put the VISA read inside a case structure and only read it when "bytes at port" was nonzero.  The problem persisted.

I have double checked all port settings.  I have tried changing the read loop rate to one 2-5x faster than the rate of the incoming data packets.  That lowers the problem frequency from 50% to 20% or so.  (Meaning, only two of ten times I try to start the vi does cRIO say it doesn't see any bytes at the serial port, when indeed there are plenty there.)

You're right that I ultimately need to add a synchronization step to the vi.  Once I fix the problem of being able to read the data in the first place, I'll hopefully be able to find the dead time between packets.

I'm eager to try any other suggestions you might have.

[Wayne.C]

David,

I suspect that reversing the power up order works everytime because this order ensures that the cRIO is up and running with the COM port properly initialized before the serial data starts coming from the instrument.  This might also explain why your PC always works.  I'm unsure as to how the cRIO COM port reacts if it is receiving data before it is configured properly.  It may be that it is hanging up and needs some sort of reset.

I did a test here with another instrument and you are right about the bytes at port.  That seems to further point to the port hanging up.  I think that I would still alter the vi to not do the VISA read unless there are more than 0 bytes to be read.  Especially with how fast you are trying to do all this.

If the port is really hanging up, you need to try and find a way to get it to recover.  It may just need time.  The vi you posted does not alter the VISA timeout.  10 seconds is the default.  Try making it something very short like 1mSec.  Try putting a delay between the VISA configure port and the rest of the vi.  Alter your vi to give it more attempts to receive data.  You may need to talk to someone at NI who knows the cRIO hardware inside out.