low latency multiple serial read

[gregoryng]

I am developing a serial DAQ box that reads using the PXIe-8130 and labview RT.  I'm reading from multiple serial instruments that send short, CR/LF-terminated packets at a rate of about 1-4Hz.  Low latency between serial data arrival and processing (timestamping) is a priority. 

One of the serial instruments is a GPS timekeeping device, and I have verified that, when reading one serial instrument, I can timestamp with an accuracy of about 50-100 microseconds.

However, when I add more instruments, or at least, more instrument handling loops without hardware connected, these latencies jump at least 10x.  The jump in latency also seems linked to the timeout period for VISA Configure Serial Port.

In other words, I suspect that the additional instruments may be locking the other serial processing threads. I have checked that the serial read is configured for asynchronous mode. 

Can anyone suggest how to approach this? Are the serial ports supposed to be blocking each others' reads like this?

For reference, here is how my VI operates:

Here is a portion of my main VI.  

The basic operation is this:

• open_aqn_serial opens a VISA resource for each serial port based on settings from read_aqn_conf.
• Each serial port is controlled by an independent while loop. Most of the work in each loop is done by the serial_handler2.vi, described below.  Feedback loops are used for sequence numbering and buffering.
• There are currently about 4 or 5 such serial instrument handler loops
• In each while loop, data from the serial port is timestamped, then passed to a queue that collects the serial data from all instruments.

Here is serial_handler2.vi:

• The VI attempts to read either enough bytes for a whole packet, or however many bytes are on the port, whichever is greater.  It is also configured to terminate read on the term char.
• The data is immediately timestamped and added to a buffer.
• The buffer holds incomplete packets (in the event of a partial read) until they are complete, then passes complete (or stale) packets to the collection queue.

[Mr.F]

Hi Greg,

What serial hardware are you using in your PXIe chassis? Also, the pictures you attached are too small to see and did not work for a VI snippet. Can you repost you Vis? Also, are you time stamping on the instrument and then time stamping again on the computer when you read? Are the two clocks (GPS and your system time) correlated?

[gregoryng]

It turns out that my problem was completely described on this wiki page: Reentrant VI

If sub.vi is normal (not reentrant), then if foo.vi tries to call
sub.vi but sub.vi is busy servicing a call from bar.vi, then foo.vi has
to wait. This can be both a very GOOD thing and a very BAD thing
depending on circumstances. It's very GOOD when sub.vi controls access
to something like a serial port, where you only want one part of your
program accessing it at a time. It's very BAD when sub.vi controls
something like ALL the serial ports, because you may want foo.vi to be
able to use one serial port while bar.vi is busy using a different
serial port. Another very bad circumstance is where foo.vi is in a
critical loop and bar.vi is not, yet because of the contention for
sub.vi, bar.vi can end up blocking foo.vi. 

[gregoryng]

Matt, thanks for replying, and sorry about the images as this is my first time posting here.  When I was editing, I was able to click to enlarge, so I assumed others would be able to, as well.  (This is how it usually works when I post images on other forums via say, imageshack.)  

The two clocks are certainly correlated.  The time on a GPS clock is reliable to precision far greater than 1us, since GPS uses timecodes to triangulate position.  

Just FYI, I am using the NI PXIe-8430/16 for serial, and the GPS unit is the Systron Donner/Symmetricom Truetime XL-DC.

[gregoryng]

Just for kicks:

[JasonSmith]

You can also decrease latency by adjusting the RX FIFO trigger level to match the size of your expected data (including any line termination characters). This is only really feasible if you data comes to you in known packet sizes, but may work for some of your connections.

If your FIFO trigger level matches your packet size, the system will receive an interrupt as soon as your entire packet is available. If the end of your packet doesn't match the FIFO trigger level, there will be a delay of 4 byte times before the UART decides to interrupt. This would only increase your latency by  40 microseconds if running at 1 MBaud, but would add about 347 microseconds at 115200 baud, or over 4 milliseconds at 9600 baud.

If your packets are of unknown size, you can decrease the latency of the serial read by setting the FIFO trigger to 1, but this will also have the detrimental effect of increasing the interrupt frequency, which impacts clock jitter and overall processor load. If low-latency serial reads are the most important aspect of your system, then you may want to give this a try.

Out of curiosity, and to help us understand our customer use cases, what are the approximate data sizes that you read on the ports? Are they all small packets, or do you use very small packets on some ports, and large packets on others?

Thanks,

Jason S.

[gregoryng]

In our case, the packets from our instruments are all of fixed, known size (one is +- one character). These are scientific instruments measuring some quantity at a their own regular intervals.  The instruments with the smallest packets are 15 bytes including termination, and the largest are 199.  So yes, I would say small on some, and large on others. Most of the instruments are 9600 baud, with one at 19200.

Thanks for the tip.  This sounds like it would help.  I can't seem to figure out how to change the setting, though. Searching for 'rx fifo' in the help doesn't come up with anything that seems relevant. Is it a property node?  Is it programatically settable on a per-port basis?  I guess that the maximum is probably something like 16 bytes?

I did say that low latency is a priority, but actually, it might be better stated that consistent latency is a priority.  However, it's got to be consistent across all the serial ports, and bounding that latency toward zero seems to be the most manageable way to make that happen.

Actually, after I resolved the issues in the prior posts, tuned the code, and tuned the thread execution classifications, the latency seems now bounded to about 500 us in a really tight distribution weighted heavily around 0.  I suspect that this should already be at least sufficient, but I don't have the latency numbers on our legacy system, and we still have a few more instruments that need to be hooked up.

Message Edited by gregoryng on 09-28-2009 07:22 PM

[JasonSmith]

In Measurement & Automation Explorer each NI serial port should have an Advanced tab where you can configure the FIFO trigger levels. If you are using a PXI-8430 you will be able to set the retrieve trigger at any level up to 127. I wouldn't recommend going quite that high since it would increase the likelihood of buffer overruns.

For your transfers of 15 bytes at 9600 baud, you should be able to trim about 4 ms of latency off the reads, although they won't be any more consistent than they were before. Keep in mind that this will only decrease your latency if the FIFO trigger level matches the packet size exactly...for your reads of 199 there isn't much you could do.

-Jason S.

[gregoryng]

Thanks Jason,

After some more investigating, it appears that I am not limited primarily by FIFO size.  I left FIFO at default, but as I add more serial handlers into the mix, it appears that they are consuming more CPU time as they poll to respond to serial data.  

Instr #5 and one other are the only ones receiving data; the others are waiting but receive no data since there is no serial port.  When I disable the serial  for these handlers dummy instruments, my observed timestamping jitter appears to scale down with the number disabled. 

The jitter at the current number of instruments tested is already a little bit high.  In our legacy system, which was a highly optimized C program on QNX, the jitter was 10x lower.  Analyzing old data, at least 95% of GPS packets (17 bytes) were timestamped within 50us of the serial arrival, whereas I'm seeing about 500us at the moment when i have 75% of the handlers enabled.  It would be nice to get somewhere near these levels again, but I understand that may not be possible since we won't have the same level of control.

I suspect that changing the FIFO size alone probably wouldn't get me down to those levels?

Message Edited by gregoryng on 10-12-2009 11:27 AM

[JasonSmith]

Let me ask you a few more questions to make sure I understand your situation:

1. Are you using only the one 16-port serial board, or is there more than one board in the system?

2. Which version of NI-Serial do you have installed?

3. When you refer to these "dummy" ports which aren't receiving data, do you have the ports open at the time?

4. What priority are your serial handlers running at

The purpose of these questions is to understand where the NI-Serial driver may be contributing to the jitter you are seeing. I believe the CPU usage you are describing is due to LabVIEW polling for the serial transfer to be complete. You can avoid this CPU usage by switching to synchronous transfers instead of asynchronous transfers, but this has the potential to cause some other starvation issues since some of your LabVIEW threads will be tied up waiting for serial transfers to be complete.

-Jason S.

Message Edited by JasonS on 10-13-2009 09:46 AM