PXIe FlexRIO DMA FIFO host read bandwidth/DMA FIFO overflow issues

[jtele1]

I'm working on an application using 2 FlexRIO modules and 2 LVDS digital I/O adapters.  I'm driving each of the DDC A/DDC B ports on the LVDS adapters with 16 bit data at 50 MHz.  The FlexRIOs are supposed to receive the data and write it to four Target-to-host DMA FIFOs (one per DDC connector, or two per FlexRIO).  The host reads the FIFOs and assembles a series of 2D arrays from each FIFO's output.  Ultimately, the individual arrays (one is each a quarter of a single image) will be assembled into single images, but I haven't gotten that far yet.

The duty cycle for the data is about 80% (that is, I'm only transmitting data to the FlexRIOs 80% of the time, the rest of the time the transmitters are disabled), so the throughput rate is about 80 Mbytes/sec/port, or 320 Mbytes/sec total spread over the four DMA FIFOs.  I'm finding that the acquired data sometimes has gaps in it which align with the length of the hardware portion of the DMA FIFOs in the FlexRIO modules.  That is, if my DMA FIFOs are set to 65535 length, I'll see a break in the acquired data at the 65536th data word.  The data is a test pattern waveform that is essentially just a counter, so it's easy to see the break in the pattern.  For the first 65535 words, the data is contiguous, then starting at word 65536 the pattern is discontinuous, and starts counting again from that point on in a contiguous pattern.  At the start of the acquisition, the FIFOs are cleared: the beginning data read from the FIFOs is always aligned correctly, so I know the process is starting at a good point.

The error doesn't always happen: sometimes I get continuous data through the 65536 point.  Also, the error happens independently between the four FIFOs: on any one particular run, one FIFO might have good data and others bad.  Rarely, all four FIFOs have good data.

The fact that the gap in the pattern occurs at the same point as the depth of the DMA FIFO indicates to me that the FlexRIO fills up the FPGA's hardware FIFO without the system managing to getting around to read it, which means that data gets dropped for the amount of time the FIFO stays full.  Then the transfer to the host kicks in, there's space in the FIFO again, and the data is once again contiguous in FIFO memory for a significant amount of data (I haven't yet tried to locate a second gap in the data from a single acquisition).  So it appears that the host either doesn't have enough bandwidth between the FlexRIOs and host RAM to keep the FIFOs from filling, or some software process is coming along on the host that is temporarily halting transfer capability on an instantaneous basis.

Is there a specification for the PXIe system transfer rates that would indicate we're trying to use too much bandwidth?  Or are there any priority controls on DMA FIFOs that would allow us to elevate the priority of the FIFO transfers such that they're guaranteed to go in preference to other system tasks?

System specs:

PXIe-1075 chassis

PXIe-8135 CPU

2 PXIe-7962R FlexRIO modules

2 NI-6585 digital I/O modules

LabView 2012 32-bit SP1 version 12.0.1

[Haagen-Dazs]

Hi jtele1,

To make sure that the data gets written in the correctly order I would recommend monitoring the Time Out of the write. If a time out occurs you could stop writing all FIFOs and then start when all the Time Outs are no longer there. Another option is to look at your host side and determine if you can read larger chunks of data at a time and allow the host side to deal with processing the data. An additional option would be to look into high throughput streaming for FlexRIO. In this setup you will be writing the data directly to disk on your host side and then you could process the data at a later time. I have linked an example below, this example was giving me trouble so please let me know if you have trouble loading this example. Depending on your situation these may not all be acceptable options but you will need to ensure that you are not filling any of your FIFOs. Lastly, from what I can tell you are using a Windows OS as your host and in that situation you have no way of controlling when your LabVIEW application gets processor time. If you were to switch to a Real Time controller you would be able to ensure when certain tasks are run and add priority to tasks. Please let me know if you have further questions. 

High Throughput Streaming

https://decibel.ni.com/content/docs/DOC-26619

[jtele1 · Accepted solution]

One suggestion from an NI apps engineer and some experimentation solved the problem. It turns out that I was calling the DMA FIFO Stop method right before triggering the FPGA data transmission via a control to the FPGA VI. I did this in order to clear the FIFO before starting data acquisition, but I didn't realize that this method also turns off data transfer between the FlexRIO and host memory. Following this call, I would trigger the FPGA code to start filling its FIFO, then begin reads. Calling the DMA FIFO Read method apparently turns the transfer back on, but it looks like the host VI was randomly slow enough at getting around to the reads such that the FlexRIO side of the FIFO would fill up and dat would be lost. I modified the host VI to insert a FIFO Start method call before the trigger signal to the FPGA, and the problem is now gone.