DMA FIFO (target-to-host) very rarely times out in Timed-Loop

[tobiy]

I have a DMA FIFO in a True/False case inside an FPGA Timed Loop. The hardware is sbRIO-9607 and the "host" is the realtime application.

The Timed Loop runs at 40 MHz on the FPGA. 320 cycles span one repeating process (125 kHz).

Every approximately 10 ms, during these 320 cycles, the FIFO is written to in exactly 16 of the cycles, so on average, every 20th iteration contains a write to the FIFO. These writes come in bunches, however, so that several writes occur in cycles right one after another.

There is plenty of space free inside the FIFO. The Realtime App constantly tries to read a bunch of 16 samples, and takes them "out" of the FIFO as soon as they are there. The "Samples Remaining" value on the RT side never exceeds 0.

The Problem:

The system runs fine in general, but the FIFO Write on the FPGA generates very rare Timeouts. A single Timeout event happens approximately every few minutes, so the error rate is approximately 1 in 100,000 writes. That sample (that times out) is forever lost, which is problematic in my case.

After this Timeout, the FIFO resumes writing normally.

The FIFO is correctly wired with a 0 on its Timeout terminal.

The Question:

How to make sure the DMA FIFOs never time out ? What causes them to time out despite plenty of free buffer space?

[Terry_ALE]

Can you post code?  How big is the FIFO on the host side?

[tobiy]

Dear Terry,

What do you mean as FIFO size on the host side ? I didn't set up anything other than the general FIFO length in the project explorer. That length is 1024 samples, enough to hypothetically last for about 850 ms without any reads from the Host. (This never occurs, the host reads whenever 16 samples become available).

Here is a snippet of the FIFO writing part:

The full project is rather large and contains sensible IP. You can see the boundary of the timed loop on the right. The FIFO write is incapsulated in a case structure, and only used in the True Case (False is empty). That means, only when the Boolean Shift Registers pass in a True value, will the FXP shift register get written to the respective FIFO.

Timing:

The writing to these FIFOs occurs in bunches. E.g. for the lower FIFO, there are no writes for around 10 ms. Then, within 320 cycles, there will be in total 16 writes. Then there will be no writes again for ~10 ms. That means that the average writing rate to this FIFO is only about ~1600 samples per second. The FIFO size is configured as 1024 samples, which is enough to last for >800 ms without any read from the Host.

The host is configured to read a bunch of 16 samples whenever it becomes available.

How I identified the problem:

I monitored the "Timed Out?" output of the FIFO and set a register high whenever this becomes high for the first time. That register will flip high after a few minutes into the program.

[tobiy · Accepted solution]

Sorry, it was me being stupid!

The FIFO Buffer size was indeed incorrectly set. It was not 1024 elements, but it had just enough elements to store a single bunch of values, but not even enough for two bunches. So some slight delay on the Host Side would immediately result in a FIFO Overflow and thus Timeout.

[Terry_ALE]

Do you run the function from this link? https://www.ni.com/docs/en-US/bundle/labview-2020-fpga-module/page/lvfpgahost/fpga_method_fifo_confi...

From the code screenshot I recommend monitoring  the value of the timeout output to a latched boolean.

[Intaris]

---

@tobiy wrote:

Sorry, it was me being stupid!

The FIFO Buffer size was indeed incorrectly set. It was not 1024 elements, but it had just enough elements to store a single bunch of values, but not even enough for two bunches. So some slight delay on the Host Side would immediately result in a FIFO Overflow and thus Timeout.

---

I need to ask: Buffer size defined on the FPGA itself or the RT-side DMA buffer size?

The DMA channel itself has a size, but with the RT system running, these can be kept remarkably small without ever losing data. The buffer on the RT-side however should always be a minimum of 2x the expected packet size to make sure there's always space for the FPGA to offload the data to RAM in the RT system.

[tobiy]

@Terry: This is exactly how I identified the problem as explained below the screenshot 😉

I did not run this linked Configuration function, so the "Depth" size is whatever is its default value.

@Intaris: The buffer size I was talking about is (probably) the FPGA size. It is the one that I can set when creating the FIFO in the project explorer.

[Intaris]

---

@tobiy wrote:

@Intaris: The buffer size I was talking about is (probably) the FPGA size. It is the one that I can set when creating the FIFO in the project explorer.

---

My experience (different FPGA target, different RT) was that even when streaming data at 1MHz from the FPGA, my FIFO never had more than 3 or 4 elements in it on the FPGA. The transfer to the DMA buffer on the RT was so fast, I needed very little space on the FPGA itself. As your case only involves a few elements, it doesn't really matter much, but in case the amounts get larger, try making the RT side buffer larger so that there's always space and see how much backlog you even get on the FPGA. You can activate the "elements remaining" on the FPGA for debugging to see how low you can actually go with buffer size on the FPGA itself.

At 40MHz, your buffer requirements will most likely be more than mine.

This answer is not meant to correct your position, just as more background info for future viewers.