LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

CPU usage issue with high speed LAN data streaming from cRIO-9067 FPGA

Hey there!

 

I'm having trouble with the network data stream from my cRIO-9067 controller to my host computer.

I'm running some data acquisition on the FPGA of the controller at high rate (100 kHz), where all the data is written to DMA FIFO buffers to transfer to the host computer, which is attached via gigabit LAN, both connected to a switch. The data rate of the stream right now is about 20 Mbit/s, so still very human. However, the controller doesn't seem to manage this situation properly, because I sometimes experience data lags (approx. 0.5 s or so) in my host VI.

These, which I found out after some testing, originate from the cRIO controller, causing the host VI to wait on data from the stream for a while. When I look up CPU usage on the cRIO in NI MAX, with the network stream running, both cores of the ARM core of the Zynq processing unit go up approaching 100% load, probably causing the delays in the network transfer. I see this issue happening although no VIs are running on the ARM cores.

 

The questions I have:

  1. What exactly causes this immense amount of CPU load on the controller, only for the network transfer?
  2. How can I possibly resolve this issue, going up to even higher data rates for the streaming (without buying a new controller)?

I'm not exactly sure if I posted in the right subsection here, just move me if you think this fits better somewhere else.

If you need any further information on the issue, just let me know.

 

Thank you!

Joe

0 Kudos
Message 1 of 4
(3,067 Views)

Some VI must be running on the cRIO in order for you to do the data transfer.  The common cause for 100% I have seen is with the DMA FIFO since it actually polls for enough data to grab instead of waiting for an interrupt.  Otherwise, I would have to see some code to see if we can find something.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 2 of 4
(3,063 Views)

First of all, thank you for your reply.

There is really nothing interesting to see in the code as it is code run on the FPGA part of the cRIO, thus not itself contributing to the CPU load. Of course, this way, I have code running on the cRIO, but only on the FPGA part. There is no distinct VI running on the RT part of the controller, which really makes me wondering what about the network transfer is eating up all my CPU power. I usually observe values in the high 80s or 90s of percentages for both cores when checking in NI MAX, no real 100% load.

The way the data transfer works is:

In the FPGA VI, I use a FIFO Invoke node (FIFO.Write) to write the data to the DMA Target-to-Host FIFO at 100 kHz (1 Value per loop iteration).

In the host VI on my computer, I use another FIFO Invoke node (FIFO.Read) to read the data from the FIFO (which is in my project) in chunks of, right now, 5000 Samples, and hence, a frequency of 20 Hz. Changing this size doesn't change too much about the situation, I'm still experiencing the lags for faster read rates. The data is automatically transfered to the computer via LAN connection, which should effectively be the only thing utilizing the ARM part of the controller (or not?). The FIFO read also uses a timeout of 1000 ms, it's never happened for me that the FIFO.Read operation actually timed out. I forgot to mention that I also have 3 FIFOs running in parallel, which are all fed with data every single FPGA main loop iteration.

 

I don't know if I get that polling issue right you adressed. Is it that when I run the FIFO read operation, the FIFO causes the RT part to constantly poll for enough data to start a new network package transfer?

 

I forgot to mention that I chose the size of the FIFOs (on the controller) relatively large, around 16k Samples, in order to avoid timeouts. FIFO timeouts sometimes happened to me, however, only due to the lags occuring in the data transfer, which caused an overrun on the host side of the FIFO. These don't happen anymore, now that I increased width of the host side FIFO part to 128k Samples. This tells me that there appears to be no issue in emptying the FPGA FIFO and transfering data to the controller RT part, as the FPGA FIFO side never timed out. The issue seems more to be the process of moving the data from the RT part to the host computer, which also made me try a different (GbE) switch, which didn't change much.

 

Thanks again!

0 Kudos
Message 3 of 4
(3,027 Views)

Okay. After some more days of investigations, here's an update.

 

The CPU apparently isn't the main issue here. The main issue (or at least main part of the issue) seems to be too fast transfer of network packages / FIFO read requests. I recognized this when I set the packet detection mode on my cRIO to Polling (1ms) instead of Line Interrupt, which made the situation much better, although not perfect. I also realized that this is achievable in a similar way by just inserting some ms wait between the FIFO read commands in the main VI - same effect. A wait of 1 ms, for example, already reduces the amount of delayed FIFO reads greatly. By the way, I recognized the delay time on the read operations is always inside the range of roughly 200 - 320 ms. It has - so far - never happened to me that a delay was outside of this range.

 

However: I can't get rid of this phenomenon completely.

So, how can I finally get rid of this issue, now that I've already tracked it down so far?!

 

I've broken down the project to only the relevant data transfer and some timing indicators, and attached the project files (doesn't contain waits yet), in case someone wants to try to reproduce the issue.

 

Thanks!

Download All
0 Kudos
Message 4 of 4
(2,994 Views)