error -63198: system run out of resources when opening multiple FPGA Vi references

[rayqin]

I am trying to open 15 of FPGA Vi refenreces using "Open FPGA Vi Reference". Each is connecting to a FPGA card PXI-7842R.  After the PXI real-time system reboots, the program only opens 6 references when it runs the first time, with the rest (9) failed with Error code -63198: system run out of resources. Close a session and retry the operation.

If the program runs the 2nd time, only 5 gets opened and the rest errored out. 4 gets opened at 3rd time. 2 gets opened at the 4th time...... until 0 gets opened at all time.

After I reboots the PXI chassis, it repeated starting with 6 references opened again.

Any idea on what is wrong?

Thanks!

---Ray

[Nick-A]

Hi Ray,

Error -63198 is an error that indicates that your RIO is out of memory.  Are you closing your FPGA VI Reference before opening the next?  Posting a snippet of your code where you are opening the references might be beneficial as well to troubleshooting this issue.

Nick

[rayqin]

Hi, Nick,

Thanks for quick reply!

We have to open 15 references at the same time to monitor data from 15 FPGA cards in a PXI chassis. The program has to be running without stop for hours. Actually the programs have been working for 5 yrs on our 1st system, with Labview 8.2 and RT/FPGA modules.  I am converting them on 2nd system to Labview 2013 with new RT and FPGA modules.

Here attached screenshots of parts of host program and fpga program.

Your help is truly appreciated!

Ray Q.

[Nick-A]

Hi Ray,

A good step to take is to look at the amount of Physical and Virtual memory in MAX.  You mentioned that after you restarted your chassis, you would be able to load less and less FPGAs.  Could you monitor the amount of "Free Physical Memory" each time?  If you can confirm that this gets reduced to 0, this will confirm that the issue is in memory.

Let me know how that goes.

Nick

[rayqin]

Hi, Nick,

Total physical Memory: 2.9GB

Free physical Memory: 2.76GB

Largest Available Memory: 1.9GB

The numbers stayed the same after each run of the program. And I did refresh it everytime. the CPU total load numbers kept changing after refreshed. So the host is OK. In MAX, if I click every FPGA card, it showed the error -63198, after the programs run several times.  I even changed to smaller FIFO sizes (from 16383 points to 1024) but the error was still the same.

Is there any way to tell if the bitfile in each FPGA is running or not? or anyway to see/adjust the limits of resources in FPGA card?

Ray Q 

[Nick-A]

Hi Ray,

It might be a good idea to start troubleshooting the FPGA's one at a time.  Create an FPGA VI that does something very simple and try connecting to just one of the FPGAs.  Increase the number of FPGA references and see if you can successfully connect to them.  I would be interested if communication can be made even if it is very simple.

When you compile the bitfile, there is a log that tells you what resources you have used and what is available in your FPGA.  This would be a good place to see if you are pushing the boundaries of resources in each FPGA.

Let me know how this goes!

Nick

[rayqin]

Hi, Nick,

I have tried using a simple FPGA vi and the error is still there. Actually I have disabled all codes in the host vi except those 15 of "Open FPGA reference". The program still opened the first 7 of them and got "0" for all other references, and then less and less after rerun. It even hasn't reached the points to run vi on each FPGA card.

I also tried opening each FPGA reference and closing it right away.  by doing this I can get valid reference out from each of 15 FPGA cards.  But we need to moniter all 15 FPGA cards at the same time

Is there any limit for how many refrences a system can open?  and can we change it? 

FYI, the real time controller is PXI-8119RT, the FPGA is PXI-7842R.

Thanks!

Ray Q.

[rayqin]

I found there is an error code 63186  ---The number of open RIO sessions exceeds the recommended limit. For optimal performance, close RIO sessions when you no longer need them.

Specifically, where to set this recommended limit?   thanks!

http://zone.ni.com/reference/en-XX/help/370984R-01/criodevicehelp/ni_rio_fcf_error_codes/

[Nick-A]

Hi Ray,

To answer you question about the recommended amount of sessions that can be open at once, here is a KnowledgeBase Article that states you should not open more than 16:

http://digital.ni.com/public.nsf/allkb/315284EE35AC823986257441004DD8A9?OpenDocument

This issue appears to be a memory issue.  The controller you are using has 4 GB of memory on it. Does your computer by any chance have more than that?  What you can try is to run the VI from the host computer and see if having access to more memory allows you to open more references.

Nick

[rayqin · Accepted solution]

Hi, Nick,

The problem was resolved. I was setting the resouce for each FPGA in its properties as "rio://100.0.0.3/RIOx" to match what it showed in MAX. I also use the same string as resource name to open vi reference in block diagram. and it errored out with -63198.  After I change both to "RIOx::INSTR" or "RIOx" for all FPGA cards, then it works. Don't understand why it required this way, but it works. You may report it for future improvement if you think so.

FYI, I have attached a screen shot here.

Anyway, thanks a lot for your help!

Ray