Can using multiple NI PCIe FPGA cards cause Windows to freeze?

[GregSands]

I have two PCIe FPGA cards (7851R Multifunction RIO and 1473R Camera Link) installed in a Dell Precision 5820 running Windows 10, with LabVIEW 2018, all updated to the latest releases.  The Camera Link card is transferring data to Windows through a DMA FIFO at about 400 MB/s, and the RIO card at about 5k/s through a second DMA FIFO.  Every so often (typically a few hours), while running a program that uses both cards, the computer will completely freeze, with a frozen screen, and no response to keyboard or mouse. Programs that use either of the two cards on their own (using the same compiled FPGA code) never freeze the computer, even when run for several days. 

By chance, on one freeze, Windows started to display an error dialog - there has never otherwise been any description of the fault, or any error logged in LabVIEW or Windows. 

The error message doesn't give much information: "An error was detected in the communication between the host computer and the FPGA target."  I am not using any external clocks, only the 100MHz image clock on the 1473R, and the normal 40MHz clock on the 7851R card.

I've checked that it's not a memory fault (running an overnight memory check), and I've tried swapping card slots on the PCIe bus. Neither was successful. I was wondering whether there are issues with conflicts when using multiple DMAs, but haven't found any warnings about that being the case.  I've changed the code so that DMA writes do not occur from both cards at the same time, but this has not solved the problem either.

Does anyone know of a possible cause of these lockups, or how to troubleshoot in this situation?  Or has used an equivalent hardware configuration? It's very puzzling that this fault would take down Windows completely.  Any and all suggestions are very welcome!

[KeddyC]

Hi GregSands,

Thanks for reaching out. This is really strange behaviour. If you have the ability to make a Service Request with us, I'd recommend doing so.

While running at those rates and for the length, I'd definitely be monitoring every resource on your computer.

See if you can find anything in the Windows Event log, and see if a .dmp file is created.

Finally, does it happen to freeze at the exact same length of time, or is there a large standard deviation?

[GregSands]

Thanks Keddy - I'll open a service request and see if anything can be done.  I just thought others here might have seen the same issue with multiple PCIe cards.

I agree it certainly is strange.  There's no pattern to the time till freezing - yesterday for example, it happened after 10 minutes, then 2 hours, then 30 minutes, then not for the rest of the day (>5 hours), but again first thing this morning after a few minutes.  I've checked the Windows logs, and there is no .dmp file written, nor any other indication in the event log - it seems that Windows has stopped before it has any chance to recognise an error - except in the sole case where that error message began to be shown.  I've monitored memory use (only gets to about 1Gb), and the behaviour is independent of whether I'm running any other software on the computer or not.  So it suggests a hardware issue, but only in the case where both cards are used together, perhaps a PCIe bus conflict of some sort.

[GregSands · Accepted solution]

Just for reference, in case any one else runs into the same issue, the problem was eventually solved after a BIOS update on the computer (even though the computer was fairly new).  Thanks to the NI Application Support Engineer who helped with working through this, and suggested checking this.