cRIO Slow Read of FPGA Front Panel: crio-9030 and crio-9039

[Michael.Proctor]

Hi,

I have a real-time application that runs on both a cRIO-9030 and cRIO-9039

There is a  Timed_Loop should cycle at 4000 us and does on the cRIO-9039.

In th etime_loop there are read several front panel controls, in a 4 different groups, as illustrated in the screen shot.

On the cRIO-9039 the read takes ~200 us and this is fast enough.

On the cRIO-9030 ( less memory and CPU cores, less FPGA gates ) the reads take 2000 - 3000 us.

The times meaured is only for the FPGA front panel reads, and not the rest of the call in the snippet shown.

The FPGA front panels are the only process running at this step.

The gates usage on the FPGA is 17% for the 9039 and 70% for the 9030, but I would believe this is not the problem.

With MAX I have installed the same set of componants and checked all other Properties of the cRIO Target and chassis are the same.

Maybe not every C-module is set exactly the same.

There is one difference between the two systems and that is the cRIO-9039 has two NI-9215 analog read modules while the cRIO-9030 has one NI-9205 analog read module.

They in the same FPGA loop, condionally disabled are the read loop time is 2000 us.

The update of of a FPGA Loop that writes to the FPGA front panel, I beleive should not affect the read by the RT controller of the front panel controls as they are pressitant, or so I believe.

1.  If I disable all the front panel reads of the FPGA the execution time fall dramtically.

So it is not other code running 

2.  If I re-enable even one of the read blocks the execution time goes over 2000 us.

I could use shared variables - but given that the cRIO-9039 perfroms adequately I hope I do not need to for project simplicity.

Hopefully someone has a hint at some configuration parameter that I might have overlooked, that is preventing the cRIO-9030 from performing as hoped.

thanks

Michael

[Davidvilla87]

Hi Michael,

I am not completly clear about your issue but I would suggest to try switching the modules from one cRIO to the other one. Let´s see how it behaves. If it is something related to the modules or the FPGA of the cRIO itself.

So, do you have this code inside a single cycle timed loop?

[Michael.Proctor]

Hi David,

The cause might be that now I discover, someone removed one of the NI 9403...

we have three of them installed but I did not notice one was misssing.

Will test in the next day or so if that was the cause.

The code is in a Real-Time SCTL.

[Davidvilla87]

Hi Michael,

Ok perfect. Hope that was the problem. 

[James_McN]

I would not expect that the missing module will cause this issue since the read just reads the current values anyway.

I actually think there maybe a flaw in the way you measure the time taken to read them. Your first timer block could execute before or after the FPGA reads. If the different targets have different core counts this may explain the difference.

You should force the first time read to execute before the FPGA reads by putting it all in a flat sequence structure and then compare between the two targets.

I expect you will get very similar results since the gate usage on the FPGA wont make a difference and a timed loop on real-time is single threaded - the concern is this means the 2000us time might be the correct one.

As an idea of expected performance I would expect each read of a 32 bit number to be in the very low us range (can't remember exactly). From the image though I can't tell how large your clusters are. You may be able to get some performance gain from packing the booleans together as each boolean is transferred from the FPGA as a 32 bit value.

[Michael.Proctor]

James,

thank you for the reply and the hints on correct timing.

Indeed a flat sequence with the fist timer in the first frame is teh correct way.

Howerver my actaulliing timing is coming from a sequence frame in a single timed loop that calls this 'read FPGA controls' vi.

The 9039 are read with with exactly the same structureand the execution time forit is very reasonble. In the 10s us and the FPGA code itself is the same but compiled for each cRIO type.

So if you think missing moduels is not the issue then it is somewhat of a mystery.

Unfortunately at present all our modules are in high usage so I have not had time to check what happens when I replace them.

I will  get back later with an update when I have time to test but filling in hte missing modules.

Michael

[Michael.Proctor]

BTW, nice blog....

Now I have a great reading list for my flight back to California from Italy.

.. and in the future you might be asked a few more question on cRIO performance !!

Like why does a SCTL in the RT side of the cRIO run better when allowed to auto-asign its core, compared to an assigned core... if a SCTL is singled threaded ?

[Michael.Proctor]

A follow up to my initially posting.

We bought and installed the missing C-Modules.

As James suggested this made no difference.

I made some code optimization - not much as we need all the functionality -

The total FPGA is now about 77%  still very low.

I have checked that every aspect of configuration of the two cCRIO have identical software installed, the Properties in the Project are identical.

So no clues to why system reads the FPGA a factor 10 or more slower.

The cRIO-9030 runs at about 53% CPU.  I have a sequence in my timed loop (TL) so I get my timing from the TL previous frame read out.

When I comment out the read from the CPU it has about 100us ( there is some other code ) compared to 3000 - 4000 us when I read the FPGA controls.

The cRIO-9039 process this in about 400-600 us.

My only other thought I think the problem started when I implemented a 9205 Analog In.

I do give it 120 us loop time in the FPGA to read.  I think this is sufficient to read 12 channels, but this should not affect the RT read of the FPGA controls.

Any help appreciated.

Michale

[Michael.Proctor · Accepted solution]

SOLVED.

I have both cRIOs in one project - even though I only ever use them independently.

Through lack of knowledge I left both FPGA in the Chassis with the name RIO0

As seen in the first inclosed image this cause the Run_Init_FPGA.vi to add the IP address tot he RIO id.  I though this did not matter because I am running the RT and FPGA on the same IP, after they are just one unit.

I got the idea that that could be the problem, so I went into MAX and changed the name of the cRIO-9030 FPGA to RIO1,  relink the Run_Init_FPGA.vi to the RIO1 and wham....

FPGA read times down to 700 us.

So... a little clue to anyone else having this problem.

Name each FPGA used in one project, even if you use them independently.

Michael

[James_McN]

Hi Michael,

Thanks for posting what you found - I have come across this once before in my life and it's so odd that it didn't even cross my mind!

Any FPGA can be targeted over the network if the RIO server is enabled on the host. Because the open saw the IP address it must have routed everything through the RIO server.