Channel Switching during DMA Interleaving/Synchronize data stored in different DMA FIFOs

[Harss]

Hi,

In my application  I wish to transport sample values and timestamps in one FPGA to RT DMA FIFO.

I am able to convert +- 24/5 fixed point into unsigned 32 and split unsigned 64 bit timestamp into two unsigned 32 bits.

I am facing an issue of channel switching on the RT due to DMA interleaving and am unable to reconstruct the values on the RT. I have read several posts related to this issue and the possible reason for this seems to be the filling up of the DMA FIFO and the timeout as mentioned in the posts. I wish to do a continous acquisition at 1MS/sec and do not wish to loose any data while carrying out the acquisition . I have a very large FIFO size on my host and on the FPGA side too the number of FIFO elements are sufficiently large upto 8191 elements.

The time out for the FIFO read on the RT is set to a numeric constant as 0.

Could you please suggest a technique by which I can deinterleave the elements in the correct order on the RT, probably other than using trigerred acquisition or data back up to eliminate the timeout .

Could setting the timeout to -1 on the RT allow me to deinterleave the samples in the correct order. However, I have read this increases the processor utilization.

Any hints on this would be really helpful.

Thanks.

[Harss]

I missed out on a thing could anyone also please help me understand if there is any way by which the FIFO read in case of two seperate FPGA to RT DMA FIFO's could be synchronized on the RT if I use two seperate FPGA to RT DMA FIFO to transfer the sample and the Timestamp values respectively.

In my case any of the two approaches would work I just need to get the sample and the Timestamp values in synch .

Could you please suggest any approach?

Thanks.

[mateh]

The channels are most likely switching because the Write to FIFO method in the FPGA VI is timing out.

Regarding the processor utilization when reading from a FIFO, this KB explains explains the appropriate way to use the Read from FIFO method on the RT/host: http://digital.ni.com/public.nsf/allkb/583DDFF1829F51C1862575AA007AC792

Can you post your project? It will be easier to make suggestions that way.

[Harss]

Hi,

Right now I do not have cRIO connected  , but yes the VI on the FPGA does time out when I attach a boolean indicator to test it.

The Timeout of the FIFO Write is set to 0.

Any suggestions on changing it to -1?

The VI used is similar to the 'NI 9223 User Controlled I/O Sampling' available in the Labview 2014 examples section.

Except the DMA interleaving where the sample values and the Time stamp values which are obtained from the 'NI Time Sync FPGA Timekeeper'

 are being used.

I will post the code as soon as I am able to connect to cRIO.

Thanks.

[Harss]

Hi,

I have attached the image file of my FPGA, RT and Host VI .

I tried changing two things in my RT VI

1:Set the requested depth to integer multiples of 3 because I am using 3 DMA channels

2:Set the Time out of the RT VI to -1.

However, the changes still could not resolve the channel switching issue.

Please let me know if you need any more details. 

Any help / suggestion would be really appreciated.

Thanks

[Harss]

Hi,

I forgot to mention that in the current scenario(in the snapshot) I tried to use network stream to pass data from a single channel from the FIFO , but it seems they all are not in order  and I get unexpected data from other channels too when they are logged in the TDMS file.

Thanks.

[mateh]

Right. One reason you're getting unexpected data from the network stream is because the FPGA FIFO is timing out. I would disable the network streams for now and focus on debugging the FIFO timeouts first. Once you get that working, then worry about the host VI.

Keep the FIFO timeout at zero in your FPGA VI. If this part is timing out, you must find out why the RT VI is not reading from the FIFO fast enough. Setting the timeout to some other number (or -1) on the FPGA side will only mask the problem and potentially cause you to miss data points because the FIFO is sitting there waiting for more space to free up.

I don't see the point in having two separate while loops in the lower part of your FPGA VI. The FIFO Write has low enough overhead to support your sample rate, so you should be able to wire the values directly into the target-host FIFO instead of using a local FIFO and going through shift registers. Could you not remove the while loop at the bottom completely? See the picture I attached. In this case you would immediately split the time into two U32s, read from the module, and write all 3 U32 values to the target-host FIFO, all in the third frame of the sequence. Also, if you need to write multiple elements to the same FIFO, wire them into an array and place the FIFO in a For Loop instead of creating 3 separate instances.

Regarding your RT VI: What is the Sample Period? 1? Seconds, milliseconds, microseconds? What we really care about is the calculated value that you're sending to the FIFO Configure method. You don't need to configure the FIFO Depth in multiples of 3, just make it sufficiently large so the buffer doesn't fill up. If your sample period is in microseconds and has a value of "1", that means you're configuring the FIFO for 30000 elements which isn't nearly large enough considering your FPGA VI is writing 3 million elements per second. For now just wire a large constant into the Requested Depth (something like 300,000 or more).

The way you have written the RT While Loop will result in unpredictable behavior. Setting the timeout to -1 in the first FIFO read won't do anything because you are reading 0 elements, and setting the second timeout to -1 also won't do anything because you're telling it to only read the number of elements that are already there, so there's no need for it to wait. You can set the timeout to 0 in both of these cases.

One other problem I see, is that no matter if there's a single element or 2 elements or 300k elements in the FIFO, the VI will immediately read whatever is there and try to write it to the network stream, which has some overhead associated with it. This is also bad because you aren't explicitly telling it to read elements out in groups of 3, which means when you decimate the array by 3 any extra elements that it reads would get thrown away... this will also cause your channels to switch. It would be better to read a large chunk of data at regular intervals, that way you can limit how often the RT VI attempts to write to the network stream. Look at the example code at the bottom of the KB I sent you earlier. That code reads the elements out of the FIFO in groups of 1000 samples. You would want to change this so it reads in multiples of 3, and make it a large enough number so that your VI doesn't try to write to the network stream so fast that it can't keep up.

[Harss]

Hi,

Thanks a lot for the detailed explanation. This has really made some things clear

I will definetly try the solution suggested by you tomorrow and get back to you asap.

In the FPGA VI . The configuration used is 'NI 9223 User Controlled I/O Sampling'. Removing the third while loop would reduce the data rate to 350 KS/s according to the link shown below

http://digital.ni.com/public.nsf/allkb/5250C3AAE0CBAAE68625777F0072438E

I would still try all the possibilities which are suggested and get back to you.

Thanks once again:)

Thanks.

[Harss]

Hi,

I just missed out on one detail . The size of the FPGA FIFO is set to 8191 elements. The acquisition is continous at 1 microseconds interval.I would increase the FIFO size on the RT but I dont realize the exact size of FIFO to be set on FPGA as well as RT to completly eliminate the time out issue. Is there any documentation providing details about the procedures used to eliminate timeout without any loss in data , in case continous acquisition is to be carried out.

[mateh]

I see what you mean about the parallel loops now, sorry for the confusion. 

The reason for having a separately configured FPGA FIFO size (Requested Number of Elements) is because a small buffer is created within the FPGA which temporarily stores the data that gets written to the FIFO until the DMA transfer to the RT-side buffer can take place. Since this transfer occurs automatically and at a high rate of speed, only a small size is needed here. Increasing Requested Number of Elements uses up more FPGA resources and is unlikely to eliminate the timeouts.

You can judge how quickly the FIFO is filling up by looking at the "Elements Remaining" terminal of the Read from FIFO method in your RT VI. This tells you the number of elements that are sitting in the RT-side buffer waiting to be read. Ideally this number will fluctuate back and forth as the FIFO is continuously written to and read from, but it should never get out of hand. If Elements Remaining approaches the FIFO depth that you specified at the start of the VI (let's say 300,000), then you know that the problem lies within your VI not being able to read fast enough.

Consider how many samples you're trying to read from the FIFO with each loop iteration, and compare it to how long it takes for the FPGA to write a given number of samples (sample period divided by 3, in this case). For a continuous acquisition, the loop shouldn't take longer to execute than the period of time represented by the number of samples you're trying to read. To put it another way, if you try to read too few samples at a time, the loop execution time will be greater than the very short period of time that it takes to write those few samples, and you'll get timeouts.