Strange FPGA FIFO DMA behavior

[giacomociani]

Hi all,

let me warn that 1) the post is long, and 2) I'm conscious that the problem I'm having has many workarounds (some I know, some I don't, some I think to know...). However, I woul like to understand exactly why it's happening, both to be able to find the most efficient solution and for personal "culture"...

So, I have a very simple VI running on an FPGA: at each cyle, it reads 8 ADC channels and build an an array of 9 elements (time+8 channels). I'll refer to one fo this 9-element array as a "data vector". Because I have to pass this data vector to the host VI, I set up a DMA FIFO in which I write the elements in order: time, ch0, ch2 ... ch7, time, ch0, ch2... etc (I can't directly wirte arrays in the DMA FIFO).

The host VI opens a reference to the fpga VI, invokes the start method and then enters a timed loop. On the first iteration it attempts to read 0 elements, obtaining the info on how many elements remain in the FIFO. Then using a feedback node it passes this information (N) to the next cycle, in which it reads K = 9*floor(N/9) elements (i.e., a integer number of data vectors) and reshape them in a Kx9 array. Finally, it formats the array and write it to a text file. Then the cycle repeats using the number of remaining element obtained from the previous iteration.

Now, the problem is that in the host VI the user has the ability ot set both the sampling frequency of the FPGA and the timed loop frequency. While this allows to find the preferred match of sampling frequency and wiriting (to file) interval, it may happen that the values are such that the FIFO saturates (i.e. the timed loop that empties the FIFO executes too slowly compared to the FPGA acquisition that fills it), and this has two main consequences. One is loss of data, that I don't care in this case as long as I notify the user that he is loosing data. The other one is that, even when i set the parameters back to a combination that allows the buffer not to be filled, I've lost trace of the "first element" of my data vector.

Now, I thought I had an explanation for that: when the buffer saturates, the "FIFO write" in the FPGA code will keep timing out untill the buffer is (at least partially) freed, that can happen in any moment of its cycle. Let's suppose that it found the FIFO full when it was trying to write ch3, and then resumed (because it found free space in the FIFO) when trying to write ch0: this would result in an incorrect sequence of data (time, ch0, ch1, ch2, ch0, ch1, ch2, ch3...) that would compromise any subsequent reading.

However, I'm observing another behaviour on another routine that makes me doubt about this explanation. (I told you the post was long!).

So, actually (I layed you before...) the FPGA code is writing two identical FIFO (FIFO2), not just one. This is because I want onether routine to run and visualize data independently from the one that writes them to file. This "visualization routine" opens a reference to the FPGA code, doesn't invoke the start methode and enters a loop that is almost identical to the one of the host VI, except that it sends the shaped array to some graph and similar instead of writing it to file. Now, because this routine is meant to be run as the user needs to watch the data, it often happens that it is started with big delay with respect to the FPGA code, resulting in FIFO2 being full. Thus, as expected, the reconstructed array is screwed up because the first element is no more what it is expected to be. However, stopping and restarting this rountine whithin an interval shorter than the time needed to fill the FIFO2 buffer (that is only possible if the sampling frequency is relatively slow, as stopping and restarting a routine takes at least a couple of seconds) makes it work. This doesn't match to what I would expect from my previous explanation, because stopping this rountine doesn't stop the FPGA code (i don't want to) and doesn't reset the FIFO2 buffer, so if there is a shift in the order of the elements from a previous timout, it should be maintaned...

Understanting why this is happening is interesting to me not only because it demontrates that I don't really understand how this FIFO thing works, but also because it would probably suggest me a simple solution to implement in both routines when I detect a write timeout from the FOGA code.

Does all this make sense to you?

Thanks

Giacomo

[JLewis]

Hi Giacomo,

I don't have an answer for your specific problem, but I did go through a similar exercise in updating the FFT examples to ensure that the host would always be able to display properly aligned FFT frames. One uses a lossless scheme where any buffer overflow results in an error; the other is designed to asynchronously grab frames for display.

I tried to comment some of the gotchas I ran across in getting this to work, so maybe you might find something that would apply to your case. The examples are at C:\Program Files\National Instruments\LabVIEW 2009\examples\R Series\FPGA Fundamentals\Analysis and Control\FFT

Regards,

Jim

[Hueter]

I had some time today and I thought this was an interesting question so based on your description I went ahead and created a LabVIEW project template of how I would implement this application. I used one DMA FIFO in a producer loop on the host then use queues to send the data to two seperate consumer queues. Check out the atachment and let me know what you think.

FPGA VI

Host VI

-Hunter

[Hueter]

I also wanted to mention, that this code does not handle the errors, so you would want to make sure you could handle an overflow. And you can use the same dynamic chunk size you were using before.

[giacomociani]

Hi Hueter,

thanks for the example. I didn't know abut queues and they seems to be a very convenient way to distribute data to more than one "task". The whole structure fo your program is good to prevent saturation of the FIFO and distribute data to other VI or loops in a FIFO-like way. I'm sure I'll solve my problem by adapting your strategies to my code.

As a matter of curiosity, I still would like to understand how to handle a FIFO saturation, especially because the fact that stopping and restarting the visualization routine solves the problem (as described in my original post) seems to suggest that there is a mechanism to recover "sync" of the FIFO queue... what would it be? 🙂

Thanks

Giacomo

[giacomociani]

Hi Jim,

thanks for the suggestions. It tooks a while to check it out, as I wasn't working on labview 2009 and I couldn't find the example. Actually, it looks like Labview2009 has some new function int he FIFO related VIs that can help better manage saturation of the buffer. However, I still can't understand the behaviour I'm experiencing... read my other post if you are interested!

Thanks

Giacomo

[Hueter]

There are really only 4 solutions to an buffer saturation

1. Read Faster (or smarter)

If you can find a way to increase the speed on the FIFO read you will be able to handle more data. You can do this by moving almost everything out of your FIFO read loop, and using the prouder consumer architecture. You can also increase your chunk size to read out more data at once. If you read more at once you reduce the loop overhead per sample and can read more in the end.

2. Write slower

As you get close to overflowing, using the elements to write property node which shows you how many empty FIFO slots you have left. Once you detect that you are getting close to this limit, you can slow down your loop or just write every other sample. Depending on the application this can work for preventing those infrequent overflows when your system resources are maxed.

3. Dump Data

If you run out of room, just throw out all (or some) of your data and make room. If the nature of your application allows for this it can be a powerful option. You might want to let your user know when you need to dump data, but as long as it is not too often this can be a good solution

4. Bigger FIFO

You can simply make the FIFO bigger, more room means longer before you overflow.

-Hunter