Strange FPGA FIFO data corruption.

[jtrujil1]

Hi all,

   I've got very tough problem I've been working on all week. We have a custom made circuit board that reads digital data from these AFE0064 charge sense chips. We are making an X-Ray detector with them. We are using an sbRIO-9602 to drive all the digital control signals and read in the 12-bit digital data. Our initial prototype read a single AFE0064 just fine, but now we are trying to scale up.

What is being done now is we have created a module with two AFE chips. Through the design of the board I can read all 128 data elements in increments of 4 at a time. Each 128 element stream counts as a row, and the driver does this 720 times. I pump all this data through a FIFO buffer from the target to the host, which in this case is the realtime processor inside the sbRIO-9602. After some data processing the final result is a nice x-ray image as show in the attached image.

That constitues one reader module. Now we want to read 12 of these modules simultaneously. The app I have created allows me to pump more data into the FIFO without haveing to change any of the digital control signals operating the FPGA. It's already running all 12 attached modules simultaneously, regardless of whether I read the data into the FIFO buffer or not. The problem is that when I do pump all the data into the FIFO, I see corruption in the image. As shown in the example below (which actually only has two modules attached at each end of the circuit board).

The weird thing is that when I run the code with fake data (a digital pattern created by the FPGA instead of the AFE0064), I don't see this corruption. When I route the data from first module into the digital input lines of the rest of the modles, I do see corruption. I originally thought this might be a problem with our hardware design, but copying the great data coming out of our good module to the rest of the digital input buffer also results in corruption even though I am bypassing the circuit design. So I think it's a software problem, and a very subtle problem with the way I'm pumping data into the FIFO buffer.

The code I'm working with is in the Dropbox link here:

https://www.dropbox.com/sh/t63yds5cp2sxdej/AABPgtXYqfTEm2Gei1_a26Wfa?dl=0

[stockson]

Hi jtrujil1,

could you please post a desciption of what is done in which part of the code? It's pretty hard to navigate thorugh the project tree with files named *test, *test2, etc and almost no folders on the FPGA target 🙂

[jtrujil1]

Hi,

Yes I wanted to last night but I couldn't keep my eyes open any longer.

Ok so here we go. Let's start with my best functioning entry point program: Main_Driver_v5p23.vi

All of the other further versions are just my variations of this program as I try to figure out what's causing my data scramble. Now Main_Driver_v5p23.vi is the host program running on the real-time target of the sbRIO-9602. On the FPGA side Main_Driver_v5p23 is calling AFE0064_Driver_v5p23.vi

The FPGA pumps data into a FIFO buffer I have called Data_Output. It is a signed 16-bit buffer with 4095 elements. The FPGA reads data in 4 MHz bursts. A single module contains 128 elements to be read. The driver can read a module 4 elements at a time. The FIFO reads this 4 element array within the FPGA sequentially, and then the cycle continues until all 128 elements are read. Then we go to the next row in the X-Ray detector and start the process all over again, end up with a data transfer of 128x720 elements. This works great for a single module, but the code is designed to scale up to multiple modules with a few user interface parameter changes (Change "Number of 32 Channel Chunks" from 4 to 48 to see what I mean.). I can read up to 1536x720 elements within the sbRIO-9602, which is the final goal. That's when the data corruption happens. As you can see in the picture the first row scans come out ok, but towards the end, the last several dozen rows get scrambled.

If you have an FPGA and want to run some dummy code you can try out Main_Driver_v5p23_Test, which runs AFE0064_Driver_v5p23_Test.vi and generates fake data (black and white bars). This dummy code runs fine, which makes me think that my FIFO buffer size and reading scheme is ok. But maybe it's not, and that's why I'm posting here. It could also be a flaw in our circuit design, but so far I have not come up with a way to prove that. Each module has it's own set of 4 data bus lines which feed the FPGA for a total of 48 bus lines in parallel. I don't think there is any crosstalk involved because I've written code that reads each module sequentially and I still see noise.

[ks30]

Hi jtrujil1,

Would it be possible for you to compare data before it is written to the FIFO and then after it is read from the FIFO? Have you verified that the data being read in the FPGA matches what you expect it to be? It would be helpful if you could isolate the probelm to the buffer.

Regards,

Kyle S.

Applications Engineer

National Instruments

http://www.ni.com/support

[jtrujil1]

Hi,

That's what I've been trying to do by replacing the digital I/O reads in 48ChannelRead.vi with constants. That way I could see if the buffer is corrupting the data throughput. 48ChannelRead_Test.vi alternates with binary 0's and 1's for the data channels. I get stripes in my output so I assume the buffer is working correctly, but maybe I need a more complicated pattern to catch the problem. So today I'm writing a version of 48ChannelRead that outputs a range of numbers. I should get a nice greyscale rainbow it the code is working correctly. Then I'll save the data and run a Perl script to check for errors in the pattern I'm expecting.

The other thing we can do is probe the digital I/O lines with an oscilloscope and check if it's the same value as what's coming out of the program. But that only gets us the first few elements due to the trigger setup we have a Lecroy WaveSurfer 24MXs that might have a bigger buffer for the waveform but I need to read the manual.

Qualitatively I know that the image coming into the buffer stays intact when I don't read a lot of data out. The fake data patterns I'm generating with 48ChannelRead_Test.vi will help me catch more detailed errors. So far I can't really blame the FIFO for the problem. I'm thinking it's some relationship between the electronics and the changes to the digital waveform that happen when I read more data out. Reading more data out will slow the digital control signal sequence to the chips because the FIFO can only read one element at a time, and inside the FPGA I have an array with 48 elements that needs to be pushed out through the FIFO sequentially over and over again.

Today I'm focused on what works, which is reading the first module of 128x720 data elements. AFE0064_Driver_v5p23 does that well. I'm creating a version of the driver that allows me to address the other modules on the board and read them individually. I can stitch the data together on the PC by just repeatedly calling the FPGA driver in a loop, changing the module address and reading 128x720 data chunks per iteration. Right now I still see corruption when reading just a single module, so that's a clue. My new code should be almost exactly the same as AFE0064_Driver_v5p23, but something I did made things worse. So today I'm looking into that. If I can't see what's corrupting a single module read in the new code then I'll start looking into the digital signals with the oscilloscope again. And test new dummy data patterns. And learn if the LeCroy can help me really poke around the setup.

[stockson]

It looks to me like a good method to debug the code.

I went throught the VIs and unfortunately without knowing the specifics of the hardware I am not sure I will be able to say much. There is one thing I noticed, but I don't know how significant it is in your case:

In this configuration, the ChipSelect.vi can execute before or after the main part of the code (we can't be sure). This can potentially cause some problems.

[jtrujil1]

Thanks for the tip Stockson, I'll try to make things a bit more deterministic. We found out that my original simplified version of the code (AFE0064_Driver_v5p22p1.vi), which just reads a single module, generates a great image. I've hacked a loop around that code since the chip select is user selectable and just stiched the images together on the real-time processor side. It's slow, but it gets the job done well enough for further hardware debugging of a full 12 module system.

I'm reviewing my code now, and little details like what you pointed out probably add up as things grow in complexity. Thanks.