Allocating Host (PC) Memory to do a DMA Transfer

[PJM_JKI]

Hi

We have an application where we need to have a custom PCIe board transfer data to the PC using DMA.

We are able to get this to work using NI-VISA Driver Wizard for PXI/PCI board.

The recommended approach is to have VISA allocate memory on the host (PC) and have the PCIe board write to it (as seen below).

While this approach works well, the memory allocated by VISA for this task is quite limited (~ around 1-2MB) and we would like to expand this to tens of MB.

Note: The documentation (and help available on the web) regarding these advanced VISA function (for instance "VISA Move out 32" and "VISA Move In 32") is parse. If someone has some deep knowledge regarding theses, please feel free to share how we could allocate more memory.

Since we are not able to allocate more memory using the VISA function at this time, we investigate doing the same operation using the LabVIEW Memory Manager Functions which allow us to allocate much larger memory block.

Below is the resulting code.

Unfortunately while we can validate that reading and writing to memory using this work well outside the context of DMA transfer, doing a DMA transfer do NOT work (although the board think it did and the computer is not crashing).

We are wondering why this is not working and would welcome any feedback.

Note: the DMA transfer implemented on the board requires contiguous memory for it to be successful. I believe that the LabVIEW Memory Manager Functions do allocate continuous memory, but correct me if I am wrong.

To troubleshoot this, I did allocate memory using the LabVIEW memory manager function and try to read it back using VISA and I got a "wrong offset" error (Note: This test may not be significant)

Another data point; while the documentation for DSNewPtr do not talk about DMA transfer, the one for DSNewAlignedHandle does. Experiment using LV memory manager Handles has not got us anywhere either.

We would welcome any feedback regarding either approach and about the LabVIEW Memory Manager Functions capabilities in that use case.

Thanks in advance.

PJM

Note 01: We are using LabVIEW 2014 if that matter at all.

Note 02: This question has also been posted on LAVA.

[the_mitten]

Hi PJM_JKI,

I saw the comment on your post on the LAVA forum, and it looked like there were a couple good suggestions posted regarding the Memory Manager functions. I wanted to go in a different direction and ask a few questions regarding your first approach using VISA.

• Can you provide a letter of volatility for the PCIe board?
• What is the result of running the VI in the first image?
• How do you know that it is only allocating memory of around 1-2 MB? Are you only reading back a certain portion of the data, or is it causing an error with data larger than this?

Regards,

[PJM_JKI]

Michael,

Can you provide a letter of volatility for the PCIe board?

I dont have this handy, but I might be able to find out from my customer if needed.

What is the result of running the VI in the first image?

---

Well, this is a tricky question since I greatly simplified what it take to get a DMA transfer going but it can be summarized loosly to what the first image look like (or typically as the steps outlined below):

1. Tell the board to initiate the DMA transfer to the specified address (by writing to a board register the specified address)
2. Wait for the board to be done doing the DMA transfer (through polling a register or receiving an interrupt)
3. Read the content of the DMA buffer into a LabVIEW native Array
4. Repeate from step 1 as needed

How do you know that it is only allocating memory of around 1-2 MB? Are you only reading back a certain portion of the data, or is it causing an error with data larger than this?

---

If we tried to allocate too much, we get a VISA error [-1073807300] (as seen below):

Please note that sometime we are able to allocate more (but never a whole lot more) depending of when we are trying to do the allocation (for instance if this is the first thing that we do after a reboot we can allocate more which make sens). As far as I know, 6-7MB has been the max we were able to allocate. What we would like to do is to initially allocate around 30MB consistently (if possible).

Note: I did forgot to mentioned that we are stuck with LabVIEW 32-bit at the moment so we can not use the "VISA Memory Allocation Ex" (see below) but I am wondering if this would allow us to allocate more.

Your feedback is greatly appreciated. Thank You!

PJM

[the_mitten]

Hi PJM_JKI,

I've done some digging in some of our internal documentation, and found that you were not the first to encounter this situation. A customer ran into a similar issue in the past while trying to allocate a certain memory buffer via VISA. I found the following response from R&D that I think may provide some deeper understanding about what the VISA allocate function is doing: 

VISA does indeed allocate contiguous memory. At the very bottom level of our driver, we are simply making a call to the OS and ask for the amount of memory user specified. The OS memory manager typically would try its best to avoid memory fragmentation. However, how well it can do is completely system dependent. A system at boot up probably has more contiguous memory available than a system that's been running for a while.

I wouldn't feel comfortable at giving any kind of guidelines to customers regarding how much memory can be allocated, because really there is no guarantee. This is a question to Microsoft, not to us. They should be able to tell better what sort of guarantees are available given their memory management code. If the system always only run a specific set of application and nothing more, you have more guarantee that the amount of contiguous physical memory available would be somewhat stable

Typically, DMA requires you allocate physically contiguous page-locked memory. What you do is give the address to the hardware and it just read from there. This kind of memory is exactly what the VISA vi allocates. On the other hand, I believe MemAlloc allocates virtual contiguous non-page-locked memory. 

---

One suggestion that they proposed was to allocate several small chunks of memory; would that be a possibility for your application? Could you break the data into smaller amounts and write to individual blocks of memory?

Regards,

[PJM_JKI]

Michael,

I've done some digging in some of our internal documentation, and found that you were not the first to encounter this situation. A customer ran into a similar issue in the past while trying to allocate a certain memory buffer via VISA. I found the following response from R&D that I think may provide some deeper understanding about what the VISA allocate function is doing: 

VISA does indeed allocate contiguous memory. At the very bottom level of our driver, we are simply making a call to the OS and ask for the amount of memory user specified. The OS memory manager typically would try its best to avoid memory fragmentation. However, how well it can do is completely system dependent. A system at boot up probably has more contiguous memory available than a system that's been running for a while.

I wouldn't feel comfortable at giving any kind of guidelines to customers regarding how much memory can be allocated, because really there is no guarantee. This is a question to Microsoft, not to us. They should be able to tell better what sort of guarantees are available given their memory management code. If the system always only run a specific set of application and nothing more, you have more guarantee that the amount of contiguous physical memory available would be somewhat stable

Typically, DMA requires you allocate physically contiguous page-locked memory. What you do is give the address to the hardware and it just read from there. This kind of memory is exactly what the VISA vi allocates. On the other hand, I believe MemAlloc allocates virtual contiguous non-page-locked memory.

---

---

Thanks, this does confirm our observation.

---

One suggestion that they proposed was to allocate several small chunks of memory; would that be a possibility for your application? Could you break the data into smaller amounts and write to individual blocks of memory?

---

The DMA controller on the board do not support scatter gather dma (but it may in the future) so this is not an option that we can use at the moment.

Once again, thanks for taking the time to research this.

PJM

[rolfk]

The officially right approach to this would be to write a kernel driver for that board and for convinience of the programming user an accompagnining user space DLL that provides a simple API to the kernel driver interface.

Using generic APIs like VISA or any other such API that tries to generalize the kernel space integration of hardware into a user space API is always only a bandaid, since its generalization always means various compromises have to be made in terms of performance, accessible features and what else.

A kernel driver comes into play pretty early on during the boot procedure and could reserve some amounts of memory right there, although it is definitely not nice citicenship of a driver to play such tricks on the OS. And it could emulate scatter gather DMA with reasonable performance since it doesn't have to go through the ring3 <-> ring 0 context switching for every new kernel API operation.

The only problem about writing a kernel driver is that it is an entirely different programming expertise that not to many can manage. Compared to high level C source code debugging, it feels like going back to washing your clothes at the river with soap and cold water and mangling it with your own hands. And if you compare it to LabVIEW programming ..... well I'm sure you get the picture 

[praveen91]

Hi

Re: Execution time of RS 422 VISA write.

Its been Observed the execution time of the VISA Write is around 4.6 ms; The data size is 64 bit and baud rate is 115000 and 57600.

How to write serial with DMA, I wasn't able to find any example; please suggest technique to write data.

[rolfk]

Unless you write your own kernel device driver for the serial hardware interface and call it directly from within your application you really can't! The serial interface as used by VISA relies on the OS serial port communication infrastructure and hides anything specific to handling of the hardware behind not only a user space API (COMM interface) but also a kernel device driver that has to conform to very specific interface requirements to integrate into the COMM driver interface that is mandated by Windows.

[AsadAA]

Hi,

I know this is an old post. but maybe could you tell us how at the end you solved your problem? were you able to use the NI Visa drivers or did you guys write kernel drivers for the PCIe card?

Thanks