.NET wrapper around dll has trouble allocating memory when called from LV, Python is OK

[Ben]

I have a customer that is having difficulty calling his dll that is in a .NET wrapper from LV (8.5.1)  due to message indicating he could not allocate enough memory. The dll accepts a file spec and is suposed to analyze the results. He says it works fine when called from Python.

I have asked the customer to try

1) calling the dll he has directly from LV

2)and through the .NET wrapper with a smaller file.

Has anyone encountered anything similar?

Or,

 Do you have suggestion on how to help the dll allocate the memory it needs when the .NET wrapper is called from LV?

No I do not have the files for you to try out yourself.

Thank you in advance for reading this thread.

Ben

[smercurio_fc]

Has it been verified that the memory is actually being allocated correcly when the .NET wrapper is called from Python? Is it possible that Python is not catching the .NET exception (I'm assuming that's the error you're getting in LabVIEW). Since you can't provide the files can you provide some information on what the DLL is allocating? Based on what I read I'm guessing it's the file contents as a byte stream?

[Ben]

Good questions Saverio.

"Is it possible that Python is not catching the .NET exception... "

Until I get back with him, I'm going to wing it a little. I am under the impression that when called from Python he is getting the parsed contents of his data file back from the dll call. Since I don't know Python, I am not sure if Python could be missing an .NET exception and still return results.

"some information on what the DLL is allocating? "

I was lead to believe the customer thinks the exception is the result of trying to malloc a buffer for the large file.

"...file contents as a byte stream? "

 I believe you are correct!

Ben

[trearick]

The problem is really specifically around memory allocation, more so than the large file.  Because we have such large data files, the underlying C code tries to allocate enough memory to pull the whole file into memory at once.  It is the large memory allocation that is failing.

I've verified the problem using a small, simple test dll.  All it does is allocate a chunk of memory on the heap (actually, it will allocate N chunks of any specified size).  Using this test code, I think I understand what's going on.

If I try to allocate a large single buffer it will often fail.  (the exact size varies, but it consistently fails around 800MB in size).

If I instead allocate many smaller buffers, I can get maybe 1.6GBytes of total memory allocated, albeit in smaller pieces. 

Labview uses the heap internally for it's own allocation and in the process it must fragment it a little more than the other environments I am trying to operate in.   I think the answer is that I'm going to have to change the C code to be able to handle splitting the file's image across multiple smaller buffers if I want to integrate with LabView.

Todd

[smercurio_fc]

It seems that you would need to do that regardless of the environment since you can never know how big of a contiguous memory chunk you will have. LabVIEW does chew up a fair bit of memory, but I can't make any comment to it fragmenting the heap any more or less than other programs, since that would depend on the other program. All in all, though, it doesn't sound like there's an inherent issue with the ability of calling this .NET wrapper of this DLL from LabVIEW.

[Ben]

Hi Todd!

Thank you for claifying.

RE: "If I instead allocate many smaller buffers, I can get maybe 1.6GBytes of total memory allocated, albeit in smaller pieces. " [emhesis added]

The largest block I have been able to get reliably is 1.2 Gbytes, but that number comes from LV 7.1, before LV was ported to a 64-bit OS. I have not tried to get more using a 64-bit OS with the newer versions of LV. This limitation was a result of the 32-bit OS where only 2G of address space was available to applications since the MSB of the address was used to flag app vs OS address space.

"I think the answer is that I'm going to have to change the C code to be able to handle splitting the file's image across multiple smaller buffers if I want to integrate with LabView."

I believe that may be the simplest solution. There are two other things that may help.

1) Even before the 64-bit OS was available HDF5 was ported to LV by DF Gray. It does so (if I understand correctly) by using part of the OS memory space. (the link to HDF is a search on HDF5 so you can read for yourself what DF has already written).

2) Try the application on a 64-bit OS.

I hope this helps,

Ben

[DFGray]

The problem is not so much lack of available memory as fragmented memory.  Pre LabVIEW 8.0, with LabVIEW loaded, the largest available memory space was on the order of 1.1GBytes.  LabVIEW 8.0 added functionality at the cost of fragmenting memory more, leaving the largest available space of about 800MBytes (which was verified above).  You can solve this issue in a couple of ways:

1. Allocate your data as several buffers.  I have been able to allocate about 1.4GBytes in LabVIEW 8.x using this method.
2. Treat the disk as your memory buffer and only read what you need at any given time.  I usually switch to this method when my wires start carrying somewhere between one and ten megabytes of data.

You can find out more about handling large data sets in LabVIEW by reading Managing Large Data Sets in LabVIEW. It contains some practical code examples.