LabVIEW Memory Space

[EthanHenry]

I've recently started writing a driver with a specially made LabVIEW DLL written by the company, however I've found it much less stable than the normal DLL used by pretty much all other languages.  Therefore, I took it upon myself to learn how LabVIEW memory spaces work and how to interact with the pointers that are sent back and forth, and it was actually working pretty well.  However as I was playing around I realized that the Byte Array type casted representation of one of my clusters was shorter than the struct it was based on.  After doing some searching, I realized the struct was most likely padded/aligned differently.  However I had been using it before by just using handles to clusters, and the data returned always seemed to be what I expected.  From some more searching, I confirmed LV doesn't do any type of aligning or padding with MoveBlock, and thus my question is this:

If LabVIEW doesn't do any padding/alignment, and I'm almost certain the DLL does (using sizeof on the python ctype struct results in more bytes than only the sum), how was I getting the correct data back?  Was I just getting extraordinarily lucky?  Is there some manual/compiler that explains how padding is often done so I can try and recreate it in LabVIEW?  I can't share code as it's all proprietary, however thankfully everything is pretty bare bones, just reading using a pointer to handles and a pointer, writing vice versa.  Thanks!

[cordm]

AFAIK, LabVIEW uses the platform alignment rules on 64 bit. How LabVIEW Stores Data in Memory

Windows: https://learn.microsoft.com/en-us/cpp/build/x64-software-conventions?view=msvc-170

For 32 bit it is different from the platform standard, which is why DLLs need the lv_prolog.h and lv_epilog.h includes. Clusters in LabVIEW Do Not Line Up With Structures in Visual C++

Flattened data is compacted: https://www.ni.com/docs/en-US/bundle/labview/page/flattened-data.html

Depending on your specific cluster the memory and flattened representation can be the same, but not in general. So you may just have been lucky.

[rolfk]

Your language is slightly confusing. It is not the DLL that does alignment or not, it is the compiler that creates that DLL. And that decision is based on either a project wide setting (the so called default alignment) when building that DLL, and additional settings per subproject or even source file in the project that can overwrite this default alignment and last but not least #pragma pack statements in the source code that can again overwrite any previous default or file specific setting.

Windows and most other OSes nowadays use an 8-byte default alignment. This makes sure that any datatype up to an (u)int64 or double is always aligned on its natural boundary in memory. For some CPU architectures this makes a significant difference since they either incur a huge penalty when needing to access unaligned data elements in memory or simply crash. Because they basically access the two adjunct naturally aligned elements, store them in two registers, bit shift each of them accordingly and then combine them to the actual value. Or the opposite but even more complicated when needing to write that value. The Intel x86 architecture has a significant chunk of hardware gates in its register access logic to minimize that overhead by performing most of it in microcode or direct hardware, which is nice to avoid performance hits but a pretty complicated architecture extension that needs to be tested during design and production every time. Architectures without such hardware support need the compiler to simulate it and in some embedded controller cases the compiler simply refuses to do it or create code that will randomly crash.

LabVIEW has for legacy reason byte packing when you run the Windows 32-bit version. This is because LabVIEW always was 32-bit from the start even on the 16-bit segmented register architecture that Windows 3.1 was. But these computers used often 4MB of RAM (yes that is not a typo!) and computers with 8MB of RAM where at that time already considered high end beasts and excessively expensive. Saving bytes by packing every structure in memory to the maximum could make the difference between an application that could do a reasonable data acquisition and display the data on a graph on screen or abort with an out of memory error dialog. When NI introduced LabVIEW for Windows NT, which was a true 32-bit OS they chose to keep the entire memory architecture the same as before to avoid backwards compatibility problems with programs developed on Windows 3.1. Since LabVIEW itself was already fully 32-bit this was in fact a quite easy move and avoided a lot of compatibility hacks, code mutation paths and what else. LabVIEW for Windows 64-bit and all other still shipping LabVIEW versions nowadays all use the platform default alignment internally, since with 16GB of memory, the bytes saved by removing filler bytes in structures at most amounts to a tiny fraction of memory wasted in comparison to what you have available for use. This alignment happens to be for all these platforms 8-bytes.

Also a byte packing environment like LabVIEW 32-bit can easily allow a user to emulate other alignment requirements by adding filler elements in clusters to match the memory structure of the aligned C struct. The other way around is a lot more complicated. If your 64-bit DLL uses byte packed parameters (yes this actually happens, I just developed a driver for such a DLL) you can't just configure a Call Library node to take out those filler bytes but instead have to build that memory struct yourself in a byte array and pass it as such.

A good thing to know is that the LabVIEW Flatten and Unflatten function still will flatten structs to a packed format for backwards compatibility reason since the flatten format in LabVIEW is supposed to be on all platforms always the same, so if you really need packed structs you may get away with using these functions. However beware of clusters with arrays and strings inside. These are variable sized elements in LabVIEW and LabVIEW therefore will ALWAYS add a 32-bit size integer per dimension in front of the array data. And that is almost never what a C struct would expect no matter its alignment requirements.

Most of the time you can get pretty far, if you know the details about how C compilers do data alignment, and what alignment settings your DLL uses and observe the byte packing of 32-bit LabVIEW for Windows. The difficulty here is often that the DLL doesn't explicitly state in the documentation what alignment it uses. If the according header file doesn't use #pragma pack statements you can usually assume that it uses default alignment but there is no guarantee. If the DLL developer is evil enough, or maybe just masochistic, he may have changed the default alignment in the project settings itself and you will never see that unless you get the entire source code and project files for that DLL.

If you really need specific byte packing even on platforms where LabVIEW does not use byte packing itself, you might get away by using LabVIEW Flatten and Unflatten if the struct does not contain arrays or strings pointers. If you absolutely and positively want to go the hard way you either have to create a wrapper DLL that translates between the LabVIEW alignment and your DLLs alignment or you could use a library like this: 

https://www.vipm.io/package/easlib_memory_manager/

With this library you can build a memory buffer with specific alignment settings element by element and also parse a buffer after it was returned by a DLL function. It is not for the faint at heart, you really need to know about C programming, how a C compiler treats structs, C memory alignment and similar topics.

This library is an example of how to use it:

https://www.vipm.io/package/easlib_win_credentials/

[EthanHenry]

Ah yes I already ran into the string and array headers in Labview and have gotten around them by casting clusters to byte arrays

Hm perhaps I misunderstood/was mixing up information, so let me lay out everything I know without any assumptions:

1. The DLL was given by a third party, along with a Python example code that uses ctypes

2. Using sizeof on a struct returned 56, even though the sum of the bytes in the struct add to 49 bytes

3. Using LabVIEW's MoveBlock function with the DLL, and reading data (an array of U16) at the address the DLL returns seems to get back the expected data.  I can also create a pointer in LV, pass it to the DLL, and then read the struct at the pointer based on the DLL's documentation and get back data that seems correct

4. I'm also able to create pointers in LV, then write data in a cluster to the address of the pointer, pass the pointer to the DLL, and have it interact seemingly without issue

All of this to say I'm not sure what alignment settings they compiled the DLL with, but I'll follow up with the third party to see if I can get any more information.  Thanks for all the info, very helpful!

[rolfk]

---

@EthanHenry wrote:

All of this to say I'm not sure what alignment settings they compiled the DLL with, but I'll follow up with the third party to see if I can get any more information.  Thanks for all the info, very helpful!

---

If you can post the *.h header file of that DLL or the Python interface that uses ctype I could probably tell you why that discrepancy exists. There are many possible reasons, including the ctypes function sizeof() being a bit overzealous.

[EthanHenry]

Huh the forums won't let me upload the file as a *.py (says the content type doesn't match, maybe cause it's using ctype?), was able to get around it by zipping it up.  The file is available from their website without signing in so I assume it's fine to distribute

Here's the main Python Interface with the DLL.  I ran the following code:

from ctypes import *
from TUCam import *
import time
import os
import sys
from datetime import datetime

info = TUCAM_FRAME()
print(sizeof(info))

Prints 56, even though the sum of the TUCAM_FRAME() struct is 49 bytes.

[rolfk · Accepted solution]

Well you have miscalculated a few things.

Since the structure is not defined with a specific pack attribute, it uses the default alignment of 8 byte.

class TUCAM_FRAME(Structure):
    _fields_ = [                         off_32 size_32 off_64 size_64
        ("szSignature",  c_char * 8),     0        8       0       8
        ("usHeader",     c_ushort),       8        2       8       2
        ("usOffset",     c_ushort),      10        2      10       2
        ("usWidth",      c_ushort),      12        2      12       2
        ("usHeight",     c_ushort),      14        2      14       2
        ("uiWidthStep",  c_uint),        16        4      16       2
        ("ucDepth",      c_ubyte),       20        1      20       1 
        ("ucFormat",     c_ubyte),       21        1      21       1 
        ("ucChannels",   c_ubyte),       22        1      22       1 
        ("ucElemBytes",  c_ubyte),       23        1      23       1 
        ("ucFormatGet",  c_ubyte),       24        1      24       1 
                  filler                 25        3      25       3 
        ("uiIndex",      c_uint),        28        4      28       4
        ("uiImgSize",    c_uint),        32        4      32       4
        ("uiRsdSize",    c_uint),        36        4      36       4
        ("uiHstSize",    c_uint),        40        4      40       4
                  filler                 --       --      44       4
        ("pBuffer",      c_void_p)       44        4      48       8
    ]                                    48               56

So the uiIndex element which is 4 byte sized needs to be aligned to a an offset that is a multiple of 4 bytes.

The pBuffer needs to be aligned to an offset that is a multiple of 4 bytes in 32-bit mode, but the offset must be a multiple of 8 bytes for 64-bit mode. 

[EthanHenry]

Oh man, seems I really didn't calculate correctly, guess that's what I get for not double checking haha.  I see though, so each element has to be offset by a number divisible by it's byte count? u16 needs to be offset divisible by 2, u64 needs to be offset by a number divisible by 8, and a pointer changes from 4 or 8 depending on bitness of 32 or 64 bit respectively, is that correct?  Also, reading one of your previous messages, I have to assume labview is not an 8 byte aligned program, is that correct?  So I'd need to offset/add filler elements to make sure memory was correctly written.

[rolfk]

Well, not always. It depends what alignment (packing) was in place when the relevant structure is defined. If no special alignment is selected then the default alignment nowadays on pretty much every platform (except possibly some lower end embedded systems) is 8 byte. 

Each element in a structure is aligned to a value that is the smaller of the specified alignment or its own size. So if you had a 16 byte sized element (the only one currently sort of possible would be a long double but that is almost never used anywhere and not all compilers support it), it would still be aligned on a multiple of 8 bytes, since the alignment in place if no specific alignment is specified is the default alignment of 8 byte.

And LabVIEW only uses 1-byte alignment for its own data in LabVIEW 32-bit for Windows. On all other platforms such as LabVIEW for Linux, Mac OS and LabVIEW 64-bit for Windows it uses the 8-byte alignment.

[EthanHenry]

Ah gotcha, but the rest of the logic checks out for this case, correct?