Initializing multiple array pointer member in a cluster, inside .so/.dll

[Cklai]

I have previously coded a C dll file that is planning to initialize a struct with multiple array variables with it. The struct file look like this:

typedef struct Pointers_Double
{
unsigned int Nx; // Number of input
unsigned int Ny; // Number of output
unsigned int I; // FIR Filter order per channel
unsigned int I_ori; // FIR Filter order per channel, without padding (original input).
unsigned int registry_size; // Double = 4
double *H; // Multichannel FIR reshaped 1D array ( order: I -> Nx (input) -> Ny(output) )
double *x_tilde; // Multichannel FIR input signal (order: I -> Nx )
} Pointers_Double

I have tested the library in Windows PC and it work, so I am migrating it to Linux RT device (with .so Linux compilation and stuff in Eclipse as well). However, I keep receiving SIGSEV error from RT Target at the initialization stage alone. Manual debugging shows that it fails to perform calloc and returns a NULL pointer for the H variable at the end. I attach the C code initialization function below:

void FIR_Double_Initialize(MIMO_FIR_Pointers_Double *ref, unsigned int const Nx, unsigned int const Ny, unsigned int const I, double *H_init)
{
    ref->registry_size = 4; // Double = 4;
    ref->Nx = Nx;
    ref->Ny = Ny;
    // Note: I >= I_ori in all cases.
    ref->I_ori = I;
    ref->I = I + ((I % ref->registry_size) != 0) * (ref->registry_size - (I % ref->registry_size)); // Make sure its a multiple of the vector size.
    ref->x_tilde = calloc(ref->I * ref->Nx, sizeof(double));
    if (!ref->x_tilde) return;
    //return;
    ref->H = calloc(ref->I * ref->Nx * ref->Ny, sizeof(double));	
    if (!ref->H) return;
    if (I == ref->I)
    {
        memcpy(ref->H, H_init, ref->I * ref->Nx * ref->Ny * sizeof(double));
    }
    else
    {
        for (int ny = 0; ny < Ny; ny++)
        {
            for (int nx = 0; nx < Nx; nx++)
            {
                memcpy(&ref->H[nx * ref->I + ny * Nx * ref->I], &H_init[nx * I + ny * Nx * I], I * sizeof(double));
            }
        }
    }
}

So this makes me wonder whether my data passing setup is actually correct. In my pointer cluster, I have define these to be a unsigned integer pointer. To think about it, getting a NULL pointer at the end make sense, because I did not initialize the input variable, and 0 = NULL. I have looked up to most examples online, and found that the size must be known previously when passing the array into dll. 

As my dll does calculate the actual size before performing calloc, I couldn't initialize a fixed-size array into the cluster and then pass it inside. How do I initialize the cluster with multiple unknown size, array variable pointer?

And also, why this setup works on Windows Host PC, but not RT target?

P/S: I'm not sure whether I should add the #pragma pack(1) option, as I'm currently running 64 bit Windows, on a 32 Bit LabView, but operating on a 64 bit Linux RT (The LabView RT is still 32 bit). Regardless, both method crash my RT target.

[rolfk · Accepted solution]

LabVIEW for Linux is 64-bit only since about LabVIEW 2016, and LabVIEW for NI Linux RT always has been 64-bit only, so your pointers need to be 64 bit integers in that cluster.

#pragma pack(1) is only needed for Windows 32-bit. All other LabVIEW platforms currently still available use native alignment which is 8 byte. But your specific cluster has no alignment issues at all as it does not use anything else than 32-bit types when compiled for 32-bit. When compiled for 64-bit the pointers will be aligned on a 64-bit boundary adding an extra 32-bit filler just before the pointers, but since LabVIEW 64-bit on all platforms uses the standard 8-byte alignment too, that is used by default by your C compiler, there is no alignment mismatch here.

The OS doesn't matter in this respect. The LabVIEW bitness does however very much. And in order to make this work for 32-bit LabVIEW on Windows those pointer integers need to be 32-bit integers. This means you need to have two types of clusters and use the correct one (and two seperate Call Library Nodes in the two separate cases of a Conditional Compile structure set to differentiate on bitness of the current LabVIEW platform. And you would need to do that on every Call Library Node that accepts this cluster!!! 🙄

But there is another option if you do not need to access the elements in that cluster from the LabVIEW diagram (often). Just configure that whole thing as a pointer sized integer instead in the Call Library Node, wire it as 64-bit integer everywhere in your diagram and do an explicit malloc() for the necessary structure in your C function, then provide another function that does the necessary deallocate (already needed anyways as you need to free the internal pointers at some point and in there also free() the structure pointer itself.

If you need to access some of the data in that "pointer" from the LabVIEW diagram at anytime you could do that by calling the LabVIEW manager function MoveBlock() with another Call Library Node and the correct offset added to that pointer value but I would refrain from that as much as possible.

It's all fairly brittle anyway as your memory pointers are not going to be managed at all by LabVIEW and there is a lot of opportunity for the user of your library to mess up things here, but to protect that you could wrap everything into a LabVIEW class that only offers a public API and doesn't let the user mess with the pointer data.

Actually if you never intend to access the pointers from the LabVIEW diagram you could live with making these pointers inside the cluster simply 64-bit integers. In the 32-bit variant of your DLL the two pointers will be stored in the first 64-bit integer and the second will be unused, but that is not an issue. Currently the first pointer when compiled for 64-bit is already stored only halfway in the second 32-bit integer as the compiler assumes that there is a 64-bit alignment in the structure but LabVIEW seeing only two 32-bit integers in there will not add any such filler integer and the second pointer allocation tries to access memory beyond the memory allocated for the LabVIEW cluster and therefore usually fails. But it is brittle anyways and you should not allow a user of your library to ever look into that cluster anyways.

When you make those two integers 64-bit what LabVIEW will pass to the DLL in 32-bit mode looks like this:

typedef struct Pointers_Double
{
    uint32_t Nx; // Number of input
    uint32_t Ny; // Number of output
    uint32_t I; // FIR Filter order per channel
    uint32_t I_ori; // FIR Filter order per channel, without padding (original input).
    uint registry_size; // Double = 4
 /* uint32_t alignment_filler; only present on non Windows LabVIEW 32-bit */
    uint64_t pointer_H; // Multichannel FIR reshaped 1D array ( order: I -> Nx (input) -> Ny(output) )
    uint64_t pointer_tx_tilde; // Multichannel FIR input signal (order: I -> Nx )
} Pointers_Double

The C compiled code will store both pointers into the first 64-bit pointer integer and leave the second pointer value alone

In 64-bit mode LabVIEW will pass this structure to the shared library:

typedef struct Pointers_Double
{
    uint32_t Nx; // Number of input
    uint32_t Ny; // Number of output
    uint32_t I; // FIR Filter order per channel
    uint32_t I_ori; // FIR Filter order per channel, without padding (original input).
    uint32_t registry_size; // Double = 4
    uint32_t alignment_filler;
    uint64_t pointer_H; // Multichannel FIR reshaped 1D array ( order: I -> Nx (input) -> Ny(output) )
    uint64_t pointer_tx_tilde; // Multichannel FIR input signal (order: I -> Nx )
} Pointers_Double

[Cklai]

Interesting. I'll try to edit the LabView code and test it out.

For your proposal in replacing the double type arrays to 64 bit integer pointers, you are referring to changing the datatype of the C struct code as well? But double and int64 pointers have the same size, isn't it?

For the 32 bit dll cluster you proposed, wouldn't it cause a misalignment here if the registry_size is set to uint, which is 1byte and will push the rest of the pointers to the next alignment? I'm not an expert in this field ...

EDIT: Ah I noticed that you didn't have the * declaration in the struct, which means that you DO intend to store this as memory address instead of an array. 

EDIT2: Right, so it is indeed for illustrative purpose and I don't need to edit the C code.

[rolfk · Accepted solution]

No need to change the C code. You just need to make sure that LabVIEW passes a large enough memory buffer to the DLL on every possible platform. Your current code passes a to small buffer to the DLL on every single LabVIEW platform except LabVIEW 32-bit for Windows (and NI Pharlap ETS realtime as that uses the same alignment by default).

You just need to be aware that on any non--64-bit platform the values you will see in the cluster for the pointers is simply garbage and can't be used at all from the LabVIEW diagram as the memory is aligned differently by the C compiler than what LabVIEW thinks it should be. As long as you do not try to access those pointer integers from the LabVIEW diagram in any way there is no problem here.

[Cklai]

Well, I have tried using 64 bit pointer element in its cluster and it work for BOTH Windows 32 and Linux!

The next thing I would wanted to try on is on PharLap ETS device, however, the same issue (error 1097) pops out again (works on pc with 64 bit pointer), even if I put 64 bit cluster... note that I have added #pragma pack(1) for this as its a 32 bit OS, and also updated the struct so it can store more objects:

#pragma pack(1)
typedef struct Pointers_Double
{
    unsigned int Tg;           
    unsigned int Tw;            
    unsigned int Tg_ori;        
    unsigned int Tw_ori;        
    unsigned int Ne_ori;       
    unsigned int Nu_ori;        
    unsigned int registry_size; 
    unsigned int Nu;            
    unsigned int Ne;            
    unsigned int Nx;            

    double *W;       
    double *G_Hat;   
    double *x_tilde; 
    double *e_n_padded;     
    double *u_n_padded;   
} McFxLMS_Pointers_Double;
#pragma pack()

What will be the issue this time?

[rolfk]

Well it's not your problem for the 1097 error, but remove the pack(1) altogether. LabVIEW only packs data on Win32 32-bit anymore, all other platforms use the default alignment for the platform which currently is 8 byte on every platform. Since your structure only uses 32-bit integers and 32-bit pointers on 32-bit LabVIEW, there is no alignment issue at all and you can just forget about the pack. If you really really want to be sure that your structure gets right for every platform even in the case that you end up adding non 32-bit elements to it, you should instead make the declaration for your cluster as follows:

#include "lv_prolog.h"
typedef struct Pointers_Double
{
    unsigned int Tg;           
    unsigned int Tw;            
    unsigned int Tg_ori;        
    unsigned int Tw_ori;        
    unsigned int Ne_ori;       
    unsigned int Nu_ori;        
    unsigned int registry_size; 
    unsigned int Nu;            
    unsigned int Ne;            
    unsigned int Nx;            

    double *W;       
    double *G_Hat;   
    double *x_tilde; 
    double *e_n_padded;     
    double *u_n_padded;   
} McFxLMS_Pointers_Double;
#include "lv_epilog.h"

Those two includes can be found in your LabVIEW/cintools directory.

You are not trying to access the 5 64-bit elements in a LabVIEW diagram anywhere are you? Because as I explained they do not contain valid pointers in the LabVIEW diagram but garbage that can not be interpreted as pointer without playing your own conditional compile C precompiler on the diagram yourself.

As to what could be the issue then? Calling Convention matches?

LabVIEW 64-bit doesn't really have a calling convention selection anymore since the only normally used calling convention for DLLS is __fastcall. LabVIEW for Windows 32-bit (and Pharlap ETS) however know two possibly modes: __cdecl and __stdcall.

Your Call Library Node needs to match the setting that you used to compile your DLL or the stack gets corrupted which LabVIEW detects and corrects but generates an according error code then. And continuing after any 1097 is normally not safe, even though it often seems to work. A stack corruption is a serious crime can cause all kinds of memory corruption problems that your software would then just continue to use and possibly generate many other more serious problems after that.