32 bit DLL crashes with "General Protection Fault", 64 bit DLL works fine

[flocki1782]

Hi community,

I've two 3rd party security dll's. One is 64 bit, one 32 bit.

Both dll's are loaded the same way to project (one project is 32 bit, the other one 64 bit):

HINSTANCE LoadSeedKeyDLL_20Byte(void)
{
    UINT oldErrorMode;
    BOOL ok=TRUE;
    HINSTANCE hDll=0;

    oldErrorMode=SetErrorMode(SEM_NOOPENFILEERRORBOX);
    hDll=LoadLibrary("SeedKey.DLL");
    SetErrorMode(oldErrorMode);
    if (hDll != NULL)
	{

		ok=ok&&((GenerateKeyExOpt__ = (GENERATE_KEY_EX_OPT) GetProcAddress(hDll, "GenerateKeyExOpt"))!=NULL);
		if (!ok)
	      return 0;	    
	}
    return hDll;   
}

Loading works and they have valid (32/64 bit addresses):

64 bit project everything works fine.

Building on the 32 bit project I get a "General Protection Fault" when leaving the function the GenerateKeyExOpt-Function() is called.

int SecurityAccess_20Byte(void)
{
	unsigned char Key[35]= {0};
	unsigned int KeyLength=0,resultKey=NOK;
	unsigned char *Seed_;
	
	Seed_ = GetSeed_20Byte();

	resultKey = GenerateKeyExOpt__(Seed_20,20,0x11,NULL,str_key_initialize,Key, 35,&KeyLength)

	return resultKey;
}

Any clue what can be the reason?

Thanks in advance!

[flocki1782 · Accepted solution]

Found the solution myself:

somehow substituting the __stdcall function call with a __cdecl definition brought the solution.

The stack gets cleaned this way.

Further details would be appreciated.

[rolfk · Accepted solution]

The reason is simple.

Your GENERATE_KEY_EX_OPT define wasn't correct to match what the underlying library really was compiled with. Since you omitted that part of the code we had to assume that that define came from the header file that came with the DLL and then it is very unlikely that it would not match (but not entirely unheard off). We can only help debug what we can see. My magic crystal ball still hasn't returned from repair. 😁

In 64-bit there is no stdcalll or cdecl calling convention. 64-bit Windows uses pretty much exclusively only fastcall (and thiscall for C++ object class methods), which is a mixture of register and stack based parameter passing with the same parameter order as cdecl to allow for variadic functions (which stdcall can't support, hence those rare exceptions in the Windows API that are actually exported as CDECL since they use variadic parameters).

Your 64-bit compiler has been intensively trained to simply ignore stdcall and cdecl and always use fastcall instead to avoid porting nightmares when trying to compile old code with it. 😁