external symbol g_allocator in header file breaks DelayLoadDLLs on windows

[RuurdBeerstra]

Describe the bug

I use the AWS SDK dll's on Windows in a binary that can also run outside the cloud on arbitrary windows systems.
Support for AWS was added a few years ago and we used the DelayLoadDLLs directive when building our binaries so the DLL's are not required in a non-AWS environment (and not used/loaded even in AWS until it actually uses AWS functionality).

After an upgrade to the latest AWS SDK this was broken: The linker told me I could not use the DelayLoadDLLs for the AWS dll's because of the "g_allocator" symbol.

4>LINK : fatal error LNK1194: cannot delay-load 'aws-crt-cpp.dll' due to import of data symbol '"__declspec(dllimport) struct aws_allocator * Aws::Crt::g_allocator" (_imp?g_allocator@Crt@Aws@@3PEAUaws_allocator@@ea)'; link without /DELAYLOAD:aws-crt-cpp.dll

Digging through the AWS source I found the culprit: aws/crt/StlAllocator.h has the line:
extern AWS_CRT_CPP_API Allocator *g_allocator;

Which implies that any component that includes AWS header files ends up with an extern DATA reference to the g_allocator.
The DelayLoadDLLs can handle references to functions, but because the value of g_allocator must be resolved at start-time of the binary, this one single pointer caused our product to have a very undesirable hard runtime dependency with AWS.

I did a crude hack to get around this. I replaced the "extern" line with this:

extern AWS_CRT_CPP_API Allocator *Get_g_allocator();
#define g_allocator Get_g_allocator()

So everywhere the code uses "g_allocator" it now calls a getter function for the current value.
In crt/aws-crt-cpp/source/Api.cpp (where g_allocator is defined) I added (after the #includes):

/* Undo hack in StlAllocator.h */
#ifdef g_allocator
#undef g_allocator
#else
#error StlAllocator must define g_allocator as a call to Get_g_allocator()
#endif

The #error is just to make sure my includes and defines and up as intended.
The getter function is trivial:
AWS_CRT_CPP_API Allocator *Get_g_allocator();
Allocator *Get_g_allocator() { return g_allocator; }

Building with these modified header files yields a working configuration that no longer suffers from the hard dependency.

The hack is, of course, ugly as sin.
But because there are some 53 uses of "g_allocator" in a some 20 different files, I did it the quick & dirty way.

Expected Behavior

That we could have a single binary that optionally uses AWS dll's (like was possible in an older version of the AWS SDK).

Current Behavior

Linking a Windows binary that uses DelayLoadDLLs on aws-sdk-core.dll yields this error:

4>LINK : fatal error LNK1194: cannot delay-load 'aws-crt-cpp.dll' due to import of data symbol '"__declspec(dllimport) struct aws_allocator * Aws::Crt::g_allocator" (_imp?g_allocator@Crt@Aws@@3PEAUaws_allocator@@ea)'; link without /DELAYLOAD:aws-crt-cpp.dll

Reproduction Steps

The code is not the problem: Try linking a binary that has a DelayLoadDLLs in the vcxproj files likes this:
aws-cpp-sdk-core.dll

And get the error about the DelayLoad not being possible because of the g_allocator symbol.

Possible Solution

Change all occurrences of g_allocator in the header files to a function call that returns the current value.

Additional Information/Context

The (crude) patch provides a workaround for us, so it is currently not an urgent issue.
But I'd like to see a fix for this in a future version.

AWS CPP SDK version used

git clone from April 25, 2022.

Compiler and Version used

Visual Studio 2019

Operating System and version

Windows 2016

[jmklix]

Thanks for bringing this up to us. I will see if this is a change that we want to make to this sdk.

[graebm]

I'm not able to reproduce the linker error.
I don't have access to Windows Server 2016, only Windows 10. I took the following steps:

• Installed Visual Studio 2019
• Compiled the SDK like so (64 bit, Release), and installed to D:\dev\install-sdk-cpp on my machine:
  • cmake -B build -S . -G "Visual Studio 16 2019" -A x64 -DBUILD_ONLY=s3 -DCMAKE_PREFIX_PATH=D:/dev/install-sdk-cpp -DCMAKE_INSTALL_PREFIX=D:/dev/install-sdk-cpp -DENABLE_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON
  • cmake --build build --target install --config Release --parallel
• Created a Visual Studio 2019 project, set up like so (I'll attach file to ticket so you can inspect it):
  • the main cpp file slept for two seconds before initializing the SDK and making an S3 call.
    • I actually tweaked the SDK before compiling so that it would printf something during DLL load. I did this to sanity-check that it was in fact being loaded after a delay.
    • I did need to add #define USE_IMPORT_EXPORT to the top the single cpp file, or I got other linker errors
  • C/C++ -> General -> Additional Include Directories: D:\dev\install-sdk-cpp\include;%(AdditionalIncludeDirectories)
  • C/C++ -> Linker -> General -> Additional Library Directories: D:\dev\install-sdk-cpp\bin;%(AdditionalLibraryDirectories)
  • C/C++ -> Linker -> Input -> Additional Dependencies: aws-cpp-sdk-core.lib;aws-cpp-sdk-s3.lib;%(AdditionalDependencies)
  • C/C++ -> Linker -> Input -> Delay Loaded Dlls: aws-cpp-sdk-core.dll;aws-cpp-sdk-s3.dll
  • Use a Post-Build event to copy the SDK DLLs to be alongside the EXE:
    • Build Events -> Post-Build Event -> Command Line: xcopy /y /d D:\dev\install-sdk-cpp\bin "$(OutDir)"

I also tried using Visual Studio 2015, and that worked as well.

---

If you tweak this project, does it work on your machine? (You'll need to modify that D:\dev\install-sdk-cpp path)

Removing the Aws::Crt::g_allocator extern is a breaking change, so I'm very hesitant to do that unless it's absolutely necessary.

[graebm]

Here's the test project. I included the compiled EXE and DLLs too
MrDelayLoad3.zip

[RuurdBeerstra]

Hello, Thank you for your message. I am a colleague of Ruurd and I took a look at the code you sent us. I was able to reproduce the issue that we have with your code, with several modifications. First of all, the CPP file should use the AWS allocator. This can be done by creating a dummy object of e.g. the Aws::String class. int main() { Aws::String{ "Trouble" }; // sleep a bit before trying to use the SDK and triggering the DLL load Secondly, there are two missing defines. WIN32 AWS_CRT_CPP_USE_IMPORT_EXPORT Without the latter, it can’t find the g_allocator symbol at all (because it does not know to look for it as an import). With these changes, I get: Severity Code Description Project File Line Suppression State Error LNK1194 cannot delay-load 'aws-crt-cpp.dll' due to import of data symbol '"__declspec(dllimport) struct aws_allocator * Aws::Crt::g_allocator" ***@***.***@aws@@3PEAUaws_allocator@@ea)'; link without /DELAYLOAD:aws-crt-cpp.dll MrDelayLoad3 C:\tmp\MrDelay2\MrDelayLoad3\LINK 1 Note that I also had to include aws-crt-cpp.lib and aws-c-common.lib as additional libraries to load. This was tested on a Windows Server 2019 system with Visual Studio 2019. I hope this helps you further along. Kind regards, Farzin Parham From: Michael Graeb ***@***.***> Sent: Friday, July 8, 2022 6:47 PM To: aws/aws-sdk-cpp ***@***.***> Cc: Ruurd Beerstra ***@***.***>; Author ***@***.***> Subject: Re: [aws/aws-sdk-cpp] external symbol g_allocator in header file breaks DelayLoadDLLs on windows (Issue #1960) Sent by an external sender ------------------------------------ I'm not able to reproduce the linker error. I don't have access to Windows Server 2016, only Windows 10. I took the following steps: * Installed Visual Studio 2019 * Compiled the SDK like so (64 bit, Release), and installed to D:\dev\install-sdk-cpp on my machine: * cmake -B build -S . -G "Visual Studio 16 2019" -A x64 -DBUILD_ONLY=s3 -DCMAKE_PREFIX_PATH=D:/dev/install-sdk-cpp -DCMAKE_INSTALL_PREFIX=D:/dev/install-sdk-cpp -DENABLE_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON * cmake --build build --target install --config Release --parallel * Created a Visual Studio 2019 project, set up like so (I'll attach file to ticket so you can inspect it): * the main cpp file slept for two seconds before initializing the SDK and making an S3 call. * I actually tweaked the SDK before compiling so that it would printf something during DLL load. I did this to sanity-check that it was in fact being loaded after a delay. * I did need to add #define USE_IMPORT_EXPORT<https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html#cmake-build-shared-libs> to the top the single cpp file, or I got other linker errors * C/C++ -> General -> Additional Include Directories: D:\dev\install-sdk-cpp\include;%(AdditionalIncludeDirectories) * C/C++ -> Linker -> General -> Additional Library Directories: D:\dev\install-sdk-cpp\bin;%(AdditionalLibraryDirectories) * C/C++ -> Linker -> Input -> Additional Dependencies: aws-cpp-sdk-core.lib;aws-cpp-sdk-s3.lib;%(AdditionalDependencies) * C/C++ -> Linker -> Input -> Delay Loaded Dlls: aws-cpp-sdk-core.dll;aws-cpp-sdk-s3.dll * Use a Post-Build event to copy the SDK DLLs to be alongside the EXE: * Build Events -> Post-Build Event -> Command Line: xcopy /y /d D:\dev\install-sdk-cpp\bin "$(OutDir)" I also tried using Visual Studio 2015, and that worked as well.

…

________________________________ If you tweak this project, does it work on your machine? (You'll need to modify that D:\dev\install-sdk-cpp path) Removing the Aws::Crt::g_allocator extern is a breaking change, so I'm very hesitant to do that unless it's absolutely necessary. — Reply to this email directly, view it on GitHub<#1960 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKYUE3T4Y6IDU474UZWKZ7LVTBLRZANCNFSM5YX3PH2Q>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[graebm]

I've been able to reproduce this, and I'm working on a fix that won't break the API...

[graebm]

Alright, this is fixed in main at commit b6ba0af

If you want a tagged version, you'll need to wait until the next daily release. The tag will be 1.9.307

Thanks for the help!