text-2.* -simdutf8 is broken with statically linked GHC

[mpickering]

See this GHC ticket - https://gitlab.haskell.org/ghc/ghc/-/issues/21787

If you have a statically linked GHC and use text build with -simdutf8 in a template-haskell splice then you get the error:

ghc: /home/zubin/.ghcup/ghc/9.4.1/lib/ghc-9.4.0.20220623/lib/x86_64-linux-ghc-9.4.0.20220623/text-2.0/libHStext-2.0.a: unknown symbol `__popcountdi2'

This happens with at least GHC-9.4 and GHC-9.2 so I imagine every GHC release is affected by this change (including ones we can't patch). Therefore it seems the most prudent to remove the usage of the code which generates this instruction which GHC can't understand. Otherwise anyone enforcing text-2.0 will create a package which doesn't work on static builds.

[Bodigrim]

__builtin_popcount{,l,ll} is widely used in cbits/measure.off. If someone can replace it with something else which GHC can readily compile without compromissing performance, I'd happily accept such patch. Otherwise looks to me as a bug in GHC static build?..

[mpickering]

@Bodigrim of course this is a GHC bug but as it stands text-2.0 should probably not be depended on

• GHC's C++ linking story is not mature enough yet to deal with the C++ code. So we have to distribute text-2.0 with -simdutf.
• The C fallback code does not work with statically linked GHC (including 8.10.7 etc)

Therefore there are two prongs of attack necessary.

1. On the GHC part, we need to fix GHC so it can understand the popcount symbol (that is tracked in #21787)
2. On the text part, the problematic instructions should to be removed until the whole GHC support window can understand them.

[wz1000]

Including extra-libraries: gcc_s in text while linking resolves this problem. The following patch allows text-2.0 to be linked in during TH evaluation on GHC 9.2.3:

--- a/text.cabal
+++ b/text.cabal
@@ -194,7 +194,7 @@ library
       ghc-options: -fcheck-prim-bounds
 
   -- https://gitlab.haskell.org/ghc/ghc/-/issues/19900
-  if os(windows) && impl(ghc < 9.3)
+  if impl(ghc < 9.3)
     extra-libraries: gcc_s

I'm not sure if this is a desired change since it apparently caused https://gitlab.haskell.org/ghc/ghc/-/issues/21196 ?
But the problematic extra-libraries section still exists on windows on text-2.0? I'm not sure what the resolution of #21196 was.

[Bodigrim]

On the text part, the problematic instructions should to be removed until the whole GHC support window can understand them.

I don't quite follow what does it have to do with GHC support window. The issue does not seem to affect GHC bootstrapping.

@wz1000 I have no experience with GHC static builds. I'm quite happy to add whatever extra-libraries needed.

[bgamari]

Allow me to recap the problem here since there are quite a few things at play:

C compilers can emit references to code defined by the compiler's runtime library in the objects they produce. This library is known as libgcc in the case of GCC and can be either compiler-rt or libgcc in the case of Clang; thankfully the latter two are largely ABI-compatible and I will refer to both as libgcc below. In this ticket the problematic reference is to the __popcountdi2 function, which is a shim that the compiler uses to implement the popcountll builtin on machines which lack the popcnt instruction (N.B. you will find that compiling int foo(unsigned long long x) { return __builtin_popcountll(x); } with -msse4 produces an object without any mention of this symbol).

Usually these symbol references are resolved during the final link (e.g. when producing an executable or shared library) since the C compiler will add its runtime to the final link dependencies. However, GHC's runtime loader (which we must use when GHC is built as a static executable due to limitations of both the glibc and musl libc implementations) can't rely on this since it loads objects which haven't yet been through the final link. Consequently, when GHC's runtime loader attempts to load an object containing such a reference, it must know where to find the symbol's definition (lest we see issues like GHC #19900 and #21787). There are a few ways by which this could be done:

1. load the compiler runtime via the dynamic loader (e.g. dlopen). This is what adding extra-libraries: gcc_s accomplished in ffaf10f (_s here standing for "shared", as in "shared object"). However, this is problematic since libgcc_s isn't always available (this is what led to GHC #21196).
2. load the compiler runtime via its static archive. This is problematic because libgcc.a isn't always available. Moreover, even when it is available it often isn't easy to find (e.g. gcc and clang both provide --print-libgcc-file-name for this but Apple ships a toolchain which reports a non-existent path).
3. resolve these symbols when the compiler is built by ensuring that the GHC RTS links against all symbols exported by libgcc and add them to the runtime linker's initial symbol table.

As far as I can tell, (3) is the only robust option. Currently GHC's internal linker implements this partially, having knowledge of a select few commonly-used symbols provided by libgcc in its initial symbol table. However, we have not attempted full coverage since the symbols exposed by libgcc depend upon the target platform and therefore such a list is rather tricky to maintain. However, we will need to extend this to cover all symbols to eliminate issues like the present one.

Since libgcc's contents is so platform dependent, giving GHC's RTS full coverage of libgcc's symbols will demand a different approach from our current manually-curated list. Rather I think we will need to probe libgcc for its symbols when the GHC is built. However, even doing this is fairly platform dependent, which is why I have been so reluctant to attempt tackling it. I will need to do so for GHC 9.6 since the status quo clearly isn't tenable.

Strictly speaking, even the above isn't quite a full solution since clang ships not one but a whole spectrum of runtime library implementations, all supporting different santizers (e.g. UBSAN, TSAN, MSAN, ASAN, etc.). These are different implementations are mutually exclusive: if you attempt to load one object built with UBSAN and another built with ASAN, things will break. Nevertheless, if we put aside the problem of sanitizers then this approach is at least a considerable improvement over the status quo.

However, if we implement the above plan for GHC 9.6 this still leaves every other previous GHC release (9.4 included) as unusable with text-2 when statically-linked. This is problematic since these releases will continue to be used for the foreseeable future and, e.g., haskell-language-server has been shipping statically-linked compilers since they are easy to package. This is the reason why @wz1000 has been pushing to avoid using __builtin_popcountll. The idea here is that we can revert to a popcount implementation provided by text for GHC versions which don't know of __builtin_popcountll and allow usage of __builtin_popcountll for GHC versions where it can be safely used. Eventually when all GHCs in common use support __builtin_popcountll we can eliminate the text-provided version.

Does this sound sensible, @Bodigrim ?

[Bodigrim]

@bgamari thanks for detailed account, much appreciated.

As discussed in #453, the first thing I need is a reproducer (expressed as a CI job), which cannot be fixed with extra-libraries: gcc_s. I guess this can be achieved by a Windows build with empty path, so that libgcc_s cannot be found. Cf. https://github.com/haskell/bytestring/blob/22b36125ac52605e807b7b96ef31e8f087248f17/.github/workflows/ci.yml#L93-L99. Would it be enough? Or do we need an Alpine job?..

Once there is a reproducer, we can discuss possible solutions. As mentioned above, we need a solution which does not (measurably) compromise performance for machines with POPCNT instruction. This can probably be achieved by checking for SSE 4.2 (there is already infrastructure in place).

[bgamari]

As discussed in #453, the first thing I need is a reproducer (expressed as a CI job), which cannot be fixed with extra-libraries: gcc_s.

Absolutely, I agree that having a test is a good starting point. Indeed the reproducer would be loading text in, e.g., ghci using a statically-linked compiler. It needn't be in an Alpine environment; as long as you using a fully-static compiler binary distribution like https://downloads.haskell.org/ghc/9.4.1-alpha3/ghc-9.4.0.20220623-x86_64-alpine3_12-linux-static.tar.xz would be sufficient.
@wz1000 do you think you could add such a CI job to your MR?

Incidentally, a Windows job with a minimal PATH would also be helpful since it will ensure that we don't reintroduce GHC #21196.

As mentioned above, we need a solution which does not (measurably) compromise performance for machines with POPCNT instruction. This can probably be achieved by checking for SSE 4.2 (there is already infrastructure in place).

Sure. I agree that enabling __builtin_popcount path is quite sensible when SSE 4.2 or similar is available. @wz1000 will amend his MR to do this.

[wz1000]

@wz1000 do you think you could add such a CI job to your MR?

Yes, I can do that if using a prereleased compiler is OK with Bodigrim. The reason the CI job I added used windows is because that is the only configuration of GHC we ship that is statically linked and can trigger this bug.

Sure. I agree that enabling __builtin_popcount path is quite sensible when SSE 4.2 or similar is available. @wz1000 will amend his MR to do this.

I'm sorry, I don't quite see how to do this. The problem isn't the availability of SSE 4.2, its the fact that the RTS linker does not know about __popcountdi2 and cannot link any code that references it. Using the __builtin_popcount path will trigger the GHC bug.

The only way I can see to make use of the popcount instruction would be to use inline assembly.