`datafusion-cli` requires more than 8GB to build

[alamb]

Describe the bug

@pmcgleenon reports on #17721 (comment)

The instance type c6a.xlarge only has 8GB RAM (the clickbench dataset is 15GB) and there are a number of issues with it, including

datafusion compilation fails on this instance. We can workaround this by using brew install

To Reproduce

You can reproduce this locally on docker using this command:

cd .devcontainer
# create a docker container
docker build -t datafusion-build .
cd ..
docker run -m 4G -v `pwd`:/datafusion  -it datafusion-build   /bin/bash

Then try to run

cargo build --release -p datafusion-cli

This will fail with an error like this:

Caused by:
  process didn't exit successfully: `/usr/local/rustup/toolchains/1.90.0-aarch64-unknown-linux-gnu/bin/rustc --crate-name datafusion_functions_aggregate --edition=2021 datafusion/functions-aggregate/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=160 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 '--warn=clippy::used_underscore_binding' --deny=unused_qualifications '--warn=clippy::unnecessary_lazy_evaluations' '--warn=clippy::uninlined_format_args' --warn=unexpected_cfgs '--warn=clippy::or_fun_call' '--warn=clippy::large_futures' '--warn=clippy::inefficient_to_string' --check-cfg 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))' --check-cfg 'cfg(tarpaulin)' --check-cfg 'cfg(tarpaulin_include)' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=727ec34a83a3b23a -C extra-filename=-41481de704778bc6 --out-dir /datafusion/target/release/deps -C strip=symbols -L dependency=/datafusion/target/release/deps --extern ahash=/datafusion/target/release/deps/libahash-16bfcefaf019ce0a.rmeta --extern arrow=/datafusion/target/release/deps/libarrow-7a0c4e030b71b835.rmeta --extern datafusion_common=/datafusion/target/release/deps/libdatafusion_common-ca5a073002b6fd0e.rmeta --extern datafusion_doc=/datafusion/target/release/deps/libdatafusion_doc-8ab038afb63bc362.rmeta --extern datafusion_execution=/datafusion/target/release/deps/libdatafusion_execution-93c2cc956c650df0.rmeta --extern datafusion_expr=/datafusion/target/release/deps/libdatafusion_expr-87611f1ca5ce8f65.rmeta --extern datafusion_functions_aggregate_common=/datafusion/target/release/deps/libdatafusion_functions_aggregate_common-05b65785638f4212.rmeta --extern datafusion_macros=/datafusion/target/release/deps/libdatafusion_macros-0bed692e75dab347.so --extern datafusion_physical_expr=/datafusion/target/release/deps/libdatafusion_physical_expr-93e2a25460efd9b7.rmeta --extern datafusion_physical_expr_common=/datafusion/target/release/deps/libdatafusion_physical_expr_common-79bb18eb023feafb.rmeta --extern half=/datafusion/target/release/deps/libhalf-feba467d2af9eeb5.rmeta --extern log=/datafusion/target/release/deps/liblog-d456af582e58c114.rmeta --extern paste=/datafusion/target/release/deps/libpaste-5dba9edad0280e64.so -L native=/datafusion/target/release/build/zstd-sys-85e0b1f1ab4f39ce/out -L native=/datafusion/target/release/build/lzma-sys-d68250324f9032a4/out -L native=/datafusion/target/release/build/aws-lc-sys-441c9757ee7585c9/out -L native=/datafusion/target/release/build/ring-dcbccc510e078148/out -L native=/datafusion/target/release/build/psm-2382bc49ac467ad4/out` (signal: 9, SIGKILL: kill)

Note the signal: 9, SIGKILL: kill

Expected behavior

No response

Additional context

No response

[alamb]

I tried limiting concurrency to work around the issue but this still fails with a SIGKILL:

cargo build -j 1 --release -p datafusion-cli

I was able to build it using nonlto mode:

cargo build  --profile=release-nonlto -p datafusion-cli
...

Finished `release-nonlto` profile [optimized] target(s) in 2m 26s

[alamb]

BTW I am not sure this is a bug we need to solve necessairly -- using a pre-built binary might be a better option