feat: switch default allocator from mimalloc to jemalloc#3684
Merged
Conversation
Switch default allocator from mimalloc to tikv_jemallocator (jemalloc), aligning with Polars' preferred allocator. Both mimalloc and jemalloc remain available via feature flags, with jemalloc as the new default. Also bumps Polars rev to bb93ba8. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…OC_CONF match - Add compile_error! in all binary entry points when both `mimalloc` and `jemallocator` features are enabled simultaneously - Use exact match for MALLOC_CONF env var instead of starts_with Addresses review #1373. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
TIP This summary will be updated as you push new changes. Give us feedback
Contributor
There was a problem hiding this comment.
Pull request overview
Switches qsv’s default memory allocator from mimalloc to jemalloc (via tikv-jemallocator) to align with Polars’ preferred allocator, updates Polars rev, and adjusts related docs/env-var handling.
Changes:
- Makes
jemallocatorthe default Cargo feature and wirestikv_jemallocator::Jemallocas the global allocator. - Updates
--version/--envlistreporting paths for jemalloc and tightensMALLOC_CONFmatching. - Bumps Polars git rev and updates documentation/tests accordingly.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_100/lesson_4.rs | Fixes expected COUNT(*) output for lesson 4 task 6. |
| src/util.rs | Updates allocator identification in --version and env-var listing logic for jemalloc. |
| src/mainlite.rs | Adds jemalloc global allocator and mutual-exclusion guard. |
| src/maindp.rs | Adds jemalloc global allocator and mutual-exclusion guard. |
| src/main.rs | Adds jemalloc global allocator and mutual-exclusion guard. |
| README.md | Updates allocator docs and Polars rev badge. |
| docs/PERFORMANCE.md | Updates allocator documentation and build examples for jemalloc default. |
| docs/FEATURES.md | Documents new default jemallocator feature. |
| docs/ENVIRONMENT_VARIABLES.md | Documents jemalloc env vars and updates --envlist description. |
| docs/contributor/PROJECT_TECHNICAL_OVERVIEW.md | Updates contributor overview to mention jemalloc default. |
| Cargo.toml | Adds tikv-jemallocator, sets default feature to jemallocator, bumps Polars rev. |
| Cargo.lock | Updates Polars git source and adds tikv-jemallocator packages (plus transitive updates). |
jemalloc's autoconf-based build system requires a Unix-like environment and fails on Windows MSVC with "C compiler cannot create executables". All Windows workflows now use --no-default-features with mimalloc. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add mimalloc to addl-qsvlite-features and addl-qsvdp-features for all Windows matrix entries (MSVC and GNU) - Add mimalloc to qsvmcp build steps via addl-qsvlite-features - Add mimalloc to Windows GNU addl-build-args (was missing entirely) - Add addl-qsvlite-features matrix var to test-publish-nightly.yml Addresses review #1376: Windows GNU targets had no allocator, and secondary binaries (qsvlite, qsvdp, qsvmcp) silently fell back to the system allocator on all Windows targets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align qsvlite build step with the append pattern used in all other
workflows: --features=lite,self_update,${{ addl-qsvlite-features }}
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cfg guards - Switch all cfg guards from feature = "tikv-jemallocator" to feature = "jemallocator" (the public crate feature name) - Make allocator cfg guards mutually exclusive with not() clauses so only compile_error! fires when both features are enabled - Clarify env vars docs that allocator-specific vars are build-dependent - Fix Polars badge markdown line break in README.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align the two remaining #[cfg(feature = "mimalloc")] guards in util.rs with the not() pattern used everywhere else in the codebase. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use conditional expressions in qsvmcp/qsvlite build steps to avoid trailing commas when addl-qsvlite-features is empty - Move MALLOC_CONF from "always included" to jemalloc-specific in env vars docs to match the actual code behavior - Narrow jemalloc unsupported target note to just Windows (not arm) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tikv-jemallocator(jemalloc) as a feature-flagged allocator and makes it the new default, replacing mimalloc. This aligns qsv's allocator with Polars' preferred allocator.mimallocandjemallocatorfeatures remain available, with acompile_error!guard preventing both from being enabled simultaneously.bb93ba8(py-1.39.3).MALLOC_CONFenv var instead of prefix match.Test plan
cargo build --bin qsv -F all_featurescompiles successfully with jemalloc defaultcargo build --no-default-features --bin qsv -F all_features,mimalloc,jemallocatortriggerscompile_error!cargo clippy -F all_featurespasses cleanqsv --versionshowsjemallocin allocator field🤖 Generated with Claude Code