Skip to content

feat: switch default allocator from mimalloc to jemalloc#3684

Merged
jqnatividad merged 9 commits into
masterfrom
bump-polars-to-use-our-preferred-allocator
Apr 6, 2026
Merged

feat: switch default allocator from mimalloc to jemalloc#3684
jqnatividad merged 9 commits into
masterfrom
bump-polars-to-use-our-preferred-allocator

Conversation

@jqnatividad

Copy link
Copy Markdown
Collaborator

Summary

  • Restores tikv-jemallocator (jemalloc) as a feature-flagged allocator and makes it the new default, replacing mimalloc. This aligns qsv's allocator with Polars' preferred allocator.
  • Both mimalloc and jemallocator features remain available, with a compile_error! guard preventing both from being enabled simultaneously.
  • Bumps Polars rev to bb93ba8 (py-1.39.3).
  • Uses exact match for MALLOC_CONF env var instead of prefix match.

Test plan

  • cargo build --bin qsv -F all_features compiles successfully with jemalloc default
  • cargo build --no-default-features --bin qsv -F all_features,mimalloc,jemallocator triggers compile_error!
  • cargo clippy -F all_features passes clean
  • qsv --version shows jemalloc in allocator field

🤖 Generated with Claude Code

jqnatividad and others added 3 commits April 6, 2026 06:45
Switch default allocator from mimalloc to tikv_jemallocator (jemalloc),
aligning with Polars' preferred allocator. Both mimalloc and jemalloc
remain available via feature flags, with jemalloc as the new default.

Also bumps Polars rev to bb93ba8.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…OC_CONF match

- Add compile_error! in all binary entry points when both `mimalloc` and
  `jemallocator` features are enabled simultaneously
- Use exact match for MALLOC_CONF env var instead of starts_with

Addresses review #1373.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jqnatividad jqnatividad requested a review from Copilot April 6, 2026 11:00
@codacy-production

codacy-production Bot commented Apr 6, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric Results
Complexity 0

View in Codacy

TIP This summary will be updated as you push new changes. Give us feedback

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Switches qsv’s default memory allocator from mimalloc to jemalloc (via tikv-jemallocator) to align with Polars’ preferred allocator, updates Polars rev, and adjusts related docs/env-var handling.

Changes:

  • Makes jemallocator the default Cargo feature and wires tikv_jemallocator::Jemalloc as the global allocator.
  • Updates --version / --envlist reporting paths for jemalloc and tightens MALLOC_CONF matching.
  • Bumps Polars git rev and updates documentation/tests accordingly.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
tests/test_100/lesson_4.rs Fixes expected COUNT(*) output for lesson 4 task 6.
src/util.rs Updates allocator identification in --version and env-var listing logic for jemalloc.
src/mainlite.rs Adds jemalloc global allocator and mutual-exclusion guard.
src/maindp.rs Adds jemalloc global allocator and mutual-exclusion guard.
src/main.rs Adds jemalloc global allocator and mutual-exclusion guard.
README.md Updates allocator docs and Polars rev badge.
docs/PERFORMANCE.md Updates allocator documentation and build examples for jemalloc default.
docs/FEATURES.md Documents new default jemallocator feature.
docs/ENVIRONMENT_VARIABLES.md Documents jemalloc env vars and updates --envlist description.
docs/contributor/PROJECT_TECHNICAL_OVERVIEW.md Updates contributor overview to mention jemalloc default.
Cargo.toml Adds tikv-jemallocator, sets default feature to jemallocator, bumps Polars rev.
Cargo.lock Updates Polars git source and adds tikv-jemallocator packages (plus transitive updates).

Comment thread docs/ENVIRONMENT_VARIABLES.md Outdated
Comment thread README.md Outdated
Comment thread src/main.rs Outdated
Comment thread src/main.rs Outdated
Comment thread src/mainlite.rs Outdated
Comment thread src/mainlite.rs Outdated
Comment thread src/maindp.rs Outdated
Comment thread src/maindp.rs Outdated
Comment thread src/util.rs Outdated
jqnatividad and others added 4 commits April 6, 2026 07:06
jemalloc's autoconf-based build system requires a Unix-like environment
and fails on Windows MSVC with "C compiler cannot create executables".
All Windows workflows now use --no-default-features with mimalloc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add mimalloc to addl-qsvlite-features and addl-qsvdp-features for all
  Windows matrix entries (MSVC and GNU)
- Add mimalloc to qsvmcp build steps via addl-qsvlite-features
- Add mimalloc to Windows GNU addl-build-args (was missing entirely)
- Add addl-qsvlite-features matrix var to test-publish-nightly.yml

Addresses review #1376: Windows GNU targets had no allocator, and
secondary binaries (qsvlite, qsvdp, qsvmcp) silently fell back to
the system allocator on all Windows targets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align qsvlite build step with the append pattern used in all other
workflows: --features=lite,self_update,${{ addl-qsvlite-features }}

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cfg guards

- Switch all cfg guards from feature = "tikv-jemallocator" to
  feature = "jemallocator" (the public crate feature name)
- Make allocator cfg guards mutually exclusive with not() clauses
  so only compile_error! fires when both features are enabled
- Clarify env vars docs that allocator-specific vars are build-dependent
- Fix Polars badge markdown line break in README.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align the two remaining #[cfg(feature = "mimalloc")] guards in util.rs
with the not() pattern used everywhere else in the codebase.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

Comment thread .github/workflows/test-publish-nightly.yml Outdated
Comment thread .github/workflows/publish.yml Outdated
Comment thread .github/workflows/publish-portable.yml Outdated
Comment thread .github/workflows/publish-nightly.yml Outdated
Comment thread src/util.rs
Comment thread docs/PERFORMANCE.md Outdated
- Use conditional expressions in qsvmcp/qsvlite build steps to avoid
  trailing commas when addl-qsvlite-features is empty
- Move MALLOC_CONF from "always included" to jemalloc-specific in env
  vars docs to match the actual code behavior
- Narrow jemalloc unsupported target note to just Windows (not arm)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jqnatividad jqnatividad merged commit 3144267 into master Apr 6, 2026
16 of 17 checks passed
@jqnatividad jqnatividad deleted the bump-polars-to-use-our-preferred-allocator branch April 6, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants