Skip to content

convert: support Mistral 3 Large MoE #17730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ngxson merged 11 commits into from
Dec 6, 2025
Merged

convert: support Mistral 3 Large MoE #17730

ngxson merged 11 commits into from
Dec 6, 2025
+188 −83

Conversation

@ngxson
Copy link
Collaborator

@ngxson ngxson commented Dec 3, 2025 •
edited
Loading

WIP, the code is quite ugly for now, but just want to get it to work.

Remember to convert with the --mistral-format argument, as the weight is not yet transformers-compatible

Output F16 weight is 1.35 Terabytes Q8_0 weight is 716GB and I don't have enough hw to test it

Edit: thanks @bartowski1182 for testing it!

Disclaimer: unlike Ministral release, this PR is not affiliated with Mistral Team

NOTE: this PR only covers the conversion to GGUF. the C++ code still missing llama 4 scaling to work, but it will be another PR

@ngxson ngxson changed the title convert: support Mistral 3 Large MoE convert: support Mistral 3 Large MoE (need help for testing) Dec 3, 2025
@bartowski1182
Copy link
Contributor

bartowski1182 commented Dec 3, 2025

So far so good with this, in a couple hours will be able to test generation

@github-actions github-actions bot added the python python script changes label Dec 3, 2025
@bartowski1182
Copy link
Contributor

bartowski1182 commented Dec 3, 2025

seems to work and produce coherent results!

@ngxson ngxson marked this pull request as ready for review December 3, 2025 16:54
@ngxson ngxson requested a review from CISC as a code owner December 3, 2025 16:54
@ngxson ngxson marked this pull request as draft December 3, 2025 16:55
@ngxson
Copy link
Collaborator Author

ngxson commented Dec 3, 2025

This PR still needs to be clean up before it is ready for review 😅

@bartowski1182 bartowski1182 mentioned this pull request Dec 3, 2025
Closed
@ngxson ngxson marked this pull request as ready for review December 3, 2025 18:10
@ngxson ngxson changed the title convert: support Mistral 3 Large MoE (need help for testing) convert: support Mistral 3 Large MoE Dec 3, 2025
@ngxson ngxson mentioned this pull request Dec 3, 2025
Merged
@loci-dev loci-dev mentioned this pull request Dec 3, 2025
Open
CISC
CISC approved these changes Dec 3, 2025
View reviewed changes
convert_hf_to_gguf.py
Comment on lines +9941 to +9942
# remap hparams from Mistral MoE format to DeepseekV2 format
# we do this way to be able to reuse DeepseekV2Model set_gguf_parameters logic
Copy link
Collaborator

@CISC CISC Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat ugly but an acceptable trade-off.

@CISC CISC mentioned this pull request Dec 4, 2025
Merged
@taronaeo taronaeo linked an issue Dec 5, 2025 that may be closed by this pull request
Can't convert mistral 3 large #17705
Closed
@csabakecskemeti
Copy link
Contributor

csabakecskemeti commented Dec 5, 2025

@ngxson Thank you so much for this.
I've also tested the conversion from your branch the convert script succeeded (with --mistral-format) but at inference time (Q8_0) I've received:
llama_model_load: error loading model: missing tensor 'blk.0.attn_k_b.weight'
Tried F16 too it also failed on the same (it should have been failed on not enough memory)

I've tried your Q4_K_M - seems working just fine.
(now downloading @bartowski1182's Q8_0 version to test on that too)

Is there any other setting or change needed for the conversion?
Note I've used Mistral's own BF16 version as the source, which has now disappeared.

@bartowski1182
Copy link
Contributor

bartowski1182 commented Dec 5, 2025

It disappeared?? 👀 I can re-upload if necessary I guess ..

Only difference is using --mistral-format

@csabakecskemeti
Copy link
Contributor

csabakecskemeti commented Dec 5, 2025

Yeah I've used the mistral format. Than I guess I have a corrupted bf16 version (I cannot think of anything else)
Yeah I can't see the BF16 version on HF.
If you ca upload that would be nice.
I made a dequantizer I've used with the Ministral 3 instruct models.
If anyone need it

https://github.com/csabakecskemeti/ministral-3_dequantizer_fp8-bf16

@bartowski1182
Copy link
Contributor

bartowski1182 commented Dec 5, 2025

I can see it here:

https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-BF16

@csabakecskemeti
Copy link
Contributor

csabakecskemeti commented Dec 5, 2025

You're right, they just removed it from the collection (it it was ever there :p) there's where I looked for. My bad

@CISC
Copy link
Collaborator

CISC commented Dec 5, 2025

@ngxson Thank you so much for this. I've also tested the conversion from your branch the convert script succeeded (with --mistral-format) but at inference time (Q8_0) I've received: llama_model_load: error loading model: missing tensor 'blk.0.attn_k_b.weight' Tried F16 too it also failed on the same (it should have been failed on not enough memory)

It looks like @ngxson forgot wkv_b remapping in the cleanup.

CISC
CISC reviewed Dec 5, 2025
View reviewed changes
Copy link
Collaborator

@CISC CISC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@csabakecskemeti This should work.

CISC
CISC reviewed Dec 5, 2025
View reviewed changes
@csabakecskemeti
Copy link
Contributor

csabakecskemeti commented Dec 5, 2025 •
edited
Loading

I can confirm it works with the changes suggested by @CISC
(Converted from FP8 dequantized model)

@ngxson @CISC
Apply suggestions from code review
49c4e2d 
Co-authored-by: Sigbjørn Skjæret <[email protected]>
@CISC
Copy link
Collaborator

CISC commented Dec 5, 2025

@ngxson Ouch, that second suggestion should not have been directly applied, GitHub messes up changes outside of preview area. :(

CISC
CISC reviewed Dec 5, 2025
View reviewed changes
@ngxson @CISC
Update gguf-py/gguf/tensor_mapping.py
15f78b6 
Co-authored-by: Sigbjørn Skjæret <[email protected]>
@ngxson ngxson merged commit dbc15a7 into ggml-org:master Dec 6, 2025
6 checks passed
JayZenith pushed a commit to JayZenith/llama.cpp that referenced this pull request Dec 7, 2025
@ngxson @CISC @JayZenith
convert: support Mistral 3 Large MoE (ggml-org#17730)
bbdf0bc 
* convert: support Mistral 3 Large MoE

* filter out vision tensors, add missing keys

* handle vocab

* add temperature_length

* fix mscale_all_dim

* clean up

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <[email protected]>

* fix

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <[email protected]>

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>
@github-actions github-actions bot mentioned this pull request Dec 7, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC approved these changes

+1 more reviewer

@bartowski1182 bartowski1182 bartowski1182 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

python python script changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Can't convert mistral 3 large

4 participants

@ngxson @bartowski1182 @csabakecskemeti @CISC