Repository navigation
[BUG]: Colossalai-OpenMoE-8b : loss value is very large and cannot converge #5212
Description
🐛 Describe the bug
I am currently running the Colossalai/examples/language/openmoe project with the following experimental setup:
datasets: load_dataset("yizhongw/self_instruct/data/finetuning/self_instruct_221203", "super_natural_instructions"),
model: openmoe-8b
A100_num_GPU：8
Epoch：3
BatchSize：4
LR：0.00001
zero_stage：1
precision：bf16
Boost plugin：ep_zero
extra_dp_size：2
max_length：2048
### Issue : loss value is very large and cannot converge
I've encountered challenges during the training process where the convergence seems unachievable. The training loss value persists at an exceptionally high level (exceeding 2.3E+10) even after running 3 epochs. The logged information provided below showcases loss values:
Furthermore, in the default setup of the openmoe project, the lr is 0.00001. Considering this, I suspect the issue might stem from an overfitting problem. Consequently, I attempted to adjustment the lr parameter(from 0.00001 to 0.0000001) ,but still encountered the same problem.
Environment
torch 1.13.1
python 3.8.17
cuda:11.7.17
Activity
Orion-Zheng commentedon Dec 30, 2023
Thank you for your valuable feedback! 😃We are working on this bug and will get back to you later.
noob-ctrl commentedon Jan 5, 2024
I also encountered this problem
noob-ctrl commentedon Jan 8, 2024
@Orion-Zheng Has this bug been solved?
obananas commentedon Feb 28, 2025
@Orion-Zheng Has this bug been solved?
JazyJiang commentedon Oct 22, 2026
Hi, thanks for reporting this, and sorry it went unanswered for so long. We're going through older issues and closing the ones that no longer apply to the current codebase. It was reported against PyTorch 1.13.1, which current ColossalAI no longer supports (PyTorch 2.x is required). We're marking it as stale and will close it in 14 days. If you still see this problem with the latest ColossalAI on PyTorch 2.x, please comment here (or open a new issue) with your environment and a minimal reproduction, and we'll take a look.
JazyJiang commentedon Nov 5, 2026
Closing this issue since there has been no response in the 14 days after it was marked as stale. If you still run into this problem with the latest ColossalAI on PyTorch 2.x, please open a new issue using the bug report template, with your environment and a minimal reproduction.