Recommended using python 3.12

pip install -r requirements.txt

Our final dataset Bespoke-Stratos-17k contains the following subsets:

• Numina: 10.5k samples from the math, olympiads, and amc_aime subset of the difficulty-labeled Numina dataset.
• APPS: ~2.5k samples from the APPS dataset.
• TACO: ~3k samples from the TACO dataset.
• STILL-2: ~1k samples from the STILL-2 dataset.

Run the following scripts to generate reasoning traces and filter out incorrect reasoning traces:

# Numina Math
python generate_numina_data.py

# APPS
python generate_apps_data.py --split test

# TACO
python generate_taco_data.py --split train
python generate_taco_data.py --split test

Run the following to combine datasets:

python combine_data.py

Using the dataset generated above, we trained Bespoke-Stratos-32B, our reasoning model distilled from DeepSeek-R1 and using Berkeley NovaSky’s Sky-T1 data pipeline. The model outperforms Sky-T1 and o1-preview in reasoning (Math and Code) benchmarks, and almost reaches the performance of DeepSeek-R1-Distill-Qwen-32B while being trained on 47x fewer examples:

We open-source everything to continue experimenting together with the community!

• 32B Model and 7B Model
• Reasoning Dataset
• Blog

@misc{
  bespoke_stratos, 
  author       = {Bespoke Labs},  
  title        = {Bespoke-Stratos: The unreasonable effectiveness of reasoning distillation},  
  howpublished = {https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation},  
  note         = {Accessed: 2025-01-22},  
  year         = {2025}
}