Log inSign up
Taiming Lu profile banner
user avatar

Taiming Lu

@TaimingLu
Ph.D student at @Princeton | Formerly @JohnsHopkins ’25, @HopkinsDSAI @JHUCompSci @jhuclsp @CCVLatJHU | AI/ML/NLP/CV
Princeton, NJ
taiminglu.com
Joined June 2024
629
Following
377
Followers
RepliesMedia
  • user avatar
    Taiming Lu
    @TaimingLu
    May 26
    Knowledge doesn't always flow downhill. We find that in LLM pretraining, a weaker teacher can improve a stronger student, and pushing the teacher further can actually hurt. New paper: Strong Teacher Not Needed? On Distillation in LLM Pretraining.
    A weaker teacher can still help and an overly strong one can hurt — what matters for distillation in LLM pretraining is teacher–student compatibility, not raw teacher strength.