Self-distillation with privileged context shifts behavior and can hurt reasoning. Repulsion reverses the shift but becomes unstable. In our experiments, contrasting both signals improves reasoning while keeping training stable.
Blog: self-distillation.github.io/repulsive_attr…
