Introducing 𝗦𝘁𝗿𝗲𝗮𝗺𝗕𝗣, a memory-efficient exact backpropagation (BP) method. Compared to gradient checkpointing, it
- Increases max sequence length by approx. 5 times.
- Costs comparable or even less BP time.
Paper: arxiv.org/abs/2506.03077
Code: github.com/Ledzy/StreamBP
