rohan anil

@_arohan_

co-founder, board @coreauto - and continuing to aspire to understand deep learning.

Pinned
rohan anil
@_arohan_
Apr 19
It turns out multi step backpropaganda is better. paper has a beautiful way of improving backpropagation. One iteration cleanly gets us backprop, multiple iterations get us a preconditioned update.
rohan anil
@_arohan_
Apr 19
Replying to @LinYorker @ryu0000000001 and @weijie444
arxiv.org/abs/2106.06199 Same update here
5
rohan anil
@_arohan_
6h
Many of the innovations that I remember from Google Brain time that made into frontier which frontier calls algorithmic progress was discovered with less than 1e20 flops and then scaled to 1e25 flops.
3
rohan anil
@_arohan_
17h
All models consistently hillclimbing writing gobbledygook this year needs to be studied. I reckon no one is paying anymore attention to what models are saying
5
rohan anil
@_arohan_
Sep 5
AdaGrad is the best!
jianlin.su
@Jianlin_S
Sep 5
Revisiting Convergence Results in Convex Optimization (IX) kexue.fm/archives/11882 Revisits AdaGrad, the seminal work on adaptive gradient algorithms, reproducing its full derivation via "convergence analysis → minimizing the upper bound → optimal preconditioning matrix."
rohan anil
@_arohan_
Sep 5
Is it easier to get 10 1K GB300s or 10k GB300s in one place. Is there a market tracking for such things, or is it in an inefficient market?
21