Pinned
My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n
00:00
Shreyas Kapur
53 posts
Shreyas Kapur
@shreyaskapur
- Replying to @shreyaskapurBuilt with Google Gemini Flash 2.0 Image generation :D
- Replying to @shreyaskapurWe develop an analogous version of “noise” for syntax trees inspired by the computer security literature on fuzzing🎲. And we teach our model to reverse this noise⏪. 2/n
- Replying to @shreyaskapurWe managed to get part of our project running in the browser, Website🌎: tree-diffusion.github.io Paper📄: arxiv.org/abs/2405.20519 Code🖥️: github.com/revalo/tree-di… Thanks for my wonderful collaborator @jenner_erik, and advisor Stuart Russell! n/n 🧵
- I had a lot of fun working on this. I didn't believe that a chess playing neural net could learn to do look-ahead just in its weights, so I was definitely the non-believer in this project.♟️Do chess-playing neural nets rely purely on simple heuristics? Or do they implement algorithms involving *look-ahead* in a single forward pass? We find clear evidence of 2-turn look-ahead in a chess-playing network, using techniques from mechanistic interpretability! 🧵
00:00 - Replying to @shreyaskapurOur implementation works on a given context-free grammar. Here is an example of our model diffusing a smaller “SVG”-like language. 4/n
00:00 - Replying to @shreyaskapurA model that *edits* code makes it really easy to combine it with a search algorithm🔎. 3/n
00:00 - Replying to @shreyaskapurThis is what that language looks like, 5/n
- Replying to @shreyaskapurOf course, our architecture is also a Vision-Language Transformer that is trained to edit code via tree diffusion. 7/n
- Replying to @shreyaskapurWe show how our approach outperforms previous methods, including rejection sampling a Vision-Language Transformer that is specifically trained on these tasks (CSGNet in this figure). 6/n
- Replying to @shreyaskapurThese languages are small, and we only show this approach on a fairly narrow inverse-graphics task. In the future, we hope to show that this approach may potentially work more generally with languages with loops and variables. 8/n
- Replying to @sdtoyer😂I'm glad you asked Sam! We've been working on a modern, functional, and performant library for graphics and diagrams in Python called iceberg,
- Come check out my tree diffusion poster at the system 2 reasoning at scale workshop at NeurIPS!Best poster moment at #NeurIPS2024
