Log inSign up
Markus Zhang profile banner
user avatar

Markus Zhang

@photon_mz
junior cs/math @stanford, intern pretraining @tilderesearch
Stanford, CA
markusz.dev
Joined August 2016
902
Following
530
Followers
RepliesMedia
  • Pinned
    user avatar
    Markus Zhang
    @photon_mz
    Jan 30
    Over the weekend, I managed to beat the NanoGPT world record by 0.3s, by untying Value Embeddings (VE) 😊 There’s a growing trend in sparse embeddings (Deepseek engram, Meta STEM), and @karpathy himself has extensively ablated VEs in nanochat
  • user avatar
    Markus Zhang
    @photon_mz
    Feb 18
    Glad to contribute my second WR to the nanogpt speedrun, -0.4s! I continued my work on VEs and found: 1. Later layers benefit from VEs more 2. Spherical Gaussian init is good 3. Gates should take input from both the VE and the residual X
    user avatar
    Larry Dial
    @classiclarryd
    Feb 16
    The NanoGPT Speedrun WR has broken below 90s, dropping from 92.1 to 89.8 from 4 recent contributions: 1. Tuned Kernels (-0.4s) from @.EmmetBicker on github & AI System Aster 2. Tuned Value Embeds (-0.4s) from @photon_mz 3. Sparse comms for bigram gradients (-0.3s) from
    Image
  • user avatar
    Markus Zhang
    @photon_mz
    Aug 9, 2025
    in high school I was obsessed with making the perfect playlist, so I scraped all of public Spotify to train embedding models. I was hella dumb and had a vocab of 10M := 10M distinct songs, and got destroyed chatgpt “prompt me a playlist”. Led me to obsession with chatgpt