Pinned
Replying to @akshay_pachaar
0.5 GB on a light weight model is completely fixated on fine tuning to your own needs, this is what we need in the open source!
Dev Patel
@_devp
21 | Member of Technical Staff @BoardyAI (Voice + Agents) | Previously Founding Engineer @ getbluejay.ai , AI @Fidelity
- External provider dependency never disappoints (in the worst way). Another reason to build things in-house as much as possible
- undoing positional encoding before compressing the K cache....simple idea that nobody was doing, unlocks 10x KV cache compression with zero accuracy lossi just beat @GoogleDeepMind's turboquant introducing Shard. 10x KV cache compression on Llama-3.1-8B. zero quality loss - 10x @ 8K context, 11.2x @ 32K - NIAH recall 1.000 across 4K-32K - LongBench Δ ≈ 0 vs FP16 turboquant tops out at 4-6x at the same quality. we doubled it.
- @owengretzinger using whisper flow + vorflux to ship during a company lunch🚀
- wrote about why building in public is the most underleveraged career move right now, and why the longer you wait, the harder it gets to close the gap. the job market isn't as bad as you think. it's just bad for people who are invisible.
