we can now do inference with 128K context length in 18.9 GB peak memory if we apply kv cache quantization, linear causal mask and int4 weight-only quantization based off gpt-fast. First bottleneck...

We curated a dataset of 18K pairs of PyTorch Github programs and their corresponding Triton code using torch.compile, some synthetic data and then used that to train an SFT model. We evaled on...