Summary of GPT-OSS architectural innovations:
1. sliding window attention (ref: arxiv.org/abs/1901.02860)
2. mixture of experts (ref: arxiv.org/abs/2101.03961)
3. RoPE w/ Yarn (ref: arxiv.org/abs/2309.00071)
4. attention sinks (ref: streaming llm arxiv.org/abs/2309.17453)
