LLMs know how to make use of short cuts! This is a good exploration on the potential LLM redundancy. Check our paper for more details!
Attention sinks in LLMs are weird. There’s ~20% of heads that don’t seem to do anything.
Do these heads matter? Turns out that if we get rid of them, benchmark scores don’t change.
