I like this idea a lot. Simple and clean. Rather than just having an equal weight residual sum of transformer layers, let each layer use attn to compute an weight for each previous layers residual. Allowing each layer of the transformer to build an appropriate input by weighting
