lib/verifysig.c: work around broken signatures with openssl>=3 #544
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main issue is that our signatures contain a sha1 id in the ASN1 but a sha256 checksum length and message.
Prior to openssl 3 this worked and the full sha256 checksum was used, because the ASN1 was decoded on the fly and the whole message is compared against the checksum.
With version 3 openssl switched to just compare hard coded prefixes and since our prefix is broken its not there.
So we work around this by also hard-coding our broken prefix and instead of using the RSA signature apis, we use the public key to decrypt the signature and compare the content manually with our broken prefix.
This is fucking cursed and uses the deprecated apis, but works. Doing this is not possible with the new api's can't use a public key to independently decrypt the signature.