Feature: Parallel SHA-256

[ashvardanian]

Describe what you are looking for

I’m generally not a fan of reimplementing trivial broadly supported functionality wrapping one native Assembly instruction, but SHA256 is so widespread that extending our dynamic dispatch mechanism can be a value added to the community.

Please let me know if your application needs SHA256 and if you are open to contribute. That set if kernels will go to a separate header file.

Can you contribute to the implementation?

• I can contribute

Is your feature request specific to a certain interface?

It applies to everything

Contact Details

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashvardanian]

We may consider the SHA-256 for the next minor release, providing 2 flavors - one single-threaded and another one for parallel execution on CPUs and GPUs. On the CPU side we'll use Intel SHA-NI and ARMv8 Crypto.

Sadly there are no UUID schemes directly benefiting from fast SHA-256, but the raw hash itself is used in Git, Apache Iceberg, and Delta Lake (developed at Databricks, now under the Linux Foundation).

[ashvardanian]

We now have basic SHA-256 functionality, but it's slower than some of the hand-written or Perl-generated ASM kernels in TLS libraries like OpenSSL. The question is, can we go faster than TLS?

[MarekKnapek]

What is parallel SHA-256? Say, you have a 1GB worth of data and you want to compute a SHA-256 hash of it ... no parallelism can help you. Ever. It is simply impossible. Maybe you think of something else when you say parallel SHA-256. Please explain what do you mean.

[ashvardanian]

Hi @MarekKnapek! I'm hoping to compute multiple SHA-256 hashes for different inputs simultaneously.