Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
How to compute the peak performance of GPU tensor cores?
Not sure if this is the right place to ask, but I am trying to understand the reported theoretical performance of tensor cores on NVIDIA GPUs.
Taking the H100 SXM5 as example, the reported peak performance for non-tensor cores is quite straightforward: the number of CUDA cores multiplied with the clock speed gives the number of MACs and doubling this figure gives the reported FLOP count. In our example: $16\,896 \,\text{cores} * 1\,980 \,\text{MHz} * 2 \,\frac{\text{FLOPs}}{\text{core}} = 66.908 \,\frac{\text{TFLOPs}}{\text{s}}$
Now, tensor cores enable matrix multiplications of two fixed sized matrices (e.g. a 16x8 with a 8x16 matrix for FP32 according to the CUDA docs) with a single operation. Using this example, this means this operation corresponds to $2 * 16 * 16 * 8 = 4\,096\,\text{FLOPs}$. Multiplying the number of tensor cores with the tensor core clock speed and this multiplier, we arrive at a theoretical peak performance of $528\,\text{cores} * 1\,830\,\text{MHz} * 4\,096 \,\frac{\text{FLOPs}}{\text{core}} = 3\,957.719 \,\frac{\text{TFLOPs}}{\text{s}},$ which is about 8 times more than what is reported.
One possible explanation for this factor 8 could be that it actually requires 8 tensor cores for one operation. Alternatively, it could be that each tensor core requires 8 cycles to finish the computations. I have been trying to find more information on tensor cores and how they work, but couldn't find anything that could explain where this factor 8 comes from.
This leaves me with the question(s): How to compute the peak performance of tensor cores? Is there some documentation (that I missed) explaining how tensor cores approximately work?
1 answer
The following users marked this post as Works for me:
| User | Comment | Date |
|---|---|---|
| mr Tsjolder | (no comment) | Apr 7, 2026 at 12:55 |
It turns out there is a distinction between the API (i.e. the CUDA docs) and the actual implementation.
In the Hopper whitepaper that is linked in the question (h100 SXM5), the implementation details are communicated by means of figures 8-11. Concretely, each figure shows 4 clusters for each architecture, corresponding to the 4 tensor cores per SM. The gray cubes apparently correspond to the number of performed MACs and the dimensions of one such a block of cubes are then the matrix dimensions that can be handled by a single tensor core per cycle.
For the example with the FP32/TF32 throughput of the tensor cores, this means that a single core computes the product of a 4x8 matrix with a 8x8 matrix, corresponding to $2 * 4 * 8 * 8 = 512\,\text{FLOPs}$ (according to figure 11). As a result, we obtain the peak tensor core performance reported in the document: $528\,\text{cores} * 1\,830\,\text{MHz} * 512 \,\frac{\text{FLOPs}}{\text{core}} = 494.714 \,\frac{\text{TFLOPs}}{\text{s}}.$
I find it a bit weird that this information is only communicated implicitly by means of a figure, but this seems to resolve the issues for most non-consumer architectures that I looked into.
0 comment threads
0 comment threads