Some metrics have unbounded cardinality

[v-homsi]

After adding a stock Evmos full node to my infrastructure, I noticed a large impact on my Prometheus servers. A brief investigation revealed the cause to be evmosd_tx_msg_ethereum_tx_gas_limit_per_gas_used. Specifically, that metric — I believe added #1106? — includes a "to" label whose values are just addresses. That means that metric has unbounded cardinality, which is unfortunately a no-go for Prometheus.

Digging around, I saw a few other instances of unbounded labelsets. Same thing applies there.

Label values basically gotta be enums — HTTP status codes, well-defined route names, etc. If you really wanna use arbitrary data, you gotta make sure there's some reasonable upper bound to the cardinality. Any kind of Cosmos/Tendermint/etc. address doesn't work, I don't think.

(To give some scale, after running the Evmos node for about a week, this single metric produced as many timeseries (~50k) as the rest of my infrastructure (20 or 30 hosts, tons of services, exporters, etc. etc.) did over 30 days 😲)

edit: Sorry, it was actually 3 metrics:

evmosd_tx_msg_ethereum_tx_gas_limit_per_gas_used
evmosd_tx_msg_ethereum_tx_gas_used_total
evmosd_tx_msg_ethereum_tx_total

Issue originally created in Evmos repo. Creating issue local to repo to close the PR.