In the datasets library, there is a handy get_dataset_config_names() function (docs) that returns all the available config names for a given dataset.
It would be useful to have a similar function in evaluate, since metrics like glue and bleurt have multiple config names, and programatic access to these options is handy.
However, a quick look at the evaluate codebase shows this isn't quite as easy to achieve as in datasets. The main limitation seems to be the absence of a builder config for metrics / evaluation modules. This is used in datasets here to quickly inspect the set of possible config names.
cc @TristanThrush this would allow us to avoid needing to import every metric explicitly in the evaluation service UI (when we check for supported metrics).
In the
datasetslibrary, there is a handyget_dataset_config_names()function (docs) that returns all the available config names for a given dataset.It would be useful to have a similar function in
evaluate, since metrics likeglueandbleurthave multiple config names, and programatic access to these options is handy.However, a quick look at the
evaluatecodebase shows this isn't quite as easy to achieve as indatasets. The main limitation seems to be the absence of a builder config for metrics / evaluation modules. This is used indatasetshere to quickly inspect the set of possible config names.cc @TristanThrush this would allow us to avoid needing to import every metric explicitly in the evaluation service UI (when we check for supported metrics).