Skip to content

Evaluate uses deprecated use_auth_token and will break with datasets-3.0 #620

Description

@albertvillanova

Evaluate uses deprecated datasets.DownloadConfig.use_auth_token and will break with datasets 3.0 release.

For the next datasets release, we have checked the transformers CI and some example tests fail due to evaluate: https://github.com/huggingface/transformers/runs/29973850925

E             File "/root/project/examples/pytorch/multiple-choice/run_swag_no_trainer.py", line 546, in main
E               metric = evaluate.load("accuracy")
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 748, in load
E               evaluation_module = evaluation_module_factory(
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 680, in evaluation_module_factory
E               raise e1 from None
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 639, in evaluation_module_factory
E               ).get_module()
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 479, in get_module
E               local_path = self.download_loading_script(revision)
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 469, in download_loading_script
E               return cached_path(file_path, download_config=download_config)
E             File "/usr/local/lib/python3.10/site-packages/evaluate/utils/file_utils.py", line 185, in cached_path
E               use_auth_token=download_config.use_auth_token,
E           AttributeError: 'DownloadConfig' object has no attribute 'use_auth_token'

The root cause is that evaluate uses datasets.DownloadConfig.use_auth_token, which was deprecated since datasets-2.14.0 last year:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions