Rouge and Meteor for multiple references

[avinashsai]

Hi,

Currently rogue and meteor supports only single references. Can we use these metrics to calculate for multiple references?

[mariosasko]

Hi @avinashsai ,

currently, multiple references are not supported. However, we could add a multiref config to fix that. When working with multiple references, we can accumulate them by either taking an average or the best score. Would you like to work on that?

[ashutoshml]

@mariosasko I can help with this issue

[lvwerra]

For meteor this is now done in #164.

[mariosasko]

Nice! Support for multiple references (score_multi) has also been added to Google's ROUGE implementation, so we can do a similar thing for rouge to resolve this issue. cc @sashavor

[lvwerra]

That's great, unfortunately they haven't made a release in ages (google-research/google-research#1199) and installing from source seems broken at the moment (google-research/google-research#1203). Once either of those changes it would be straightforward to integrate score_multi into rouge.

[vishal-burman]

Multiple references for meteor is still not working.

>>> meteor = evaluate.load('meteor')
>>> predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
>>> references = [['It is a guide to action that ensures that the military will forever heed Party commands', 'It is the guiding principle which guarantees the military forces always being under the command of the Party', 'It is the practical guide for the army always to heed the directions of the party']]
>>> results = meteor.compute(predictions=predictions, references=references)
>>> print(round(results['meteor'], 2))

results in this stack error:

[nltk_data] Downloading package wordnet to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package omw-1.4 to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package omw-1.4 is already up-to-date!
Traceback (most recent call last):
  File "sample.py", line 6, in <module>
    results = meteor.compute(predictions=predictions, references=references)
  File "/home/vishalburman/miniconda3/envs/eval/lib/python3.8/site-packages/evaluate/module.py", line 425, in compute
    self.add_batch(**inputs)
  File "/home/vishalburman/miniconda3/envs/eval/lib/python3.8/site-packages/evaluate/module.py", line 503, in add_batch
    raise ValueError(error_msg) from None
ValueError: Module inputs don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: ['It is a guide to action which ensures that the military always obeys the commands of the party'],
Input references: [['It is a guide to action that ensures that the military will forever heed Party commands', 'It is the guiding principle which guarantees the military forces always being under the command of the Party', 'It is the practical guide for the army always to heed the directions of the party']]

[lvwerra]

Did you install evaluate from source? This change is not released, yet. You can install the latest version with:

pip install git+https://github.com/huggingface/evaluate.git

Let me know if this works.

[vishal-burman]

Thanks! @lvwerra . Works now.