Toxicity Measurement

[sashavor]

Initial draft of the toxicity metric -- would love your thoughts, @mathemakitten and @lvwerra !

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[lvwerra]

Thanks for adding this! The PR is in pretty good shape already! Mostly added some comments about efficiently loading the pipeline.

Just looking at the functionality this seems to me also a case where it is not so clear why this shouldn't be a measurement (essentially you look at text files and it doesn't matter so much whether they are generated or human written).

[mathemakitten]

Add that type is str. Should we specify that right now we only allow for one label here? Toxicity is often a multi-class prediction problem wrt toxicity along several axes (e.g. identity-based hate vs. racism) but right now we only handle one class.

[sashavor]

and how would you aggregate the results across different labels?
e.g. if you have
{'offensive': 0.65, 'hate': 0.98}, then what?

[mathemakitten]

Ah yes, aggregation would be a bit tricky. I think the Perspective API (as an example) reports these results back unaggregated across categories — there's an individual score for each category of 'identity hate', 'toxicity', 'sexism', 'racism', 'sexually explicit' etc. and they don't aggregate across categories.

I assume the idea is that as an end user of a toxicity API you'd want to handle cases of sexually explicit content differently than identity-based hate, so the granularity is helpful/necessary. In this case an equivalent process would be to not aggregate when there are several types of toxicity specified, and report back per-toxicity-class (e.g. "toxic_labels" is a list instead of a str). What do you think?

[sashavor]

For now I can only find binary hate speech classification models on the Hub, so maybe we keep it like this for now?

[lvwerra]

Looks good, left mostly nits! I think there is an issue with the docstring based on the CI error. Also if you merge main into your branch the timeout issue of the CI should not be there anymore.

[lvwerra]

From the CI:

Toxicity has inconsistent leading whitespace: '    `aggregation` (optional): determines the type of aggregation performed on the data. If set to `None`, the scores for each prediction are returned.'

[lvwerra]

Thanks for adding this - added a few final remarks suggestions. Then it's good to go :)

[lvwerra]

no references, code on github we can reference here?

[sashavor]

Not really, there is just the dataset that the toxicity model was trained on? https://github.com/bvidgen/Dynamically-Generated-Hate-Speech-Dataset

Not sure if that's helpful

[mathemakitten]

Is there a reason we wouldn't want to link to the RealToxicityPrompts repo? The classifier is different (Perspective vs. FAIR classifier) but it's the same idea, and RealToxicityPrompts is a canonical citation for the toxicity metric in the past few years.

https://github.com/allenai/real-toxicity-prompts/

[sashavor]

I'm not sure it's super useful, since it's a general toxicity measure used by lots of other repos, not only real toxicity prompts (who also use a completely different model + approach)

…

On Wed, Aug 24, 2022 at 12:28 PM helen ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In measurements/toxicity/toxicity.py <#262 (comment)>: > + codebase_urls=[], + reference_urls=[], Is there a reason we wouldn't want to link to the repo corresponding to the dataset we use here? They use Perspective as opposed to the FAIR model for scoring, but the data and general idea is the same. https://github.com/allenai/real-toxicity-prompts — Reply to this email directly, view it on GitHub <#262 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMMIITX3DV7KPUTC6UESO3V2ZERZANCNFSM566LTUCA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Sasha Luccioni, PhD Postdoctoral Researcher (Mila, Université de Montréal) Chercheure postdoctorale (Mila, Université de Montréal) https://www.sashaluccioni.com/ [image: Image result for universite de montreal logo]