Skip to content

Add confusion matrix#528

Merged
lvwerra merged 13 commits into
mainfrom
confusion
Dec 27, 2023
Merged

Add confusion matrix#528
lvwerra merged 13 commits into
mainfrom
confusion

Conversation

@osanseviero

Copy link
Copy Markdown
Contributor

This PR adds the confusion matrix metric. Some flake errors forced me to change two small type checks.

@osanseviero
osanseviero requested a review from lvwerra December 16, 2023 16:37

@lvwerra lvwerra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

@lvwerra

lvwerra commented Dec 21, 2023

Copy link
Copy Markdown
Member

Actually looks like the doctests are failing. Did you test if the docstring example result match reality?

@osanseviero
osanseviero requested a review from lvwerra December 23, 2023 18:38
@osanseviero

osanseviero commented Dec 23, 2023

Copy link
Copy Markdown
Contributor Author

It should be good now :)
Edit: test passing locally but not here, will look into it next week

@osanseviero

Copy link
Copy Markdown
Contributor Author

Sorry for all the noise with the commits. After some testing, I could pinpoint the issue of different string representations in Windows vs Ubuntu. E.g.

Ubuntu

{'confusion_matrix': array([[1, 0, 1],\n       [0, 2, 0],\n       [1, 1, 0]])}\n

Windows

{'confusion_matrix': array([[1, 0, 1],\n       [0, 2, 0],\n       [1, 1, 0]], dtype=int64)}\n

An alternative would be to just print the shape or sum of the confusion matrix, but this loses a bit of the point. This is also a bit confusing because other metrics such as mean_iou or mse don't face this issue.

@osanseviero

Copy link
Copy Markdown
Contributor Author

Ok - the solution was to use ellipsis so it works with or without dtype. Let me know if this is ok. I also changed the Actions workflow so that if the ubuntu test fails, the windows one keeps running. This will consume more resources per workflow run, so let me know if I should revert that part - it would have been useful to identify this bug much faster.

@lvwerra

lvwerra commented Dec 27, 2023

Copy link
Copy Markdown
Member

Thanks for fixing it. Looks all good!

@lvwerra
lvwerra merged commit 8dfe057 into main Dec 27, 2023
@lvwerra
lvwerra deleted the confusion branch December 27, 2023 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants