whisper : mark speakers/voices (diarization)

[abelbabel]

Hi,

I'm not so much into the details of whisper or whisper.cpp and I don't know if it is currently even possible with the foundation, but it would be nice if speakers could be marked or speaker-changes / voice-changes.

This would be very handy when processing interviews, radio/tv shows, films, etc.

Kind regards,
abelbabel

[ArtyomZemlyak]

I think its a very not easy task - about quality.
I recomend use for this another model. But in my research of this field, now not exist very good open source solution for this.
But u can check pyannote for this. Some already implemented it with whisper usage:
https://github.com/Majdoddin/nlp

[abelbabel]

yeah, also saw this

openai/whisper#264

Seems as if they do it with two runs: one for the spoken text, one for the speakers and then merging the results.

[jaybinks]

Personally, id be more than happy for whisper to just do speaker detection based on left & right channels on a stereo audio file. But I can achieve this by just running it twice.

[ggerganov]

@jaybinks
This can be added very easily as a built-in option.
A naive algorithm would be for each transcribed segment to measure the signal energy during the time interval for that segment in the 2 channels and predict the speaker based on which one is bigger.

[R4ZZ3]

One option would be to use pyannote.audio to diarize first --> then run whisper on each recognized section @abelbabel

[ggerganov]

@jaybinks
Added support for stereo-channel diarization - add the --diarize argument to main.
Not sure if it works, because I don't have any data to test with

[abelbabel]

Personally, id be more than happy for whisper to just do speaker detection based on left & right channels on a stereo audio file. But I can achieve this by just running it twice.

Does this approach have the assumption that you only have two speakers and each speaker is well separated each on a single channel? This is a special case which is only applicable to special recordings in an audio studio - from my point of view. Or am I wrong?

[jaybinks]

This absolutely is a special case, but its also simple to implement and allows the problem to be broken up. I'm lucky that in my scenario, I have a separate mic per speaker in the conversation so it's perfectly isolated.

…

On Sun, 27 Nov 2022, 9:51 am abelbabel, ***@***.***> wrote: Personally, id be more than happy for whisper to just do speaker detection based on left & right channels on a stereo audio file. But I can achieve this by just running it twice. Does this approach have the assumption that you only have two speakers and each speaker is well separated each on a single channel? This is a special case which is only applicable to special recordings in an audio studio - from my point of view. Or am I wrong? — Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALQR62XRG2NRLGNR5BEUQLWKKO7HANCNFSM6AAAAAARH4FNAI> . You are receiving this because you were mentioned.Message ID: ***@***.***>