MSR-Codec

Audio codecs are a critical component of modern speech generation systems. This paper introduces a low-bitrate, multi-scale residual codec that encodes speech into four distinct streams: semantic, timbre, prosody, and residual. This architecture achieves high-fidelity speech reconstruction at competitive low bitrates while demonstrating an inherent ability for information disentanglement. We construct a two-stage language model for text-to-speech (TTS) synthesis using this codec, which, despite its lightweight design and minimal data requirements, achieves a state-of-the-art Word Error Rate (WER) and superior speaker similarity compared to several larger models. Furthermore, the codec’s design proves highly effective for voice conversion, enabling independent manipulation of speaker timbre and prosody.

Our paper on this project has been published! You can read it here: MSR-Codec.

• Audio Codec
• Speech Generation
• Low-bitrate
• Information Disentanglement

Clone and Install

Here are instructions for installing on Linux.

• Clone the repo

git https://github.com/herbertLJY/MSRCodec.git
cd MSRCodec

• Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
• Create Conda env:

conda create -n msrcodec -y python=3.10
conda activate msrcodec

• Please install the textless first, following https://github.com/facebookresearch/textlesslib/tree/main , or

git clone https://github.com/facebookresearch/textlesslib.git
git clone https://github.com/facebookresearch/fairseq.git
export PYTHONPATH=textlesslib/:fairseq/:$PYTHONPATH

• Install requried libraries:

pip install -r requirements.txt

Checkpoint download

Please download the pretrained checkpoints into the folder: ckpt

• Hubert model
• Hubert quantizer
• CAM++
• MSR-Codec

You can simply run the demo with the following commands:

bash infer.sh

Alternatively, you can modify the script to generate your speeches：

python ignore inference.py \
    --prompt_wav assets/src/121_127105_000009_000000.wav \
    --prompt_text "We say, of course, somebody exclaimed, that they give two turns!" \
    --gen_text "How are you? This is lightspeed studio from tencent." \
    --output_dir out_dir

Here are some demos generated by MSR-Codec for zero-shot TTS and voice conversion, please visit our demo page.