This repo implements the CIF Aware Missing Modality Imagination Network (CIF-MMIN) for the following paper: "Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities"

python 3.8.0
pytorch >= 1.8.0

First you should change the data folder path in data/config and preprocess your data follwing the code in preprocess/.

The preprocess of feature was done handcrafted in several steps, we will make it a automatical running script in the next update. You can download the preprocessed feature to run the code.

• For Training CIF-MMIN on IEMOCAP:

  First training a model self-supervise model with all audio, visual and lexical modality as the pretrained encoder.

  bash scripts/CAP_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]

  Then

  bash scripts/our/CAP_CIF_MMIN.sh [num_of_expr] [GPU_index]

• For Training CIF-MMIN on MSP-improv:

  bash scripts/MSP_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]

  bash scripts/our/MSP_CIF_MMIN.sh [num_of_expr] [GPU_index]
  

• For Training CIF-MMIN on MOSI:

  bash scripts/MOSI_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]

  bash scripts/our/MOSI_CIF_MMIN.sh [num_of_expr] [GPU_index]
  

Note that you can run the code with default hyper-parameters defined in shell scripts, for changing these arguments, please refer to options/get_opt.py and the modify_commandline_options method of each model you choose.

Baidu Yun Link IEMOCAP A V L modality Features 链接：https://pan.baidu.com/s/1i4_ZKFwGUE4cVrxi20dKPg?pwd=id33 提取码：id33

MSP-IMPROV A V L modality Features 链接：https://pan.baidu.com/s/1UzyiC2idpXM8pz0RU5YOCQ?pwd=pcel 提取码：pcel

CMU-MOSI A V L modality Features 链接：https://pan.baidu.com/s/1N8fl8gQC5HTWXuESPUz1pg?pwd=vm51 提取码：vm51

MIT license.

Copyright (c) 2023 S2Lab, School of Inner Mongolia University.