LIMITR: Leveraging Local Information for Medical Image-Text Representation

LIMITR is a multi-modal representation learning model for chest X-ray images and reports.
The model is based on a novel alignment scheme between the visual data and the text, which takes into account both local and global information. Furthermore, the model integrates domain-specific information of two types -- lateral images and the consistent visual structure of chest images. Our representation is shown to benefit three types of retrieval tasks: text-image retrieval, class-based retrieval, and phrase-grounding.

LIMITR manuscript
Gefen Dawidowicz, Elad Hirsch, Ayellet Tal
Technion – Israel Institute of Technology
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

We used Python 3.8 with pytorch 1.11

To clone this repository:

git clone https://github.com/gefend/LIMITR.git

To install Python requirements:

pip install -r requirements.txt

1. Download MIMIC-CXR dataset MIMIC-CXR.
2. Update the path to MIMIC directory (DATA_BASE_DIR) on ./LIMITR/constants.py.
3. The splits we used for evaluation and training are available on ./mimic_csv directory.

The pretrained model checkpoint and its corresponding config are available on Hugging Face:

Update the desired training configuration on ./configs/mimic_config.yaml

Train the model with the following command:

python run.py -c ./configs/mimic_config.yaml --train

Test the model with the following command:

python run.py -c ./configs/mimic_config.yaml --test --ckpt_path=ckpt_path

Update ckpt_path with the desired checkpoint for evaluation.

LIMITR can extract and save global image and text (report) embeddings for an evaluation dataset using the extract_features.sh script.

The input CSV must contain the following columns:

The script runs extraction for both gt and output reports and then merges the per-batch files into final embedding files:

bash extract_features.sh \
  --ckpt_path /path/to/epoch=8-step=44963.ckpt \
  --features_base_dir /path/to/output/dir \
  --feature_folder my_features \
  --evaldata_csv_path /path/to/evaldata.csv

The following merged files are saved to features_base_dir/feature_folder/:

@InProceedings{Dawidowicz_2023_ICCV,
    author    = {Dawidowicz, Gefen and Hirsch, Elad and Tal, Ayellet},
    title     = {LIMITR: Leveraging Local Information for Medical Image-Text Representation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21165-21173}
}