[ arXiv | Data | Documentation | Tutorials | Cite ]

Welcome to the official GitHub repository of the HEST-Library introduced in "HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis", NeurIPS Spotlight, 2024. This project was developed by the Mahmood Lab at Harvard Medical School and Brigham and Women's Hospital.

• HEST-1k: Free access to HEST-1K, a dataset of 1,276 paired Spatial Transcriptomics samples with HE-stained whole-slide images
• HEST-Library: A series of helpers to assemble new ST samples (ST, Visium, Visium HD, Xenium) and work with HEST-1k (ST analysis, batch effect viz and correction, etc.)
• HEST-Benchmark: A new benchmark to assess the predictive performance of foundation models for histology in predicting gene expression from morphology

HEST-1k, HEST-Library, and HEST-Benchmark are released under the Attribution-NonCommercial-ShareAlike 4.0 International license.

• 8.02.26: 18 new Xenium (including Xenium 5k) samples added to HEST (v1.3.0)!

• 6.01.26: 27 new high-quality Visium HD samples added to HEST (v1.2.0)!

• 21.10.24: HEST has been accepted to NeurIPS 2024 as a Spotlight! We will be in Vancouver from Dec 10th to 15th. Send us a message if you wanna learn more about HEST (gjaume@bwh.harvard.edu).

• 23.09.24: 121 new samples released, including 27 Xenium and 7 Visium HD! We also make the aligned Xenium transcripts + the aligned DAPI segmented cells/nuclei public.

• 30.08.24: HEST-Benchmark results updated. Includes H-Optimus-0, Virchow 2, Virchow, and GigaPath. New COAD task based on 4 Xenium samples. HuggingFace bench data have been updated.

• 28.08.24: New set of helpers for batch effect visualization and correction. Tutorial here.

To download/query HEST-1k, follow the tutorial 1-Downloading-HEST-1k.ipynb or follow instructions on Hugging Face.

NOTE: The entire dataset weighs more than 2TB but you can easily download a subset by querying per id, organ, species...

git clone https://github.com/mahmoodlab/HEST.git
cd HEST
conda create -n "hest" python=3.11
conda activate hest
pip install -e .

To run HEST-Benchmark and load patch encoder models, install benchmark extras:

pip install -e ".[benchmark]"

sudo apt install libvips libvips-dev openslide-tools

If a GPU is available on your machine, we recommend installing cucim on your conda environment. (hest was tested with cucim-cu12==24.4.0 and CUDA 12.1)

pip install \
    --extra-index-url=https://pypi.nvidia.com \
    cudf-cu12==24.6.* dask-cudf-cu12==24.6.* cucim-cu12==24.6.* \
    raft-dask-cu12==24.6.*

NOTE: HEST-Library was only tested on Linux/macOS machines, please report any bugs in the GitHub issues.

You can then simply view the dataset as,

from hest import iter_hest

for st in iter_hest('../hest_data', id_list=['TENX95']):
    print(st)

The HEST-Library allows assembling new samples using HEST format and interacting with HEST-1k. We provide two tutorials:

• 2-Interacting-with-HEST-1k.ipynb: Playing around with HEST data for loading patches. Includes a detailed description of each scanpy object.
• 3-Assembling-HEST-Data.ipynb: Walkthrough to transform a Visum sample into HEST.
• 5-Batch-effect-visualization.ipynb: Batch effect visualization and correction (MNN, Harmony, ComBat).

In addition, we provide complete documentation.

The HEST-Benchmark was designed to assess 11 foundation models for pathology under a new, diverse, and challenging benchmark. HEST-Benchmark includes nine tasks for gene expression prediction (50 highly variable genes) from morphology (112 x 112 um regions at 0.5 um/px) in nine different organs and eight cancer types. We provide a step-by-step tutorial to run HEST-Benchmark and reproduce our results in 4-Running-HEST-Benchmark.ipynb.

HEST-Benchmark was used to assess 25 publicly available models. Reported results are based on Ridge Regression with PCA (256 factors). Ridge regression can penalize models with larger embedding dimensions; PCA-reduction is used for fairer comparison. Model performance is measured with Pearson correlation.

Our tutorial in 4-Running-HEST-Benchmark.ipynb will guide users interested in benchmarking their own model on HEST-Benchmark.

Note: Spontaneous contributions are encouraged if researchers from the community want to include new models. To do so, simply create a Pull Request.

• The preferred mode of communication is via GitHub issues.
• If GitHub issues are inappropriate, email guillaume.jaume@unil.ch (and cc homedoucetpaul@gmail.com).
• Immediate response to minor issues may not be available.

If you find our work useful in your research, please consider citing:

Jaume, G., Doucet, P., Song, A. H., Lu, M. Y., Almagro-Perez, C., Wagner, S. J., Vaidya, A. J., Chen, R. J., Williamson, D. F. K., Kim, A., & Mahmood, F. HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis. Advances in Neural Information Processing Systems, December 2024.

@inproceedings{jaume2024hest,
    author = {Guillaume Jaume and Paul Doucet and Andrew H. Song and Ming Y. Lu and Cristina Almagro-Perez and Sophia J. Wagner and Anurag J. Vaidya and Richard J. Chen and Drew F. K. Williamson and Ahrong Kim and Faisal Mahmood},
    title = {HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2024},
    month = dec,
}