Pancreatic cancer is the third leading cause of cancer-related deaths in the United States. Tragically, 80–85% of cases are diagnosed too late for effective treatment. The disease’s silent progression and anatomical complexity make early detection a formidable challenge. But early detection saves lives — and that’s where AI can make a difference.

To address this, NVIDIA partnered with Johns Hopkins University, along with contributions from 11 institutions worldwide, to launch PanTS — the Pancreatic Tumor Segmentation Dataset, the largest and most comprehensive resource ever released for pancreatic CT analysis.

PanTS at a Glance

• 36,390 CT scans from 145 medical centers across 18 countries
• 993,000+ expert-validated voxel-wise annotations
• Covers pancreatic tumors, pancreas subregions, and 24 surrounding anatomical structures including vascular, skeletal, and abdominal/thoracic organs

Each scan includes rich metadata: patient age, sex, diagnosis, contrast phase, in-plane spacing, and slice thickness. This diversity enables robust model generalization across populations and imaging conditions.

MONAI-Powered Annotation and AI Training

PanTS was built using MONAI Label, NVIDIA’s open-source AI framework for medical imaging. Radiologists used MONAI to perform interactive 3D segmentation, enabling scalable, human-in-the-loop annotation workflows

This approach ensured consistency and speed across nearly a million annotations, setting a new standard for medical imaging datasets.

Human-in-the-Loop Workflow

• AI-generated segmentations using MONAI-based models (e.g., VISTA3D)
• Radiologist validation and refinement
• Multi-rater consensus for high-quality annotations

Benchmark-Leading Performance

Leveraging MONAI, NVIDIA’s open-source medical imaging AI framework, PanTS drastically improves AI accuracy in pancreatic tumor detection and sets a new benchmark for research.

Models trained on PanTS significantly outperform those trained on existing public datasets:

Metric | Improvement Over Baselines

Tumor Segmentation (Dice) | +4.9%

Tumor Segmentation (NSD) | +3.1%

Out-of-Distribution Tumor Detection (AUC) | +14%

Full Anatomy vs. Tumor-Only Training (Dice) | +10.3%

These gains are directly attributable to PanTS’s scale and anatomical richness

Open Science and Developer Access

PanTS is released under a non-commercial license, with a public training set and a reserved test set for third-party benchmarking. This ensures reproducibility and rigorous evaluation.

Global Collaboration and Impact

PanTS was co-developed by NVIDIA and Johns Hopkins University, with contributions from institutions across Europe, Asia, and North America. It represents a global effort to accelerate AI innovation in oncology, radiotherapy planning, and surgical decision support

Conclusion: Building Toward Earlier Diagnosis

PanTS is more than a dataset — it’s a call to action. By enabling earlier and more accurate tumor detection, it has the potential to improve survival rates and transform pancreatic cancer care.

Whether you’re a developer building segmentation models or a researcher benchmarking AI performance, PanTS offers the tools and data to push the boundaries of medical imaging.

Let’s build the future of cancer detection — together.

Core Contributors

• Wenxuan Li, Zongwei Zhou, Alan Yuille — Johns Hopkins University
• Yucheng Tang, Daguang Xu — NVIDIA
• Collaborators from University of Bologna, UC Berkeley, UCSF, Peking University Third Hospital, and more

Core Materials

• Pre-print paper
• Dataset Download

Additional Materials

• GitHub Repository
• Explore MONAI
• Learn with MONAI

1. Dr. Vikash Gupta, Center for Augmented Intelligence in Imaging, Mayo Clinic Florida
2. Jiahui Guan, Senior Solutions Architect, NVIDIA

GitHub URL: https://github.com/vikashg/monai-inference-demo

MONAI has become the de facto standard in medical image AI, offering tools for model training and clinical integration. It provides out-of-the-box tools for the preprocessing and postprocessing of radiological images. MONAI features a broad selection of predefined neural network architectures and training routines based on PyTorch.

The MONAI Model Zoo offers access to pre-trained models based on peer-reviewed research, allowing developers to either fine-tune these models or use them to perform inference. In the latest 1.3 release, the MONAI model zoo introduced a Pythonic API, simplifying the processes for downloading and fine-tuning models.

On the opposite end of the spectrum is MONAI Deploy Express, which provides a comprehensive deployment solution. It uses a MONAI Application Package (MAP) to create application bundles that can be deployed. These MAPs can take a DICOM input and generate an appropriate DICOM image as output. MONAI Deploy Express is an open-source solution that can be integrated into hospital ecosystems, serving as an inference server and a DICOM router.

MONAI effectively covers two opposite ends of the workflow, with MONAI core at one end, focusing on data preprocessing and model training, and MONAI Deploy Express at the other, offering a comprehensive suite of tools for a complete clinical deployment. However, research labs often find themselves in need of a middle ground — a solution that bridges the gap between operating a full-fledged MONAI Deploy Express instance and writing new code for each inference task. The ideal solution is a local inference server that can be consistently referenced for the same purposes. The advantages of having a dedicated model inference server include:

1. A common endpoint that can be invoked for inference tasks
2. A common directory to maintain models and apps

In this post, we will explore the following:

1. How to write a handler function for a MONAI-based application
2. How to write a test function for the aforementioned handler
3. How to deploy a MONAI model using a torchserve inference server
4. How to deploy a MONAI model using a Triton inference server
5. Using a REST API for executing inference
6. A comparison between the two model servers

Prerequisites and Assumptions

We’ll be assuming familiarity with a few key concepts:

• MONAI Transforms: Knowledge of how to use MONAI for transforming medical imaging data
• REST API: Understanding the basics of REST API design for web services interaction
• Deep Learning Concepts: A grasp of fundamental deep learning principles

All of the necessary code, sample data, and models for this guide will be made available through a GitHub Repository.

Creating the ModelHandler

First, we’ll break down the code for Model Handler. This handler consists of four main functions: preprocess, post-process, inference, and handle.

Here, we’ll break down the code into understandable chunks.

1. Initializing the ModelHandler, which inherits from the BaseHandler

class ModelHandler(BaseHandler):
  def __init__(self):
  self._context = None
  self.initialized = False
  self.explain = False
  self.target = 0
  self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

2. Write the data preprocessing function

def preprocess(self, data_fn):

  transforms = Compose([LoadImage(image_only=True),
    EnsureChannelFirst(),
    Resize(spatial_size=(256, 256,24)),
    ScaleIntensityRange(a_min=20, a_max=1200, b_min=0, b_max=1, clip=True), 
    AddChannel()])

  fn = data_fn[0]['filename'].decode()
  img_fullname = os.path.join(input_dir, fn)
  data = []
  batch_size = 1

  for i in range(batch_size):
    tmp = {}
    print(img_fullname)
    tmp["data"] = transforms(img_fullname)
    data.append(tmp)

  return data

The preprocessing function, as it is aptly called, takes a filename data_fn. It loads the data from the specified data directory and performs preprocessing using the MONAI transform chain. In this particular case, the Nifti image is loaded and resized to a spatial size of 256 X 256 x 24, and the intensity values are scaled between 0 and 1. We then generate a list of data. Since we pass only a single file name, it is important that we hardcode the batch_size to 1.

3. Post-processing function

def postprocess(self, inference_output):
  post_trans = Compose([Activations(sigmoid=True), AsDiscrete(threshold=0.5)])
  postprocess_output = post_trans(inference_output)
  SaveImage(output_dir, output_postfix='seg', output_ext='.nii.gz')(postprocess_output[0])

  return [1]

Like the preprocessing function, the post-processing function applies the post-processing transforms to the model’s output. It then saves the output to the predefined data directory. If everything goes well, it will return success.

4. Inference Function

def inference(self, data, *args, **kwargs):
  with torch.no_grad():
  marshalled_data = data.to(self.device)
  results = self.model(marshalled_data, *args, *kwargs)

  return results

Create a Test Function

Before creating a model server, we must ensure that the model performs inference as expected. We’ll utilize the previously created ModelHandler and create a test function to validate the workflow.

from handler import ModelHandler
from ts.torch_handler.unit_tests.test_utils.mock_context import MockContext
from pathlib import Path
import os

MODEL_PT_FILE='traced_segres_model.pt'
CURR_FILE_PATH = Path(__file__).parent.absolute()
EXAMPLE_ROOT_DIR=CURR_FILE_PATH
TEST_DATA=os.path.join(CURR_FILE_PATH, 'test.nii.gz')


def test_segresnet(batch_size=1):
  handler = ModelHandler()
  print(EXAMPLE_ROOT_DIR.as_posix())
  ctx = MockContext(model_pt_file = MODEL_PT_FILE,
    model_dir= EXAMPLE_ROOT_DIR.as_posix(),
    model_file = None,)
  handler.initialize(ctx)
  handler.context = ctx
  handler.handle(TEST_DATA, ctx)

if __name__ == '__main__':
  test_segresnet()

Create a TorchServe endpoint

To set up a TorchServe endpoint with your handler, follow these steps:

1. Create a torch model archive (.mar) using the torchscript model (traced_segres_model.pt) and model handler you wrote earlier. Specify the model name, serialized file, handler path, and version.

torch-model-archiver - model-name segresnet - serialized-file ./traced_segres_model.pt - handler ./handler.py -v 1.0

2. Move the model to the model_store folder

mv segresnet.mar model_store

3. Start torchserve. Specify the model store directory, the model to deploy, and the TorchServe configuration file.

torchserve - start - model-store model_store/ - models segresnet=segresnet.mar - ts-config ./config.properties

4. Test the torchserve is running properly by sending a request to the health endpoint.

curl http://localhost:8080/ping

Expected Response:

{
 "status": "Healthy"
}

5. List all the models deployed on torchserve model server by executing the following command:

curl http://localhost:8081/models?

Expected Output:

{
 "models": [
   {
     "modelName": "breast",
     "modelUrl": "breast.mar"
   },
   {
     "modelName": "segresnet",
     "modelUrl": "segresnet.mar"
   }
 ]

6. Make a prediction request by passing the filename as data

curl -d "filename=test.nii.gz" http://127.0.0.1:8080/predictions/segresnet

If the prediction is successful, it will return ‘1’. You can now see the AI output in the specified output directory.

This process allows for deploying and managing multiple models on TorchServe, enabling a flexible and scalable serving environment for your AI applications.

Triton Inference Server-based model server for MONAI models

Triton Inference Serve enables the deployment of any AI model. It supports multiple libraries like TensorRT, TensorFlow, PyTorch, ONNX, and OpenVINO. In this blog, we will demonstrate MONAI models being deployed on the Triton Inference Server. As MONAI supports PyTorch, we will deploy a PyTorch model on this inference server.

Steps:

1. Convert the model to torchscript using the following code.
2. Write the config.pbtxt file needed for the triton inference server
3. Similar to the torchserve, we need to write handler for image client.

Code to convert the PyTorch model to the traced torch script model. The code can be downloaded from the Github link.

from model_del import ModelDefinition
import torch
 
model_def = ModelDefinition(model_name='SegResNet')
model = model_def.get_model()
model_fn = './model/SegResNet/best_metric_model.pth' # Filename
model.load_state_dict(torch.load(model_fn))
x = torch.zeros(1, 1, 256, 256, 24)
traced_model = torch.jit.trace(model, x)
traced_model.save('./model/SegResNet/traced_segres_model.pt')

This model is saved in a directory.

Now, we need to write the config.pbtxt file.

name: "lv_segmentation"
platform: "pytorch_libtorch"
max_batch_size: 2
input [
  {
    name: "input__0"
    data_type: TYPE_FP32
    dims: [1, 256, 256, 24 ]
  }
]
output [
  {
    name: "output__0"
    data_type: TYPE_FP32
    dims: [1, 256, 256, 24]
  }
]

Explanation of the config.pbtxt file

The field name corresponds to the name of the model. The field platform refers to the deep learning library used. In the present case, we used PyTorch to develop the model and thus the value pytorch_libtorch. The following field max_batch_size is self-explanatory as the maximum number of images that can be inferred at once. The last two fields input and output has three sub-fields: name, data_type and dims. The name refers to the input/output node of the model. In most cases (especially for PyTorch), the nodes names by default are input__0 and output__0. So, we encourage the users to use these values in that field. If it fails, the users should do further investigation. TYPE_FP32 refers to the data_type. The last field dims corresponds to the size of input image and the expected output (image). In this particular case, we are demonstrating a segmentation model and so the input and output dims are the same.

More information about the config.pbtxt file is available at the official Github for Triton: https://github.com/triton-inference-server/server/blob/main/docs/getting_started/quickstart.md

Deploying on Triton Inference Server

To deploy the models, arrange your models in a directory structure more like this:

These are two models: breast_density and lv_segmentation. The model file model.pt is inside the folder named 1 for both models. The 1, in this case, refers to the model version. Once the directory is set, the triton server can be started using the following command:

docker run - gpus=1 -p 8000:8000 -p 8001:8001 -p 8002:8002 -v ${PWD}/model_repository/models:/models nvcr.io/nvidia/tritonserver:23.12-py3 tritonserver - model-repository=/models

The above command maps the model_repository/models directory to the tritonserver. For the first run, it will download the docker images nvcr.io/nvidia/tritonserver:23.12-py3. Make sure that you are downloading the appropriate tritonserver image. More information is available at https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver

Writing the image-client

Now, we can focus on writing the image-client.py. The image-client file mainly contains two functions: preprocess and post_process. The preprocess function looks as follows. One important thing to note in this function is that, though we use MONAI transform chain, it converts the output to a numpy array. This is because triton-inference-server can make predictions on numpy arrays.

def preprocess(img_path="MR.nii.gz"):
  transforms = Compose([LoadImage(image_only=True), 
    EnsureChannelFirst(),
    Resize(spatial_size=(256, 256, 24)),
    ScaleIntensityRange(a_min=20, a_max=1200, b_min=0, b_max=1, clip=True)])
  img_tensor = transforms(img_path)
  results_np = np.expand_dims(img_tensor.numpy(), axis=0)

  return results_np

In this case, we also need to post-process the output and save it as a Nifti or a DICOM-RT file. In this case, we are saving it as a Nifti file. In this function, we use SimpleITK to save a nifti file where we get the “meta” information for the image for a reference image (most likely the input image).

def post_transform(inference_output, out_dir='./', ref_image=None):
  post_trans = Compose([Activations(sigmoid=True),
    AsDiscrete(threshold=0.5), ])
  image_itk = sitk.GetImageFromArray(np.transpose(postprocess_output, [2, 1, 0]))
  image_itk.SetSpacing(ref_image.GetSpacing())
  image_itk.SetOrigin(ref_image.GetOrigin())
  image_itk.SetDirection(ref_image.GetDirection())
  sitk.WriteImage(image_itk, os.path.join(out_dir,
  "segmentation.nii.gz"))

Finally, the driver function for the image-client is written as:

def main():
  img_path = "/data/MR.nii.gz"
  transformed_image = preprocess(img_path=img_path)
  client = httpclient.InferenceServerClient(url="localhost:8000")
  inputs = httpclient.InferInput("input__0", transformed_image.shape, datatype="FP32")
  inputs.set_data_from_numpy(transformed_image, binary_data=True)
  outputs = httpclient.InferRequestedOutput("output__0", binary_data=True, class_count=0)
  
  results = client.infer(model_name="lv_segmentation", inputs=[inputs], outputs=[outputs])
  inference_output = results.as_numpy("output__0")
  
  ref_img_fn = './tmp/MR/MR_preprocessed.nii.gz'
  reader = sitk.ImageFileReader()
  reader.SetFileName(ref_img_fn)
  ref_image = reader.Execute()
  post_transform(inference_output, out_dir='./', ref_image = ref_image)

As before, the complete program is available on the GitHub page. The image-client is called from the command line as

python image-client.py

The image-client is similar to the model_handler. The main difference between model_handler for TorchServe is that for triton inference server, the input to the model should be in a numpy format as opposed to the meta-dictionary format for torchserve.

Conclusion

MONAI has established itself as a go-to standard for processing medical images. MONAI-Deploy-App-SDK and MONAI Deploy are components that provide clinical integration tools for model deployment. However, we believe there is a huge valley of possibilities between MONAI Deploy’s clinical integration and deployment strategies, and no deployment is available in MONAI. Here, we are taking the PyTorch model trained using MONAI and deploying it using model deployment servers like torchserve and Triton Inference Server.

We would like to thank Dr. Mutlu Demirer, Dr. Barbaros Selnur Erdal, and Dr. Richard D. White from the Center for Augmented Intelligence in Imaging at Mayo Clinic, Florida, for their guidance and support on this project.

Academic medical centers, like AZ Delta, are on the cutting edge of medical imaging AI research. Researchers create deep learning models to address all kinds of institutional needs, with workflows to address clinical, quality assurance, and administrative needs. With the volume of data these centers produce, there is arguably no problem that cannot be solved in this space. But what happens after a model is created?

Taking a model from research into production is fraught with challenges at every turn. There are disconnects between research and clinical practice. There is a limited availability of resources and talent to bring these solutions across the divide. And, up to now, there has been no industry-accepted best practice, forcing researchers and IT operations teams to use ad-hoc, impractical solutions. To address these challenges, there needs to be a standardized model to package AI models, integrate the applications into PACS workflows, and maintain the application lifecycle to operate and manage these applications. This process must be repeatable and scalable to manage the ever-increasing volume of projects necessary for healthcare delivery of the future.

Sectra and NVIDIA have teamed up to help academic medical centers to meet this challenge. Sectra Amplifier Services has established the Amplifier Platform infrastructure to bring AI applications from research into reality. NVIDIA infrastructure, coupled with the Project MONAI (Medical Open Network for AI) framework, has created the accelerated pipeline and platform to develop and deploy medical imaging AI applications at scale. With these two components being brought together, academic medical centers around the world, including AZ Delta, can take advantage of these technology breakthroughs to the benefit of both their patients and clinical and administrative staff.

Revolutionizing AZ Delta’s integration of AI applications

With these technologies, AZ Delta brought them to bear to streamline the first of many workflows, with a team that included Dr. Peter De Jaeger, manager of the innovation team and IT, alongside data scientist Dr. Nathalie Mertens and in close collaboration with Dr. Kristof De Smet. The application segments the Psoas muscle from abdominal CT images, including defining the length and volume of the muscle. This use case was selected because of the difficulty in good segmentation, since the muscle can have different orientations and has a tapered geometry. A possible application is an additional objective parameter that reflects the health status of a patient, which has general applicability for every patient from whom a CT abdomen is taken. This requires further research in clinical practice to show outcomes.

“Within AZ Delta, we aim at integrating AI models into the clinical workflow to harmonize precision, efficiency, and accurate patient outcomes,” said Dr Mertens. “This way, AZ Delta embraces a future where medical experts converge with innovation and technology to reach optimal healthcare. Thanks to Sectra, an approachable solution to achieve this goal has been built.”

Sectra Amplifier Services made it easy for AZ Delta

AI adoption will accelerate if the output is seamlessly integrated with the existing ecosystem. This shouldn’t matter whether the output comes from commercial or research applications, seamless integration in the workflow is key to both.

Sectra Amplifier Services facilitates a faster route to AI adoption in clinical practice, providing a single technical and administrative infrastructure to access commercial applications. As an enterprise imaging vendor, Sectra creates a single point of contact for contracting, deploying, and managing applications over the lifecycle of usage.

The Amplifier Services is an open platform that enables healthcare institutions to leverage the Amplifier Platform to build and deploy their own applications as well.

MONAI accelerates AI solutioning and seamlessly integrates into Sectra Amplifier Services

Project MONAI was created to enable medical imaging AI researchers to do their life’s work. Co-founded by academic and industry partners from around the world, including NVIDIA, Project MONAI was created to simplify and accelerate deep learning pipelines for the medical domain. It builds upon common frameworks researchers already use, including PyTorch, and enables the specialization needed to work with medical imaging, going beyond the pixels into the representations of what those pixels mean.

Creating an end-to-end medical AI solution is done by taking ground truth, training a deep-learning model against that truth, packaging up that model into an AI application, and connecting that with the medical imaging ecosystem. Project MONAI includes SDKs that make this possible, for annotation, training and fine-tuning, federated learning, and inference.

Connecting those AI applications into the Sectra Amplifier Platform completes the picture for academic medical centers like AZ Delta, and makes it easier, and scalable for healthcare enterprises.

Learn more about Sectra Amplifier at https://medical.sectra.com/product/sectra-amplifier-marketplace/, and Project MONAI at https://monai.io/.

Authors: James O’Callaghan, PhD, Riccardo Lattanzi, PhD, and Kai Tobias Block, PhD

Center for Advanced Imaging Innovation and Research (CAI2R)

Deploying AI applications into the radiology workflow just got easier with the introduction of MONAI Application Package (MAP) support to the mercure DICOM Orchestrator. By leveraging mercure’s flexible DICOM routing and processing capabilities, extensive monitoring functions, and its open-source model, organizations can create custom workflows that are tailored to their specific needs. A user-friendly web interface makes it easy to integrate, configure, and administer MAPs and provides visualization of job status and audit trails. The deployment standardization approach taken by MONAI may be combined with the simplicity and flexibility of the mercure open-source software to expedite clinical translation of AI models in radiology for a variety of use cases.

Benefits of using mercure to deploy MAPs

The mercure DICOM orchestrator handles interactions and requirements of clinical environments when deploying AI models into radiology workflows. Key benefits include:

• Simple integration: compatible with clinical infrastructure (DICOM compliant), supports cloud deployments, easy to use.
• Flexibility: versatile processing modules, configurable for a wide range of use cases, scalable as demand increases.
• Supports best clinical practices: provides notification functionality to reduce delays in provision of care, monitoring and audit trails, standardization through MAP compatibility.
• Open-source: highly customizable, vendor agnostic with no required collaboration agreements, unlimited free installations so organizations can deploy AI models without incurring licensing costs.

Deployment is simple with mercure

The intuitive web-based interface makes mercure very easy to use. Installation is automatic — by following the quick start guide, users can swiftly get up and running with a fully functional test environment on their machine.

MAPs can be configured using an intuitive web interface:

· Modules page: This is where MAPs can be added as a ‘MONAI’ module type.

· Targets page: This is where details of destination devices (e.g. PACS, VNA, etc.) are provided for routing the results generated by a MAP.

· Rules page: This is where users specify the MAP to run, the target to send results to, and filtering criteria to select incoming data for processing. Notifications can be configured to provide alerts and results to radiologists via email and messaging applications.

· Queue page: Enables users to monitor DICOM data sent to mercure for processing and to review audit trails.

Open-source modules for rapid MAP deployment

There is a growing catalogue of open-source modules and tutorials so that new mercure users can rapidly deploy their first AI applications using trained open-source models. Modules have been published on Docker Hub and can be run out-of-the-box simply by providing the docker tag in the modules page of the mercure user interface. Source code is provided in the mercure GitHub repository to provide users with templates to build their own MAPs as described below.

MONAI Classify

This module provides a template for developing MAPs that perform classification tasks. It extends functionality of the ‘Lung nodule_ct detection’ MONAI bundle to enable mercure to send notification emails when lung nodules are detected. It also outputs DICOM images with bounding-box labels around detected nodules.

Open-source code is available here. The module can be installed in mercure using the docker tag: mercureimaging/map-monaiclassify

MONAI Segment

This module provides a template for developing MAPs that perform segmentation tasks. It is based on the ‘Spleen ct segmentation’ MONAI bundle, which segments DICOM CT images and outputs DICOM CT images with spleen labels.

Open-source code is available here. The module can be installed in mercure using the docker tag: mercureimaging/mercure-monaisegment

Other mercure modules

· TotalSegmentator : A module to deploy the open-source TotalSegmentator model in mercure for segmentation of 104 classes in CT images.

· Pyapetnet: A module to deploy the open-source pyapetnet model for anatomy-guided PET reconstruction.

· Example Inference: A tutorial module that performs a simple slice-by-slice U-Net-based segmentation of the prostate and creates a colored segmentation map that is blended with the input MRI images.

Deploy your own AI application

The quickest way to get started with mercure and deploy your first AI application is to follow the ‘Rapid deployment How-To’ jupyter notebook tutorial. This tutorial is part of an educational exhibit (INEE-47) at the RSNA 2023 annual meeting. In six simple steps, mercure is installed, the MONAI segment module is configured, and segmentation results from provided test data are displayed to demonstrate the simplicity of deploying AI models with mercure.

Learn more

mercure documentation

mercure GitHub repository

mercure Docker Hub

Auto3DSeg is the answer to this demand. Designed with developers in mind, it seamlessly bridges the gap between innovation and practical application, providing an efficient and user-friendly solution. By harnessing the capabilities of MONAI and modern GPU technology, Auto3DSeg empowers developers — both novices and experts — to achieve hassle-free, state-of-the-art performance in 3D medical image segmentation.

Understanding MONAI Auto3DSeg

Auto3DSeg is a MONAI native project, aiming to demonstrate best practices of common 3D segmentation workflows for several algorithms. For non-expert users, it allows them to start with only a few lines of code to automatically train models on their 3D CT or MRI data. For expert users, it provides recipes of best practices for segmentation training with MONAI components, to achieve state-of-the-art baseline segmentation performance, customize it, and further build upon it. Special efforts were put into improving the computational performance of Auto3DSeg, focusing on minimizing training and inference time while maximizing GPU compute utilization.

Key Features:

• Dataset Analysis: Auto3DSeg sets the stage for subsequent steps by analyzing the dataset’s intensity, size, and spacing.
• Algorithm Generation: Algorithm folders are automatically configured based on initial data assessment.
• GPU Integration: Innate GPU support accelerates model training, validation, and inference.
• Hyper-parameter Optimization: Auto3DSeg refines model parameters for optimal performance and accuracy.
• Model Ensemble: Auto3DSeg creates and integrates multiple models, enhancing accuracy and reliability.

Real-world Applications

Let’s explore Auto3DSeg’s capabilities through recent challenges. A team of NVIDIA researchers successfully applied Auto3DSeg in several recent MICCAI 2023 challenges.

The team members are Andriy Myronenko, Dong Yang, Yufan He, and Daguang Xu.

BraTS 2023–Multiple 1st and 2nd Place wins: Auto3DSeg showcased its abilities placing 1st and 2nd in multiple BraTS competitions. (1st Place in Brain Metastases, 1st place in Brain Meningioma, 1st place in Brats-Africa Glioma, 2nd place in Adult Glioma, 2nd place in Pediatric Glioma). See the leaderboardhere: https://www.synapse.org/#!Synapse:syn51156910/wiki/621282

KiTS 2023–1st Place: Auto3DSeg excelled in the KiTS 23 segmentation challenge at MICCAI 2023, achieving top-tier performance in 3D kidney segmentation. See the leaderboard here: https://kits-challenge.org/kits23/#kits23-official-results.

SEG.A. 2023–1st Place: Auto3DSeg demonstrated adaptability and robustness by winning the Aorta segmentation challenge at MICCAI 2023. See the leaderboard here: https://multicenteraorta.grand-challenge.org/.

MVSEG 2023–1st Place: In the MVSEG23 challenge, Auto3DSeg showcased its versatility by securing the top spot in segmenting mitral valve leaflets from 3D echocardiography volumes. See the leaderboard here: https://www.synapse.org/#!Synapse:syn51186045/wiki/622048.

Here are a few images from the segmentation challenges above:

Previous Challenge Achievements

Auto3DSeg’s track record includes:

• 1st Place in MICCAI 2022 challenge HECKTOR 2022 for head and neck tumor segmentation in PET/CT images.
• 2nd Place in MICCAI 2022 challenge INSTANCE22 for intracranial hemorrhage segmentation on Non-Contrast head CT (NCCT), ranking first in Dice score.
• 2nd Place in MICCAI 2022 challenge ISLES’22 for ischemic stroke lesion segmentation, ranking first in Dice score.

Learn More

For a comprehensive understanding of Auto3DSeg, check out our resources:

YouTube Walkthrough: Dive deeper into Auto3DSeg’s mechanics and advantages.

GitHub Tutorials: Explore detailed tutorials to unlock the full potential of this transformative tool.

By streamlining 3D medical image segmentation with MONAI Auto3DSeg, developers and researchers can make significant strides in medical diagnosis, treatment planning, and research.

Throughout this journey, MONAI has deepened its offering in radiology, expanded to pathology, and most recently, included support for streaming modalities starting with endoscopy. Now three years later, MONAI has over 600,000 downloads. It is used in over 450 GitHub projects, has been cited in over 150 published papers, and academic and industry leaders are using MONAI in their research and clinical workflows.

We’re excited to announce that MONAI is continuing to expand open-source healthcare AI innovation with v1.0. With a focus on providing a robust API that is designed for backward compatibility, this release ensures that you can integrate MONAI into your projects today and benefit from the stability of an industry-leading framework into the future.

Let’s look at the features included in the MONAI Core v1.0, MONAI Label v0.5 releases, and a new initiative called the MONAI Model Zoo.

MONAI Core v1.0

With the release of v1.0, MONAI Core focuses heavily on a robust and backward-compatible API design and also includes additional features like MetaTensors, a Federated Learning API, the MONAI Bundle Specification, and an Auto3D Segmetentation framework.

MetaTensor

The MetaTensor enhances the metadata-aware imaging processing pipeline by integrating both torch tensors and imaging meta-information. This combined information is essential for delivering clinically useful models, supporting image registration, and joining multiple models into a cohesive workflow.

MONAI MetaTensor Docs

MONAI Bundle

The MONAI Bundle is a self-contained model package with pre-trained weights and all associated metadata abstracted through JSON and YAML-based configurations. By focusing on ease of use and flexibility, you can directly override or customize these configs or utilize a hybrid programming model that supports config to Python Code abstraction.

MONAI Bundle Docs

Federated Learning

The MONAI Federated Learning module provides a base API that defines a MONAI Client App that can run on any federated learning platform. With the new federated learning APIs, you can utilize MONAI bundles and seamlessly extend them to the federated learning paradigm.

The first platform to support these new Federated Learning APIs is NVIDIA FLARE, the federated learning platform developed by NVIDIA. We welcome the integration of other federated learning toolkits to the MONAI Federated Learning APIs to help build a common foundation for collaborative learning in medical imaging.

MONAI and Federated Learning high-level workflow using the new MONAIAlgo FL APIs

MONAI Federated Learning Docs

NVIDIA FLARE + MONAI Example

Auto3D Segmentation

Auto3D is a low-code framework that allows data scientists and researchers of any skill level to train models that can quickly segment regions of interest in data from 3D imaging modalities like CT and MRI.

Developers can start with as little as 1–5 lines of code, resulting in a highly accurate segmentation model. By focusing on accuracy and including state-of-the-art models like Swin UNETR, DiNTS, and SegResNet, data scientists and researchers can utilize the latest and greatest algorithms to help maximize their productivity.

Auto3D Tutorial

Auto3D Segmentation Training and Inference workflow

MONAI Model Zoo

We’re excited to announce the MONAI Model Zoo, a hub for sharing pre-trained models that allow data scientists and clinical researchers to jump-start their AI development.

In the first release, there are 15 pre-trained models from MONAI partners, including King’s College London, Charité University, University of Warwick, Vanderbilt University, and Mayo Clinic.

These models utilize the MONAI Bundle specification, making it easy to get started in just a few commands. With the MONAI Bundle and Model Zoo, we hope to establish a common standard for reproducible research and collaboration, and we welcome everyone to contribute to this effort by submitting their pre-trained models for downstream tasks.

MONAI Model Zoo Landing Page

MONAI Label v0.5

MONAI Label now supports MONAI Core v1.0 and continues to evolve by improving overall performance, offering new models for radiology, and expanding into endoscopy with integration into the CVAT viewer for annotation and releasing new endoscopy models.

MONAI Label has been updated to support MONAI Core v1.0. For radiology, we’ve focused on improving overall performance and released a new vertebra model. For endoscopy, we’ve continued to improve CVAT integration and released three new models.

Radiology

In this release, MONAI Label focuses heavily on improving the overall performance of radiology applications.

By utilizing caching for pre-transforms in the case of repeated inference for interaction models, we were able to speed up the overall interactive loop. There is also support for DICOM Web API responses and fixes to the DICOM Proxy for WADO and QIDO.

Additionally, MONAI Label has released a new Multi-Stage Vertebra Segmentation model with three stages that can be used together or independently. The Vertebra model demonstrates the power of a multi-stage approach for segmenting several structures on a CT Image.

MONAI Label’s new Vertebra model segments several structures in CT images, shown running in 3D Slicer.

Endoscopy

MONAI Label now supports 2D segmentation for endoscopy. By continuing to expand on the previous CVAT integration, MONAI Label has integrated active learning into CVATs automated workflow.

Three new models are being released including Tool Tracking Segmentation, InBody vs. OutBody De-identification classification model, and DeepEdit for interactive tool annotation.

MONAI Label Endoscopy Sample Applications

Conclusion

This is a momentous milestone for Project MONAI and we are looking forward to further serving the medical imaging community. We want to hear your feedback! Connect with us on Slack and GitHub. Please share your successes and report any issues you might have with MONAI.

Interested in joining the MONAI Community? Get started on our MONAI YouTube Channel, where we have tutorials, archived bootcamps, and walkthrough guides.

Stay tuned for the latest news on our hosted events! Whether you’re new to MONAI or already integrating MONAI into your workflow, the MONAI Website and Twitter account are the best places to stay up to date!

MONAI focuses on two main topics for these releases, MONAI for Pathology and MONAI Bundles.

First, MONAI is helping create a starting point for Pathologists, Data Scientists, and Researchers who want to get started using Deep Learning in their Pathology workflow. Currently, Deep Learning is not typically used in Pathology for various reasons, including lack of integration into the typical workflow and general performance limitations.

Project MONAI is helping address these issues with transformers, data loaders, and network architectures in MONAI Core and integration into Pathology Viewers to help with AI-Assisted Annotation with MONAI Label.

Next, the MONAI Bundle format defines a portable and standardized packaging format for storing and sharing models. By creating a standard layout and supporting MONAI Bundles throughout each framework, you’ll be able to seamlessly utilize MONAI bundles from annotation, training, and deployment.

MONAI Core v0.9

Highlights include:

• MONAI Bundle
• Object Detection in Medical Images
• SWIN Transformers and SWIN UNETR Architecture
• DeepEdit and NuClick for Pathology
• MetaTensor API Preview

MONAI Bundles include all the information necessary for a model development life cycle, including training, fine-tuning, and inference. The MONAI Bundle API is defined within the `monai.bundle` module namespace and is an easy-to-use API that helps separate the deep learning hyperparameter settings from code and helps to decouple the component details from the higher-level learning paradigms.

This release also contains the essential components for object localization and categorization workflows. These modules include handling 2D and 3D bounding boxes, network blocks and architectures of RetinaNet, and common utility modules.

This release has included or updated three new networks, including SWIN UNETR, DeepEdit, and NuClick. SWIN UNETR is a state-of-the-art transformer model inspired by the previous success of the UNETR model included in a prior release of MONAI Core. And NuClick is a CNN-based framework for interactive segmentation of objects in histology images and is used as a sample application with MONAI Label.

Last, a preview release of the MetaTensor API is being introduced. The MetaTensor is a refactoring of the data representation in MONAI to include the metadata associated with the primary imaging modalities and is essential in many biomedical imaging applications.

For full release notes, check out the MONAI Core release notes.

MONAI Label v0.4

Highlights include:

• Pathology Sample Applications for Segmentation Nuclei, DeepEdit Nuclei, and NuClick
• DSA, QuPath, and CVAT Integration
• GraphCut and GMM-based methods for Scribbles

Three new Pathology based sample applications that can be used as an easy starting point or as the basis for your custom annotation application include Segmentation Nuclei, DeepEdit Nuclei, and NuClick.

MONAI Label now integrates with three pathology viewers, including QuPath, Digital Slide Archive or DSA, and CVAT. These allow you to quickly choose your favorite viewer and start using MONAI Label with it today.

If you’re interested in contributing feedback to the direction of our Pathology work in MONAI, we’ve created a survey to help understand the community’s needs.

For full release notes, check out the MONAI Label release notes.

MONAI Deploy v0.4

Highlights include:

• MONAI Bundle inference operator
• Multi-Model support

With the introduction of MONAI Bundles through all of the Project MONAI frameworks, MONAI Deploy App SDK is making it easy to take those MONAI Bundles and quickly create your own AI Application. MONAI Bundle Operator enables using a MONAI-trained model in an inference app with minimal or no coding. The Bundle inference operator is enhanced to support both in-memory and file I/O for edge cases where app developers/model trainers only need to test the inference logic with the single operator app.

We’ve also tested and confirmed multi-model support. Each bundle operator is designed to support a single model bundle, but the application can have multiple model bundles named uniquely.

For full release notes, check out the MONAI Deploy release notes.

With these new releases for Project MONAI, MONAI continues to expand into new fields like Pathology while also making it easier to create reproducible and standardized ways to share Medical AI Models. These efforts continue to create an environment for data scientists, researchers, clinicians, and pathologists to integrate deep learning into their workflows.

Join the MONAI Community on GitHub or Slack.

Viewer Integration

MONAI Label now integrates with three pathology viewers, including QuPath, Digital Slide Archive or DSA, and CVAT. You can choose your favorite viewer and quickly start labeling pathology images today.

Sample Applications

MONAI Label also provides new Pathology based sample applications that can be used as an easy starting point or as the basis for your custom annotation application.

These three applications include Segmentation Nuclei, DeepEdit Nuclei, and NuClick.

Segmentation Nuclei gives a reasonable basis for multi-label segmentation, including labels for Neoplastic Cells, Inflammatory, Connective and Soft Tissue cells, Dead cells, and Epithelial; however, it doesn’t provide the ability to do interactive segmentation.

DeepEdit Nuclei combines both Interactive and Auto Segmentation that combines all of the labels from the standard segmentation model into a single Nuclei label.

Last is an implementation of NuClick, a Unet-based approach to speed up collecting annotations for microscopic objects requiring minimum interaction from the annotator.

Performance

One of the most significant pain points in the Pathology workflow is the performance of loading images and performing inference. MONAI Label focuses on performance throughout the Pathology Workflow, improving time to inference on patches and whole slide images and offering the ability to use RAPIDS cuCIM to speed up the loading of images.

Inference Performance

For performance on Patches and Whole Slide Images, both NuClick and DeepEdit perform significantly faster than typical Machine Learning Nuclei Detection, with DeepEdit performing up to six times faster than the standard NucleiDetection algorithm, as shown below. We measured performance with varying amounts of nuclei in the image, and the results show up to a 6x speedup in inference. This performance increase can mean the difference between 3 minutes for GPU-based inference or 20 minutes for CPU-based inference.

RAPIDS cuCIM

A significant portion of a typical deep learning pipeline is spent on I/O. For Pathology, this is the case since images are typically extremely large, and the process of loading and encoding these images can become a significant bottleneck. That’s why MONAI has integrated RAPIDS cuCIM as one of its optional image loaders.

Below you’ll see the performance increase of up to 6x depending on the number of threads being used for loading an image compared to an alternative library like OpenSlide.

Pathology Adoption of Deep Learning

MONAI Label is creating a starting point for Pathologists and Data scientists to work together and utilize the benefits of Deep Learning. By enabling a workflow that integrates directly into a Pathologist viewer and allowing for continuous learning, MONAI aims to accelerate the adoption of Deep Learning in Pathology.

“MONAI Label will enable pathologists and scientists to build accurate models without knowing anything about AI. This is an important step in making AI a universal tool for research.”

— Lee A.D. Cooper, PhD — Associate Professor of Pathology, Director, Computational Pathology, Director, Center for Computational Imaging and Signal Analytics, @ Northwestern University Feinberg School of Medicine

If you’re interested in contributing feedback to the direction of our Pathology work in MONAI, we’ve created a survey to help understand the community’s needs.

Citations:

Diaz-Pinto, Andres & Alle, Sachidanand & Ihsani, Alvin & Asad, Muhammad & Nath, Vishwesh & Pérez-García, Fernando & Mehta, Pritesh & Li, Wenqi & Roth, Holger & Vercauteren, Tom & Xu, Daguang & Dogra, Prerna & Ourselin, Sebastien & Feng, Andrew & Cardoso, Manuel Jorge. (2022). MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images.

Navid Alemi, Mostafa Jahanifar, et al. “NuClick: a deep learning framework for interactive segmentation of microscopic images.” Medical Image Analysis 65 (2020): 101771.

Jahanifar, Mostafa, Navid Alemi Koohbanani, and Nasir Rajpoot. “Nuclick: From clicks in the nuclei to nuclear boundaries.” arXiv preprint arXiv:1909.03253 (2019).

MONAI has released three new versions of its existing tools and is introducing a new tool to its deployment offering, introducing MONAI Deploy Inference Server. Researchers can quickly deploy and run MONAI Application Packages (MAPs) as scalable inference on their data using an existing Kubernetes cluster as cloud-native microservices.

MONAI Core v0.8

The first new release is MONAI Core v0.8 which expands on the available learning methods, including Self-Supervised and Multi-Instance learning support. It also includes a new AutoML technique called Differentiable Network Topology Search, or DiNTS, and new visualization techniques for the various transforms already available in MONAI.

Self-Supervised allows us to utilize unlabeled data by generating pre-trained weights using the unlabeled data with self-supervised tasks based on different augmentation types. MONAI Core now provides an example tutorial using the TCIA-Covid19 to generate the pre-trained weights. Those weights are then used with the Beyond the Cranial Vault (BTCV) dataset as the fine-tuning dataset.

Multi-Instance Learning (MIL) is a supervised learning technique that uses bags of labeled data versus individually labeled data. MIL is a crucial algorithm for classifying whole slide images (WSI), which can have billions of pixels and requires extraordinary computational and annotation resources. MONAI now includes a new Network Architecture called MILModel, which provides three Multi-Instance Learning modes — mean, max, and attention-based methods. These attention-based methods are based on state-of-the-art research that helps account for dependencies in Deep Learning-based Multiple Instance Learning (https://arxiv.org/abs/2111.01556)

DiNTS Neural Architecture Search has been applied to search high-performance networks for medical image segmentation. The DiNTS method addresses some of the common issues with large-scale 3D image datasets, like flex multi-path network topologies, high-search efficiency, and GPU memory usage. You can find example notebooks using DiNTS and the Medical Segmentation Decathlon (MSD) datasets to achieve state-of-the-art performance.

MONAI Core v0.8 release includes a transformer visualization notebook for existing MONAI transforms, including visualizing images with matplotlib based on MONAI matshow3d API, with TensorBoard-based MONAI plot_2d_or_3d_image API, and with ITKWidgets. It also includes how to blend two images with the same shape.

MONAI Label v0.3

The second new release is MONAI Label v0.3 which includes multi-label segmentation for existing applications, increased performance by supporting multi-GPU training, and better Active Learning User Experience.

First, Multi-label segmentation support includes updating the existing DeepEdit and DeepGrow networks provided by MONAI Label. It now consists of an upgraded UI interface to support multi-label tasks, modifying training scripts to work with multi-label tasks, and creating a robust naming and error system for label name and number associations.

Next, to help increase the performance of the MONAI Label training loop, multi-GPU support has been added to the existing workflows, including updating the data loader. You’re now able to indicate how many GPUs you want to use during the training process.

Last, user experience is an integral part of the training process. To help enable a simple and more intuitive experience for Active Learning, we’ve provided options to allow you to train specific models and allow users to skip images if they feel the current image selection isn’t a good one,

MONAI Deploy

The third new release is MONAI Deploy App SDK v0.2, which includes two new base operators for DICOM interactions: one for DICOM Series Selection and another exporting DICOM Structured Reports SOP for classification results.

Expanding on the MONAI Deploy offerings, a new component called MONAI Inference Service (MIS) has been released. This tool allows researchers to build and deploy a scalable inference server for their data using an existing Kubernetes cluster.

Highlights Include:

• Register a MAP in the Helm Charts of MIS.
• Upload inputs via a REST API request and make them available to the MAP container.
• Provision resources for the MAP container.
• Provide outputs of the MAP container to the client who made the request.

We’ve also included new MONAI Deploy tutorials that walk you through creating a MAP, deploying MIS, and pushing your MAP to MIS to be run in a Kubernetes cluster.

MONAI Deploy Tutorials: Web-based or Jupyter Notebooks

MONAI continues to expand its core capabilities throughout the Medical AI workflow. Get started today by checking out the notebooks mentioned throughout the sections above, or head over to our GitHub repos to start contributing today!

MONAI v0.6 and MONAI Label v0.1 are now available! MONAI Label is an intelligent image labeling and learning tool that enables users to create annotated datasets and build AI annotation models quickly.

We’re excited to announce our latest release of MONAI, version 0.6. We continue to expand on our APIs by including a new network called UNETR implemented in PyTorch. We’re also adding new functionality to use existing pre-trained PyTorch models created for NVIDIA Clara Train.

Alongside our core release, we’re releasing a new project that has officially hit version 0.1, called MONAI Label. MONAI Label is an intelligent open-source image labeling and learning tool that reduces the time and effort of annotating new datasets and enables the adaptation of AI to the task at hand by continuously learning from user interactions and data. We’re providing sample applications that use some of our existing PyTorch models to help you get started quickly.

MONAI v0.6

UNETR: Transformers for Medical Image Segmentation

UNETR is a transformer-based model for volumetric (3D) medical image segmentation and is currently the state-of-the-art model on the BTCV dataset leaderboard for the task of multi-organ semantic segmentation. UNETR has a flexible implementation that supports various segmentation tasks.

You can find a tutorial for 3D multi-organ semantic segmentation using UNETR in our tutorials repo.

Decollate Batches

We’ve included the ability to decollate batches to simplify post-processing transforms and enable flexible operations on a batch of model outputs. By building off previous work on inverse spatial transforms, decollate is an “inverse” operation of the PyTorch collate function. Decollate enables post-processing transforms for each item independently and allows for randomized transforms to be applied for each predicted item in a batch. This also provides inverse operations for data items in different original shapes since the inverted items are returned in lists instead of tensors.

A typical process of decollate batch is illustrated as follows:

You can find a Jupyter notebook tutorial showing a typical decollate workflow here.

Medical Model ARchive (MMAR) support

MONAI now includes Pythonic support for the Medical Model ARchive, MMAR, format provided by NVIDIA Clara Train. By enabling support for this format, developers can now use pre-trained models created for Clara Train directly in MONAI.

Find all of the Clara Train models on NGC here. We’ve also included a new tutorial showing you how to use one of the pre-trained MMAR models for transfer learning. The results are shown below:

Metric Enhancement

The base API for metrics has been enhanced to support the logic for iteration and epoch-based metrics. By enabling support for both metric methods, MONAI metrics are now more extensible and are a great starting point for creating custom metrics. The APIs also support data-parallel computation; with the Cumulative base class, intermediate metric outcomes can be automatically buffered, cumulated, synced across distributed processes, and aggregated for the final results.

We’ve included a multi-processing computation example that shows how to compute metrics based on saved predictions and labels in a multi-processing environment.

CUDA extension

MONAI continues to accelerate domain-specific routines in common workflows by introducing C++/CUDA modules as extensions of the PyTorch native implementations. We now provide two ways to build a C++ extension from PyTorch:

• Via `setuptools` for modules including `Resampler`, `Conditional random field (CRF)`, and `Fast bilateral filtering using the permutohedral lattice`.
• Via just-in-time (JIT) compilation for Gaussian mixtures module. Using JIT compilation allows for dynamic optimization according to the user-specified parameters and local system environment.

Backward compatibility and enhanced CI/CD

As we move closer to the MONAI 1.0 release, we’re focusing on creating the proper mechanisms to provide a fast and collaborative codebase development.

As a starting point, we’ve created some basic policies for backward compatibility. New utilities are introduced on top of the existing semantic versioning modules and the git branching model. We’re also working on a complete CI/CD solution that is efficient, scalable, and secure.

MONAI Label v0.1

MONAI Label is a server-client system that facilitates interactive medical image annotation by using AI. As a part of Project MONAI, MONAI Label shares the same principles as MONAI and focuses on being Pythonic, modular, user friendly, and extensible.

Open-source and easy-to-install, MONAI Label can run locally on a single machine with one or multiple GPUs. The server and client run on the same machine and don’t currently support multiple users or communication with an external database.

To quickly install and run MONAI Label using DeepEdit, it’s as easy as following the steps below:

$ pip install monailabel

$ monailabel datasets --download --name Task02_Heart --output C:\Workspace\Datasets

$ monailabel apps --download --name deepedit_left_atrium --output C:\Workspace\Apps

$ monailabel start_server --app C:\Workspace\Apps\deepedit_left_atrium --studies C:\Workspace\Datasets\Task02_Heart\imagesTr

Once you start the MONAI Label Server, by default, it will be serving at http://127.0.0.1:8000/. Opening the URL in the browser will provide you with the list of Rest APIs available. You can also use the 3D Slicer extension by filling in the MONAI Label Server field with the serving URL.

Who should use MONAI Label?

MONAI Label focuses on two types of users, Researchers and Clinicians.

For Researchers, MONAI Label gives you an easy way to define their pipeline to facilitate the image annotation process. They can use the provided Slicer MONAI Label plugin or customize their own workflow to process inputs and outputs sent to the App.

For Clinicians, MONAI Label gives you access to a continuously learning AI that will better understand what the end-user is trying to annotate.

MONAI Label comprises the following key components: MONAI Label Server, MONAI Label Sample Apps, MONAI Label Sample Datasets, and a 3DSlicer Viewer extension.

MONAI Label Server

The MONAI Label server is the main integration point. It creates the REST API that allows communication between the MONAI Label server and the client (e.g., Slicer plugin, OHIF plugin, etc.).

Sample Apps

The included sample apps for MONAI Label are the following:

• Left-atrium semantic segmentation in the heart using both DeepEdit and DeepGrow.
• Spleen semantic segmentation using both DeepEdit and DeepGrow.
• Multiple label segmentation (e.g., heart ventricles segmentation and liver and tumor)

These sample applications showcase the speed-up provided when using MONAI Label when used to create your annotation model. For example, using the Spleen Application, you can quickly begin to utilize your annotation model after only a few images are segmented.

The figure below compares the “Interactive” vs. “Standard” way of annotating and training.

In the “Interactive” way, the annotation and model training complement each other. The user can assist, in the form of clicks, to guide the AI model to annotate the object of interest better. This method allows you to quickly start using your annotation model, giving you a significant speedup in your overall annotation process.

While the “Standard” way is where the user uses a classical technique of using the paintbrush or click-based contours to annotate the image. This method requires annotating all images before training, which means a longer time before you can begin to utilize your model.

Sample Datasets

MONAI Label uses the Medical Segmentation Decathlon datasets to showcase how easy it is to create MONAI Label Apps using the three different paradigms: DeepGrow, DeepEdit, and automatic segmentation

Annotation Paradigms

MONAI Label currently employs the following annotation algorithms:

• DeepGrow is a click-based interactive segmentation model, where the user can guide the segmentation with positive and negative clicks. Positive clicks guide the segmentation towards the region of interest, while negative clicks guide the model away from the over-segmented areas.
• DeepEdit is an algorithm that combines the power of two models in one single architecture. It allows the user to perform inference using a standard segmentation method and interactive segmentation using clicks.
• Automatic Segmentation is the non-interactive paradigm available in MONAI Label. It allows the researcher to create a segmentation pipeline using a standard UNet or any network available in MONAI (e.g., UNet, Highresnet, ResNet, DynUnet, etc.) to segment images automatically.

3DSlicer Extension

The 3D Slicer extension handles calls and events created by user interaction and sends them to the MONAI Label server. The current version supports click-based interaction and allows the user to upload images and labels.

The MONAI Label server also supports other interaction styles such as closed curves and ROI. A researcher can modify this plugin to make it more dynamic or customized to their MONAI Label Apps.

Summary

We’re excited as we continue expanding on our portfolio of projects that are a part of Project MONAI. We also have two new working groups focused on Deployment and Digital Pathology, so keep an eye out for more releases later this year, including a prototype that focuses on the end-to-end Medical AI lifecycle.

MONAI v0.6 and MONAI Label v0.1 was originally published in PyTorch on Medium, where people are continuing the conversation by highlighting and responding to this story.