🩺 Introduction

AI in healthcare is advancing quickly, but adapting large models to medical reasoning often requires huge compute budgets. To explore what’s possible on a shoestring, I ran a lite fine-tuning experiment on MedGemma-4B-IT — a vision-language model adapted to medicine — using less than $5 of compute.

This was not a full training run: I trained for only 100 steps and evaluated on just 100 samples. Still, the model reached ~83% accuracy while producing step-by-step reasoning outputs. This proof-of-concept shows how lightweight experiments can help us test ideas before scaling up.

🔍 What is MedGemma?

MedGemma is an open-source extension of Google’s Gemma model for healthcare tasks. Like Gemma, it’s a vision-language model, meaning it can process both:

• Images (e.g., chest X-rays)
• Text (clinical questions, findings, reports)

This makes it suitable for medical VQA (Visual Question Answering) and diagnostic reasoning.

• 📄 Reference:
• Gemma: Open Models Based on Gemini Research and Technology (Google DeepMind, 2024)
• MedGemma (open-sourced adaptation on Hugging Face: unsloth/medgemma-4b-it)

📊 The Datasets: REXGradient-160K & REXVQA

REXGradient-160K

A large-scale chest X-ray dataset released by Stanford’s Rajpurkar Lab and Gradient Health:

• 160,000 studies from 109K patients across 79 clinical sites
• Includes 273,000 images and paired radiology reports
• Designed for robust multimodal AI research
• 📄 Reference:
• ReXGradient-160K: A Large-Scale Multi-Institutional Chest X-ray Dataset to Accelerate Medical AI Research (Rajpurkar Lab & Gradient Health, 2025)

REXVQA

A benchmark built on top of REXGradient-160K to evaluate visual question answering in radiology:

• ~653,000 multiple-choice questions paired with X-rays
• Covers 5 reasoning skills: presence, location, negation, diagnosis, geometry
• Generated from reports using radiologist-informed GPT-4o prompts, with strong clinical vetting
• 📄 Reference:
• ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding (Rajpurkar Lab, 2025)

⚙️ Lite Fine-Tuning Setup

Using Unsloth + Hugging Face TRL, I fine-tuned MedGemma-4B-IT:

• LoRA adapters → only ~0.89% of parameters (~38M) updated
• 4-bit precision → single GPU, memory-efficient
• Prompting → structured as Findings → Impression → Solution for interpretability
• Training → 100 steps (~15 minutes runtime) on A40 40 gb vram
• Evaluation → 100 test samples from REXVQA

💸 Total cost: under $5.

📈 Lite Results

• Accuracy: ~83% on the 100-sample subset
• Outputs: Clear reasoning steps before answers, e.g.:

<start_working_out>
Findings: The heart size remains normal. Stable bibasilar scarring noted.
Impression: Stable scarring without acute changes.
<end_working_out>

<SOLUTION>
C - Stable bibasilar scarring
</SOLUTION>

⚠️ Important: These are preliminary results — not full benchmarks. The small evaluation is meant to illustrate feasibility, not publishable accuracy.

🚀 Why This Matters

• Low barrier to entry → Quick, cheap experiments can validate ideas before large-scale training.
• Explainability → Structured reasoning improves trust in clinical contexts.
• Open resources → Anyone can reproduce or extend this work.

📂 Resources

• 📝 Notebook: GitHub — TuneCraft: medgemma-4b-it REXVQA SFT Lite
• 🤗 Model Checkpoint: Hugging Face — SerdarHelli/medgemma-4b-it_rexvqa_sft
• 📊 Datasets:
• REXGradient-160K
• REXVQA

💡 Final Thoughts

This was a lite experiment — 100 steps of training, 100 samples of evaluation, <$5 of compute. Even with such a small setup, MedGemma-4B-IT showed encouraging reasoning ability.

Scaling this to full training and evaluation could unlock powerful, explainable medical AI models. But the takeaway here is simple: affordable, domain-specific AI is possible today.

Hello Everyone,

Today, I will develop a custom Facial Emotion Recognition CNN model using ModelArts and Huawei OBS. We will load our data to our bucket using Huwaei OBS. Then, we will train a CNN model on ModelArts. Finally, We will build an AI real-time service as an API.

Introduction

What is Facial Emotion Recognition?

Facial Emotion Recognition is a technology that analyzes emotions from many sources, including images and videos. It is a member of the group of technologies known as “affective computing,” a multidisciplinary area of study on the capacity of computers to recognize and understand affective states and human emotions that frequently relies on Artificial Intelligence.

In this study, we will train a CNN model, which will be VGG19, with custom hyperparameters to recognize facial emotion. Also, we will use the FER2013 dataset to train our model. FER2013 contains approximately 30,000 facial RGB images of different expressions with seven labels. FER2013 is a well-studied dataset and has been used in ICML competitions and several research papers. It is one of the more challenging datasets, with human-level accuracy only at 65±5%. After training our model, we will build a real-time service. [1]

What are the ModelArts?

A one-stop shop for AI development, ModelArts is designed for programmers and data scientists of all levels. You can manage full-lifecycle AI workflows and quickly design, train, and deploy models from the cloud to the edge. With important capabilities, including data preparation and auto labeling, distributed training, automated model creation, and one-click workflow execution, ModelArts supports AI creativity and speeds up AI development.

All phases of AI development are covered by ModelArts, including data processing, model training, and model deployment. ModelArts’ core technologies enable a variety of heterogeneous computer resources, giving developers the freedom to choose and employ resources as needed. TensorFlow, MXNet, and PyTorch are just a few of the well-known open-source AI development frameworks that are supported by ModelArts. Additionally, ModelArts enables you to apply personalized algorithm frameworks that are suited to yours. For More Information:

What Is ModelArts?_ModelArts_Huawei Cloud

A Step-by-Step Implementation

1. Upload Data To OBS: First, we need to upload our data to our OBS bucket to train our custom model. There are many various options to upload data. But I would choose to use OBS utils. I think it is more manageable.

· In the first step, we must create an Access key to Access OBS utils.

· Then, we will create a bucket to store our data.

· Finally, We will upload our FER data using OBS Utils.

2. Training a Custom Model Using ModelArts: After we upload our data, we can start to train our custom model using ModelArts.

· Firstly, We need to write our training code.

· Then, we will create a requirement text file to set up the necessary libraries.

· In this step, we will upload our codes and file to the created bucket.

· In the final step, we will train our model using ModelArts Training Jobs.

3. Building Real-Time Service Using ModelArts: Now, we can build a real-time service with our trained model.

· We need to write our inference code to configure and develop the service.

· Finally, we will build the API service for Facial Emotion Recognition using ModelArts.

1. Upload Data To OBS

Creating an Access Key

First of all, we need an access key and secret key to access using development tools, including APIs, CLI, and SDKs. On the management console, hover over the username in the upper right corner and choose My Credentials from the drop-down list.

Then, we should choose Access Keys from the navigation pane. Now, we can create an Access key.

We will click Create Access Key and enter the verification code or password. It is essential that we download the access key file and keep it properly. If the download page is closed, we will not be able to download the access key. However, we can create a new one.

We can get our access key, secret key, and access key id quickly from the CSV file which we downloaded previously.

Creating a Bucket

Now, we will use Huawei OBS (Object Storage Service). We create a bucket and load our data into it. Object Storage Service (OBS) is a cloud storage service optimized for storing massive amounts of data. It provides unlimited, secure, and highly reliable storage capabilities at a relatively low cost. On the management panel, click over the search field in the upper right corner and write OBS, then choose it from the drop-down list.

After clicking Create Bucket button, we will see a panel as follows. In this part, we should remember which region we choose because once a bucket is created, the region cannot be changed, and also, we will also choose the same region on the ModelArts part to train our model. I will select the AP-Singapore region and give a name to it as ferdatahw.

Now, we created a bucket. Let’s click over our bucket and create the necessary folders we will use in the training process and building service. On the left navigation panel, let’s click over Objects.

In this section, we will create three folders. One of them will be a folder we will use to obtain data from there. Another one will be a folder we will use our model’s and API’s code. The final one will be a folder we will save our own model into it. I created three folders, giving names to their data, code, and Out in order.

Uploading Data Using OBS Utils

As you can remember, in the first stage, we created an access key. Now, we will load our data into the bucket we created a few minutes ago. But after the start, we should download data into our local to upload data into our bucket. As you know, we will develop the Facial Emotion Recognition AI model in this study. I would love to use the FER2013 dataset because it has a lot of images, and its size is not huge. With this link, you can download it. After downloading the dataset, let’s extract the RAR file to your chosen local place.

Huwaei OBS tools offer many tools to manage your bucket. For example, you can upload your files manually or use the OBS Utils and OBS Web Browser, etc. In this part, I will use the OBS Utils; it is effortless to use. Let’s download OBSUtil from the Huwaei OBS console.

The OBS Utils is a command line tool for accessing and managing OBS on HUAWEI CLOUD. This tool can perform common configurations on OBS, such as creating buckets, uploading and downloading files/folders, and deleting files/folders. If you are familiar with the command line interface (CLI), the OBSUtil is recommended for batch processing and automated tasks. More information and documentation :

Introduction to obsutil_Object Storage Service_Tools Guide_obsutil_HUAWEI CLOUD

After downloading the OBS Utils, you will get a RAR file. Then, let’s open obsutil.exe. A command console will be opened. We must enter our access id, access key, and region path to access OBS. Let’s enter this code line onto the OBSUtil.

obsutil config -i=your_access_id -k=your_access_key -e=obs.ap-southeast-3.myhuaweicloud.com

If your connection is successful, you will get a response as “Update config file successfully!”. We connected to the OBS. Now we transfer our data to the bucket which we created previously. Let’s upload our data.

obsutil cp pathdirectory_yourextracteddata_fer2013 obs://ferdatahw/data/ -f -r

For Example :

obsutil cp C:/Users/pc/Desktop/val obs://ferdatahw/data/ -f -r

obsutil cp C:/Users/pc/Desktop/train obs://ferdatahw/data/ -f -r

You can check out was the transfer successful or not on the OBS console. In the training process, we upload our code into our bucket manually because our code will consist of a few files and will be easy to upload and change files manually.

2. Training a Custom Model Using ModelArts

Custom Training Code

Let’s write our main training code. When a ModelArts model reads data stored in OBS or outputs data to a specified OBS path, perform the following operations to configure the input and output data. We should parse the input and output paths in the training code. Also, we can parse hyperparameters in the training code. As you can see in this figure, we will select the OBS path or dataset path as the training input and the OBS path as the output on the modelarts.

Let’s Code!

Creating Requirements Text File

We will create a file named pip-requirements.txt in the code directory, and specify the name and version number of the dependency package in the file. Before the training boot file is executed, the system automatically runs the following command to install the specified Python packages:

imutils==0.5.4

numpy==1.21.6

keras>=2.1

argparse==1.1

Uploading Training Code And Requirement Text File

Finally, we will upload our training boot file and requirement.txt into our code directory, which we created previous OBS stage. Let’s upload our files manually. Because we have two files and we don’t need any tools such as the OBS Utils to upload.

Training Using ModelArts Training Jobs

Let’s open ModelArts console. In this study, we don’t use the ExeML tool, ModelArts SDK, or others. We will do custom training using only Tensorflow. However, there are many different paths you can follow. For More Information:

Progressive Knowledge_ModelArts_Huawei Cloud

On the left panel, let’s go to Training Management and then choose Trainin Jobs from the drop-down list. Now, we will create a training job. Now, you will see a panel. Let’s examine this part by part. For More Information:

Creating a Training Job_ModelArts_User Guide (Senior AI Engineers)_Training Management (New Version)_Performing a Training_HUAWEI CLOUD

In the first area, we can give a name and description of our training job.

In the second part, there are many fields. In the first field, “Created By,” we will choose Custom Algorithms because, as you know, we would love to do custom training. In the second field of this part, we will choose Present Images. Then, as you can see, we will choose TensorFlow 2.1 because we will use Tensorflow Framework to train our model in this study. In the “Code Directory” field, we will choose the folder which we created previously on the OBS stage. And again, in the boot file, we will select our main training code file.

In this part, we must enter to parameters training_url and data_url. These are a path where we will save our model and a path where from we will obtain our dataset, respectively. Also, We can enter Hyperparameters, and Environment Variables to pass to our code :

In the final part, we can configure what we want. I preferred GPU to train.

Now, it is ready to submit. If you go to your training job, you will see the logs of your code.

We got a 0.65 accuracy score on FER2013 data. We can check whether our model is saved or not in our obs path.

3. Building Real-Time Service using ModelArts

Inference Code

Now, We will develop an AI application as a web service. First, our model requires inference code; ensure that the code is stored in the model directory, which is Out; we saved our model there before. The file name is fixed to customize_service.py. There must be one and only one such file. Our inference code must be inherited from the BaseService class. The following table lists the import statements of different types of model parent classes. For More Information:

Specifications for Compiling Model Inference Code_ModelArts_User Guide (Senior AI Engineers)_Model Package Specifications_HUAWEI CLOUD

Let’s code

Finally, We will upload the inference code file into our directory, where we saved our model. Let’s upload our files manually. Then, we will be ready.

Building AI Application and a Real-Time Service

Now, let’s go back to the ModelArts console. On the left management console, click over the AI Application Management and choose AI Applications from the drop-down list. Then, we will create an AI application.

We will choose OBS on the panel because we saved our model into a bucket and did custom training. Also, in the previous stage, we uploaded the inference code of our model, as you remember. We will select Tensorflow as AI Engine and choose the version of it that we used at the training.

In the final stage, we will build our real-time service. It is the easiest part. On the left management panel, click over the AI Application Management and choose AI Applications from the drop-down list. Then, We will create an AI application.

We should select the AI application which we built. That’s it. 😊 Cheers!

Summary

To summarize, this is how you can easily generate your own real-time API service for Facial Emotion Recognition with ModelArts.

We trained a CNN model based on FER2013 dataset with custom hyperparameters, reaching a 65% accuracy score on the validation dataset. Then, We built a real-time service using the model. In an addition, if you get a higher score, you can try data augmentation techniques, adding new datasets, or using transfer learning. So, this study showed that using Huawei ModelArts tools is so easy to build an AI real-time service, and train a model.

References

Goodfellow, Ian J., et al. “Challenges in representation learning: A report on three machine learning contests.” International conference on neural information processing. Springer, Berlin, Heidelberg, 2013.

• What Is ModelArts?_ModelArts_Huawei Cloud
• FER-2013
• Introduction to obsutil_Object Storage Service_Tools Guide_obsutil_HUAWEI CLOUD
• Progressive Knowledge_ModelArts_Huawei Cloud
• Creating a Training Job_ModelArts_User Guide (Senior AI Engineers)_Training Management (New Version)_Performing a Training_HUAWEI CLOUD
• Specifications for Compiling Model Inference Code_ModelArts_User Guide (Senior AI Engineers)_Model Package Specifications_HUAWEI CLOUD

👨‍💻 Facial Emotion Recognition Using ModelArts was originally published in Huawei Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this study, the purpose is the classification of thyroid tumors on ultrasound images with 6 different categories:

• 1 (Benign) • 2 (Benign)

• 4a (Malign) • 4b (Malign)

• 4c (Malign) • 5 (Malign)

For this end, we will develop a deep learning algorithm using ultrasound images labeled and evaluate the performance of the algorithm.

Colombia National University presented an open access database of thyroid ultrasound images. The dataset consists of a set of B-mode Ultrasound images, including a complete annotation and diagnostic description of suspicious thyroid lesions by expert radiologists. [1]

Firstly, we will import the libraries we will use.

import os
import xml.etree.ElementTree as ET
from natsort import natsorted
import pandas as pd
from PIL import Image
import numpy as np
import requests
from zipfile import ZipFile
from io import BytesIO
import cv2
import matplotlib.pyplot as plt
import tensorflow as tf
import math
import random
from six.moves import xrange
import collections
import string

Then , we will download and prepare the data.

def download_dataset(save_path):
    r = requests.get("http://cimalab.intec.co/applications/thyroid/thyroid.zip")
    print("Downloading...")
    z = ZipFile(BytesIO(r.content))    
    z.extractall(save_path)
    print("Completed...")

# XML and Jpeg     
def to_dataframe(path):
    dirs=natsorted(os.listdir(path))
    xml_list=[]
    img_list=[]
    for i in range(len(dirs)):
        if '.xml' in dirs[i]:
            xml_list.append(dirs[i])
        if not '.xml'  in dirs[i]:
            img_list.append(dirs[i])
    xml_list=natsorted(xml_list)
    img_list=natsorted(img_list)
    tirads=[]
    for j in range(len(xml_list)):
        tree = ET.parse(path+'/'+xml_list[j])
        a=tree.findall("./tirads")
        if a[-1].text!=None:
            case=[xml_list[j],a[-1].text]
            tirads.append(case)
    data=[]
    for k in range(len(tirads)):
        xml=tirads[k][0][:-4]
        for z in range(len(img_list)):
            if xml+'_1.jpg'==img_list[z] or xml+'_2.jpg'==img_list[z] or xml+'_3.jpg'==img_list[z]:
                m=[img_list[z],tirads[k][1]]
                data.append(m)

    df = pd.DataFrame(data,columns =['Jpeg_Name', 'Tirads'])
    return df

The dataset is imbalanced, and several images contain thyroid cancers that are not labeled. In addition, some images have two thyroid ultrasound images from the same subject. Before algorithm training, images were processed. In data preperation ,firstly , we normalized images . Secondly, we cropped images to find to the biggest coutour of image.Also, there are text about classification of thyroid tumors on images. They were deleted because of to prevent the bias.

The Code :

#Cropp Function
def croping(img,x, y, w, h):
    if abs(w)<abs(h):
        img2=np.zeros([h,h])
        img2[:,h-w:h]=img[y:y+h, x:x+w]
    if abs(h)<abs(w):  
        img2=np.zeros([w,w])
        img2[w-h:w,:]=img[y:y+h, x:x+w]
    else:
        return img
    return img2

def convert_one_channel(img):
    #if some images have 3 channels , although they are grayscale image
    if len(img.shape)>2:
        img=img[:,:,0]
        return img
    else:
        return img

#Remove Fill area from Image and Resizeing
def crop_resize(path,resize_shape):
    img=plt.imread(path)
    img=convert_one_channel(np.asarray(img))    
    kernel =( np.ones((5,5), dtype=np.float32))
    ret,thresh = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY)
    thresh = thresh.astype(np.uint8)
    a1,b1=thresh.shape
    thresh=cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel,iterations=3 )
    thresh=cv2.erode(thresh,kernel,iterations =5)
    contours, hierarchy = cv2.findContours(thresh.copy(), cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    c_area=np.zeros([len(contours)])
    for i in range(len(contours)):
        c_area[i]= cv2.contourArea(contours[i]) 
    cnts=contours[np.argmax(c_area)]    
    x, y, w, h = cv2.boundingRect(cnts)
    roi = croping(img, x, y, w, h)
    roi=cv2.resize(roi,(resize_shape),interpolation=cv2.INTER_LANCZOS4)
    return roi


# TO Data Matrix
def to_imgmatrix(resize_shape,path,df):
    path=path+'/'  
    images=crop_resize(path+df["Jpeg_Name"][0],resize_shape)
    for i in range (1,len(df["Jpeg_Name"])):
        img=crop_resize(path+df["Jpeg_Name"][i],resize_shape)
        images=np.concatenate((images,img))
    images=np.reshape(images,(len(df["Jpeg_Name"]),resize_shape[0],resize_shape[1],1))
    return images

def prepare_data(path,resize_shape):
    df=to_dataframe(path)
    data=to_imgmatrix(resize_shape,path,df) 
    return df,data

download_dataset("/content/Data")

#We want to resize 256x256 

df,data=prepare_data("/content/Data",(256,256))

Let’s see df:

df.head()

We need to y as categorical data to give as an input to model, so the code :

# We need numeric category
def to_categoricalmatrix(df):
    #There are little categories, so i handled manually
    Y=np.zeros([len(df["Tirads"])])
    for i in range(len(df["Tirads"])):
        if df["Tirads"][i]=="2":
          Y[i]=0
        if df["Tirads"][i]=="3":
          Y[i]=1
        if df["Tirads"][i]=="4a":
          Y[i]=2
        if df["Tirads"][i]=="4b":
          Y[i]=3
        if df["Tirads"][i]=="4c":
          Y[i]=4
        if df["Tirads"][i]=="5":
          Y[i]=5
    return Y

# to integer
y=to_categoricalmatrix(df)
y=tf.keras.utils.to_categorical(y, dtype='float32')

Befero the training our model , we also need to normalize to the images :

#normalize function
def normalize(data):
  for i in range(len(data)):
    data[i,:,:,:]=data[i,:,:,:]*(1/np.max(data[i,:,:,:]))
  return np.float32(data)
  
# we need noormalize to images
x=normalize(data)

Let’s see , random examples from X :

import random
random_number=random. randint(0,len(df["Tirads"]))
plt.figure(figsize = (20,10))
tit="Classification : "+np.str(df["Tirads"][random_number])
plt.title(tit,fontsize = 40)
plt.imshow(x[random_number,:,:,0],cmap="gray")

Before the training , we need to split dataset. In final , we have 347 images.

#Splitting test ,validation ,and train
x_train=np.copy(x[:300,:,:,:])
x_test=np.copy(x[313:,:,:,:])
x_valid=np.copy(x[300:313,:,:,:])

y_train=np.copy(y[:300,:])
y_valid=np.copy(y[300:313,:])
y_test=np.copy(y[313:,:])

Pre-processing is the initial stage in refining image data, such as removing distortion, so that it may be utilized to process data more effectively. We apply many methods of pre-processing in this study, including augmentation, which tries to avoid overfitting so that even if the device encounters a difficulty with micro variations, the software can still make accurate predictions.

So ,The augmentation used is a random rotation distance of 20, random zooming area in the range of 0.2, a random contrast in the range of 0.1, and, a random horizontal flip.

from tensorflow.keras import layers
#Data Augmention for to prevent Overfitting and to improve accuracy
data_augmentation1 = tf.keras.Sequential([
 layers.experimental.preprocessing.RandomFlip(
    "horizontal"),
layers.experimental.preprocessing.RandomZoom(height_factor=(-0.2, 0.2),fill_mode="constant"),
layers.experimental.preprocessing.RandomRotation(factor=(-0.2, 0.2),fill_mode="constant"),
tf.keras.layers.experimental.preprocessing.RandomContrast(0.1)])

x_train1=data_augmentation1(x_train)
y_train1=np.copy(y_train)
i=1

#22
while(i<22):
  x_aug=data_augmentation1(x)
  x_train1=np.concatenate((x_train1,x_aug),axis=0)
  y_aug=np.copy(y)
  y_train1=np.concatenate((y_train1,y_aug))

  #20
  if i == 20:
    break
  i += 1

During training, the input to our VGG-19 is a fixed-size 256 × 256 gray scale image. The image is passed through a stack of convolutional (conv.) layers, where we use filters with a very small receptive field: 3 × 3 .The padding is same for 3 × 3 convolution layers.Max-pooling is performed over a 2 × 2 pixel window, with stride 2. A stack of convolutional layers is followed by three Fully-Connected (FC) layers: the first two have 1024 channels each.The final layer is the soft-max layer.. All hidden layers are equipped with the rectification (ReLU (Krizhevsky et al., 2012)) non-linearity.

The Code:

def VGG19(input_shape,filters):
    inputs=tf.keras.layers.Input(shape=input_shape)
    
    x = tf.keras.layers.Conv2D(filters//16,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
    x=tf.keras.layers.Dropout(0.1)(x)
    x = tf.keras.layers.Conv2D(filters//16,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)

    x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(filters//8,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.Dropout(0.2)(x)
    x = tf.keras.layers.Conv2D(filters//8,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)
    
    x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(filters//4,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.Dropout(0.3)(x)
    x = tf.keras.layers.Conv2D(filters//4,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Conv2D(filters//4,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)

    x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(filters//2,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.Dropout(0.4)(x)
    x = tf.keras.layers.Conv2D(filters//2,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Conv2D(filters//2,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)

    x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(filters,(3,3),activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.Dropout(0.5)(x)
    x = tf.keras.layers.Conv2D(filters,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x=tf.keras.layers.BatchNormalization()(x)
    last = tf.keras.layers.Conv2D(filters,(3,3), activation = 'relu', padding = 'same', kernel_initializer = 'he_normal',name='top_conv')(x)
    
    model=tf.keras.Model(inputs,last,name="VGG19")
    return model

In fully connective layers, we will use L1-L2 regulazier to prevent overfitting .

base_model=VGG19(input_shape=(256,256,1),filters=512)
x = base_model.output
f=tf.keras.layers.Flatten(name="flatten")(x)
#To prevent overfitting and unbalancing , used regularizer
d2=tf.keras.layers.Dense(1024,activation="relu",kernel_regularizer=tf.keras.regularizers.l1_l2(0.00001))(f)
dp9=tf.keras.layers.Dropout(0.5)(d2)
d3=tf.keras.layers.Dense(1024,activation="relu")(f)
dp10=tf.keras.layers.Dropout(0.5)(d2)

final=tf.keras.layers.Dense(6,activation="softmax")(dp10)
model = tf.keras.Model( inputs =[ base_model.input], outputs = final)

We will use “Categorical Cross-Entropy Function” for loss function. Also, the model will be trained with 35 epochs and 16 batch size. After each 15 epochs , we want to decrease learning rate in order to slowly converge to the model and prevent overfitting.

metrics=tf.keras.metrics.AUC(
    num_thresholds=200, curve='ROC',
    summation_method='interpolation'
)
#categorical_crossentropy
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001), loss="categorical_crossentropy",metrics=metrics)


def lr_scheduler(epoch, lr):
    decay_rate = 0.1
    decay_step = 15
    if epoch % decay_step == 0 and epoch:
        return lr * decay_rate
    return lr

#after each 15 epochs , we want to decrease learning rate for converge to model
lr_call = tf.keras.callbacks.LearningRateScheduler(lr_scheduler)
epochs=35
history=model.fit(x=[x_train1],y=[y_train1],batch_size=16,epochs=epochs,callbacks=[lr_call],validation_data=(x_valid,y_valid))

Lets see our validation and training loss .

plt.figure(figsize = (20,10))
plt.title('Loss')
plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='test')
plt.legend()

Let’s evaluete our model :

import sklearn
predict=model.predict(x_test)
auc = sklearn.metrics.roc_auc_score(y_test, predict)

Our AUC score is 0.734. ROC curve and AUC score are an important metric for the image classification.

Next ROC Curve :

y_test=np.reshape(y_test,(34*6))
predict=np.reshape(predict,(34*6))
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score

# keep probabilities for the positive outcome only
ns_probs = [0 for _ in range(len(y_test))]
# calculate scores
ns_auc = roc_auc_score(y_test, ns_probs)
lr_auc = roc_auc_score(y_test, predict)
# summarize scores
print('No Skill: ROC AUC=%.3f' % (ns_auc))
print('Model: ROC AUC=%.3f' % (lr_auc))
# calculate roc curves
ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)
lr_fpr, lr_tpr, _ = roc_curve(y_test, predict)
# plot the roc curve for the model
plt.figure(figsize = (20,10))
plt.title("ROC Curve",fontsize = 40)
plt.plot(ns_fpr, ns_tpr,label='No Skill')
plt.plot(lr_fpr, lr_tpr, label='Model')
# axis labels
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.rcParams["font.size"] = "15"

# show the legend
plt.legend()
# show the plot
plt.show()

Grad-CAM is a strict generalization of the Class Activation Mapping. Unlike CAM, Grad-CAM requires no re-training and is broadly applicable to any CNN-based architectures. We also show how GradCAM may be combined with existing pixel-space visualizations to create a high-resolution classdiscriminative visualization (Guided Grad-CAM). In this study ,we used GradCam method to examine results.

Now GradCam :

#The GradCam observes the results
def make_gradcam_heatmap(img_array, model, last_conv_layer_name, classifier_layer_names ):
    # First, we create a model that maps the input image to the activations
    # of the last conv layer
    last_conv_layer = model.get_layer(last_conv_layer_name)
    last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)
    # Second, we create a model that maps the activations of the last conv
    # layer to the final class predictions
    classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])
    x = classifier_input
    for layer_name in classifier_layer_names:
        x = model.get_layer(layer_name)(x)
    classifier_model = keras.Model(classifier_input, x)
    # Then, we compute the gradient of the top predicted class for our input image
    # with respect to the activations of the last conv layer
    with tf.GradientTape() as tape:
        # Compute activations of the last conv layer and make the tape watch it
        last_conv_layer_output = last_conv_layer_model(img_array)
        tape.watch(last_conv_layer_output)
        # Compute class predictions
        preds = classifier_model(last_conv_layer_output)
        top_pred_index = tf.argmax(preds[0])
        top_class_channel = preds[:, top_pred_index]
    # This is the gradient of the top predicted class with regard to
    # the output feature map of the last conv layer
    grads = tape.gradient(top_class_channel, last_conv_layer_output)

    # This is a vector where each entry is the mean intensity of the gradient
    # over a specific feature map channel
    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))

    # We multiply each channel in the feature map array
    # by "how important this channel is" with regard to the top predicted class
    last_conv_layer_output = last_conv_layer_output.numpy()[0]
    pooled_grads = pooled_grads.numpy()
    for i in range(pooled_grads.shape[-1]):
        last_conv_layer_output[:, :, i] *= pooled_grads[i]

    # The channel-wise mean of the resulting feature map
    # is our heatmap of class activation
    heatmap = np.mean(last_conv_layer_output, axis=-1)

    # For visualization purpose, we will also normalize the heatmap between 0 & 1
    heatmap = np.maximum(heatmap, 0) / np.max(heatmap)
    return heatmap

from tensorflow import keras
img_array=x_test[0,:,:,:]

img_array=np.reshape(img_array,(1,256,256,1))
preds = model.predict(img_array)
last_conv_layer_name = "top_conv"
classifier_layer_names = ["flatten"]   
              
# Generate class activation heatmap
heatmap = make_gradcam_heatmap(
    img_array, model, last_conv_layer_name, classifier_layer_names
)
img = keras.preprocessing.image.img_to_array(x_test[0,:,:,:])

Create Heatmap :

from tensorflow import keras
img_array=x_test[0,:,:,:]

img_array=np.reshape(img_array,(1,256,256,1))
preds = model.predict(img_array)
last_conv_layer_name = "top_conv"
classifier_layer_names = ["flatten"]   
              
# Generate class activation heatmap
heatmap = make_gradcam_heatmap(
    img_array, model, last_conv_layer_name, classifier_layer_names
)
img = keras.preprocessing.image.img_to_array(x_test[0,:,:,:])

In figure 5 , as can be seen, the model targetted tyhriod tumor for classification. This shows that the model focuses the tyhriod tumor for the classfication

In this study, the thyroid nodules classified 6 different classses which are 1 (Benign) , 2 (Benign) ,4a (Malign) , 4b (Malign),4c (Malign) ,5 (Malign). Data augmentation is a technique used to increase the number of training data artificially by changing the ratio of width to height, changing colors, or using horizontal flip. It is reported to be an essential technique required by deep learning algorithms to achieve good performance . We used data augmention technique in this study becasue we have limited and unbalanced data.Despite that the most study is about the classification of thyroid tumors as bening and malign , in our study , thyroid tumors were classified 6 different classes. In this line , considering 6 classes, the unbalancing issue on dataset occurs. In this case , the most important issue is unbalancing of dataset and not enough data for each class. However, considering these issues ,our proposed model shows promising results with 0.734 AUC score. This score shows that the model has skills to classify thyroid tumors.

The Full Code : https://github.com/SerdarHelli/The-Classification-of-Thyroid-Tumors-on-UltraSound-Images-using-Deep-Learning-Methods

References

[1] Pedraza, Lina & Vargas, Carlos & Narváez, Fabián & Durán, Oscar & Muñoz, Emma & Romero, Eduardo. (2015). An open access thyroid ultrasound-image Database. Progress in Biomedical Optics and Imaging — Proceedings of SPIE. 9287. 10.1117/12.2073532.

[2] Selvaraju, Ramprasaath R., et al. “Grad-cam: Visual explanations from deep networks via gradient-based localization.” Proceedings of the IEEE international conference on computer vision. 2017.

[3] Abadi, Mart\’{\i}n, Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., … others. (2016). Tensorflow: A system for large-scale machine learning. In Symposium on Operating Systems Design and Implementation (pp. 265–283)

NFTs (Non-Fungible Tokens) are digital forms of actual valuables. Over the last few years, NFTs have considerably grown in popularity. Besides a countless number of artists, lots of companies enter the realm of NFT. These digital assets are already being referred to as ‘’the art of the future’’.

Core Neuro Art is a one-of-a-kind digital collection of MR pictures that have been combined with art. These collectibles are minted on the Polygon blockchain and are displayed on the OpenSea platform. Still avaiable on Opensea: https://opensea.io/collection/coreneuroart

“The brain created art itself and this is their meeting after all…”

Core Neuro Art was born with the question of ‘’ What would it be like if we processed the art into brain MR images?’’ asked by two biomedical engineer friends working in medical imaging laboratory. Concepts including our ability to gather new knowledge about human physiology, detect anomalies, and treat diseases are evolving in combination with quickly developing technology. It is obvious that it will offer the opportunity to perform remote surgery, early diagnosis, and treatment with artificial intelligence in the near future. In addition to this digital hype, Core Neuro Art seeks to raise awareness of the rise in medical imaging by integrating MR pictures with art..

Metaverse, Healthcare, and Core Neuro Art

Facebook recently declared that it will rename “Meta” as a new emphasis for the “Metaverse,” an interactive sharing place. In other words, Facebook CEO Mark Zuckerberg refers to the “Metaverse” as the “next-generation internet,” which also comprises virtual reality products and services. Various major international corporations, including Microsoft, have indicated that they will participate in the “Metaverse.” So, how will the healthcare business integrate into the “Metaverse,” and what exactly is Core Neuro Art attempting to convey?

In the “Metaverse,” numerous innovative concepts for the field of health can be considered. It might, for example, offer us digital patient treatment and patient follow-up across the world, and medical and surgical education, etc. It is inevitable that tremendous potentials for the healthcare industry emerge in this evolving digital environment.

The first virtual reality surgeries on live patients have been done by Johns Hopkins neurosurgeons. “During the initial procedure on June 8, 2020, surgeons inserted six screws into a patient’s spine for spinal fusion surgery to treat the patient’s chronic, excruciating back pain,” according to a Johns Hopkins Medicine publication. On June 10, surgeons removed a cancerous tumor known as a chordoma from a patient’s spine in the second surgery. Both patients are in great health, thus according to doctors.”

In order to highlight these developments, Core Neuro Art brings its NFT collection to the forefront in the artistic scene. After artificial intelligence technology enhanced the quality and resolution of brain MR images, the images were interpreted with artistic interpretation, resulting in a one-of-a-kind digital collection.

In the Future, Core Neuro Art

Core Neuro Art has interpreted the brain with a unique perspective in this adventure where we are just beginning to appreciate the human brain and wanted to draw attention to the potential brought by technology. Moreover, they will also generate virtual reality artwork using 3D Brain MR images, even though exact timing is unknown. As part of an NFT collection, this artwork will be waiting for us. Stay tuned!

References

• https://www.hopkinsmedicine.org/news/articles/johns-hopkins-performs-its-first-augmented-reality-surgeries-in-patients
• https://www.medivis.com/