ADA Research Group

This is the second blog post in the series about our research on Neural Architecture Search (NAS) for Earth Observation (EO). In Part 1 we introduced NAS and how it can be applied to EO. We talked about a NAS framework: AutoKeras [1], and briefly discussed how it can be customized for EO tasks.

In this blog post, we talk about how AutoKeras can be used to create methods for EO imagery. More specifically, we are going to tackle the task of classification of EO imagery through the work “Automated Machine Learning for Satellite Data: Integrating Remote Sensing Pre-trained Models into AutoML Systems” by Nelly R. Palacios Salinas, Mitra Baratchi, Jan N. van Rijn and Andreas Vollrath [2]. It is the first work that focused on developing AutoML methods for EO.

Even though we will discuss how you could create methods like this yourself, Palacios et al. already present a framework for the classification of EO imagery that can be used with just a few lines of code and without requiring in-depth prior knowledge of AutoML or deep learning in general. The links to the code as well as the full Python notebook for this blog can be found at the end of the post.

Classification

Image classification is the task of assigning labels to an image. For instance, you’d like to know whether your image is showing a desert or a forest or some other type of land cover. Classification has, among others, applications in tasks like urban planning, hazard detection, and monitoring of the environment [3]. In EO the term “image classification” is sometimes used to refer to what is called segmentation in computer science. This is the task of assigning labels to individual pixels. For instance, this is the case when you classify pixels into the classes “building” or “not building” for building segmentation tasks. We, however, will only talk about classification in the traditional sense where you consider the complete image.

EO imagery

The task of classifying natural images is very different from the task of classifying EO images. In EO images, areas of interest can be very small on the image. For instance, if you want to differentiate between “savanna” and “bare ground”, individual plants could cover only a few pixels or less, depending on the resolution of the image. In some satellite images, like those obtained from Sentinel-2 with a resolution of 10 m per pixel, a plant could even be much smaller than a single pixel. Additionally, an EO image can cover a large area that includes multiple land cover types and can contain image features on different scales, from a tree covering a few pixels to larger patterns in the image like a lake. These properties need to be taken into account when designing NAS frameworks for the classification of EO imagery.

NAS for classification of EO imagery

Many NAS frameworks exist for the classification of natural images, including NASnet [4], AutoGAN [5], and ENAS [6]. Whereas these frameworks work exceptionally well for natural images, these have not been designed with the complexity of satellite imagery in mind, and therefore their performance and applications for EO data are limited. One of the main reasons for these limitations lay down in the use of datasets with simpler images like CIFAR-10 [6] for training and evaluation.

Currently, NAS methods are being developed specifically for classifying EO images. For instance, the work of our research group members Palacios Salinas et al. [2] shows that their classifier developed in AutoKeras is able to outperform 71% of the baseline methods created for natural images. These results were achieved by customizing AutoKeras’ ImageClassifier.

Classification in AutoKeras

AutoKeras has a ready-to-use NAS system for image classification, which is called the ImageClassifier. The search space of the ImageClassifier consists of various code modules that can be morphed, repeated, and combined to form a neural network. This type of AutoKeras block that can combine multiple types of blocks is called a hyperblock. The options include ResNet [7], Xception [8], and a convolutional block. It’s possible to use pre-trained weights from ImageNet [8], which is a natural image dataset. If you need a refresher on AutoKeras, read our previous blogpost on this topic.

Tutorial: Hyperblock

AutoKeras has documentation on how to implement your own blocks, but let’s take it a step further and take a look at how we can implement a hyperblock that will allow you to choose from different blocks. We’re going to implement a hyperblock that will let the NAS framework choose between ResNet and Xception.

import autokeras as ak
import tensorflow as tf

class HyperBlock(ak.Block):
    def build(self, hp, inputs):
        inputs = tf.nest.flatten(inputs)[0]
        
        if hp.Choice("model_type",["resnet", "xception"]) == "resnet":
            outputs = ak.ResNetBlock().build(hp,inputs)
        else:
            outputs=ak.XceptionBlock().build(hp,inputs)
        return outputs

Here we define a new class that builds either a ResNet or a DenseNet block. You can modify the HyperBlock to make it possible to choose which model you want beforehand:

from typing import Optional

class HyperBlock(ak.Block):
    def __init__(self, model_type: Optional[str] = None,**kwargs):
        super().__init__(**kwargs)
        
        if model_type is not None and model_type != "resnet" and model_type != "xception":
            raise Exception(f"invalid model_type {model_type}")

        self.model_type=model_type

    def get_config(self):
        config = super().get_config()
        config.update({"model_type": self.model_type})
        return config

    def _build_model(self,hp, output_node,model_type: str):
        if model_type=="resnet":
            return ak.ResNetBlock().build(hp,output_node)
        elif model_type=="xception":
            return ak.XceptionBlock().build(hp, output_node)

    def build(self, hp, inputs):
        inputs=tf.nest.flatten(inputs)[0]

        # Let AutoKeras choose a model
        if self.model_type is None:
            model_type= hp.Choice("model_type", ["resnet", "xception"])
            with hp.conditional_scope("model_type",[model_type]):
                outputs = self._build_model(hp,inputs,model_type)
        # Select model yourself
        else:
            outputs = self._build_model(hp,inputs,model_type)

As you can see, this becomes considerably more complicated. Let’s break down the changes:

• We added an __init__ method so you can specify the model_type parameter
• Error handling is important: check whether a valid model_type parameter has been passed
• get_config: add the new block parameter to the config
• build: because you now have 2 scenarios (let AutoKeras choose a model or select yourself), we now need a conditional scope. The value of model_type which is selected by AutoKeras, will now only be active within the scope.
• _build_model: helper function to avoid code repetition. This becomes especially helpful if you have many options.

Here you go, your first hyperblock! You can also make your own block based on a specific neural network architecture and include it in your hyperblock. We will show you how to do this in Part 3 of this series.

Transfer learning

The ImageClassifier also gives you the option to use models that are pre-trained on ImageNet. However, as we discussed, this is not very useful for EO imagery (or even natural images, see: Rethinking Pre-training and Self-training [9]). Palacios Salinas et al. solved this by loading weights obtained from pre-training on some common EO datasets: RESISC45 [10], EUROSAT [11], So2SAT [12] and UC Merced [13].

Tutorial: loading weights

We’re going to look at how we can load these weights in our HyperBlock. Luckily for us, the weights can be downloaded from https://tfhub.dev.

EO_VERSIONS= {
    "resisc45": "https://tfhub.dev/google/remote_sensing/resisc45-resnet50/1",
    "eurosat": "https://tfhub.dev/google/remote_sensing/eurosat-resnet50/1",
    "so2sat": "https://tfhub.dev/google/remote_sensing/so2sat-resnet50/1",
    "ucmerced": "https://tfhub.dev/google/remote_sensing/uc_merced-resnet50/1",
}

Before we can use these weights in our HyperBlock, we need to make a ResNet block that can load the weights.

import tensorflow_hub as hub

class EOResNetBlock(ak.Block):
    #Remote sensing pretrained modules based on: https://github.com/palaciosnrps/automl-rs-project/blob/714bbe36c68fd0f2b989bfee89eac9497d7acf45/autokeras/blocks/basic.py"""
    def __init__(
         self, 
         version: Optional[str] = None,
         **kwargs,
         ):
            super().__init__(**kwargs)

            if version is not None and version not in EO_VERSIONS.keys() and set(version) <= set(EO_VERSIONS.keys()):
                raise Exception(f"invalid version {version}")
        
            self.version=version

    def get_config(self):
        config = super().get_config()
        config.update({"version": self.version})
        return config

    def build(self, hp, inputs=None):
        input_node = tf.nest.flatten(inputs)[0]

        if self.version is None:
            version= hp.Choice("version", list(EO_VERSIONS.keys()))
        elif isinstance(self.version,list):
            version = self.version
        else:
            version = [self.version]
            
        module = hub.KerasLayer(EO_VERSIONS[version],tags='train',trainable=False)
        min_size = 224
        if input_node.shape[3] not in [1, 3]:
            if self.pretrained:
                raise ValueError(
                    "When pretrained is set to True, expect input to "
                    "have 1 or 3 channels, bug got "
                    "{channels}.".format(channels=input_node.shape[3])
                )

        if input_node.shape[1] < min_size or input_node.shape[2] < min_size:
            input_node = tf.keras.layers.experimental.preprocessing.Resizing(
                max(min_size, input_node.shape[1]),
                max(min_size, input_node.shape[2]),
            )(input_node)
        if input_node.shape[3] == 1:
            input_node = tf.keras.layers.Concatenate()([input_node] * 3)
        if input_node.shape[3] != 3:
            input_node = tf.keras.layers.Conv2D(filters=3, kernel_size=1, padding="same")(
                input_node
            )	

        output_node = module(input_node)
        return output_node

In fact, this block is very similar to the HyperBlock, but instead of model_type it has a version parameter that specifies which weights to use. Once again, we check whether a correct version has been specified by the user. The build function is a bit different:

• We do not need a conditional scope now, because we don’t build blocks in a conditional statement.
• A module is created that serves as an interface to pre-trained tensorflow models.
• The number of channels of the input data is checked to make sure it agrees with the pre-trained model.

Now we can add our pre-trained block to the HyperBlock:

class HyperBlock(ak.Block):
    def __init__(self, model_type: Optional[str] = None, version: Optional[str] = None,**kwargs):
        super().__init__(**kwargs)
        
        if model_type is not None and model_type != "resnet" and model_type != "xception" and model_type != "eo_resnet":
            raise Exception(f"invalid model_type {model_type}")

        self.model_type=model_type
        self.version = version

    def get_config(self):
        config = super().get_config()
        config.update({"model_type": self.model_type})
        return config

    def _build_model(self,hp, output_node,model_type: str):
        if model_type=="resnet":
            return ak.ResNetBlock().build(hp,output_node)
        elif model_type=="xception":
            return ak.XceptionBlock().build(hp, output_node)
        elif model_type=="eo_resnet":
            return EOResNetBlock(version=self.version).build(hp,output_node)

    def build(self, hp, inputs):
        inputs=tf.nest.flatten(inputs)[0]

        # Let AutoKeras choose a model
        if self.model_type is None:
            model_type= hp.Choice("model_type", ["resnet", "xception", "eo_resnet"])
            with hp.conditional_scope("model_type",[model_type]):
                outputs = self._build_model(hp,inputs,model_type)
        # Select model yourself
        else:
            outputs = self._build_model(hp,inputs,self.model_type)
        return outputs

We can use our block to create a custom NAS with the AutoModel class of AutoKeras and train it on the UC Merced dataset. The number of trials determines how many networks are sampled by AutoKeras. We now set it to 1, just to test whether it works.

import tensorflow_datasets as tfds
# load the dataset 317.MiB
train_set=tfds.as_numpy(tfds.load("uc_merced", download=True,as_supervised=False, batch_size=-1,split="train[:80%]"))
test_set=tfds.as_numpy(tfds.load("uc_merced", download=True,as_supervised=False, batch_size=-1,split="train[80%:]"))

weights_versions = {k: v for k,v in EO_VERSIONS.items() if k != "ucmerced"}

input_node = ak.ImageInput()
output_node = HyperBlock(model_type="eo_resnet")(input_node)
output_node = ak.ClassificationHead()(output_node)
eo_nas_model = ak.AutoModel(input_node, output_node, max_trials=1,overwrite=True)

eo_nas_model.fit(x=train_set["image"], y=train_set["label"], epochs=10)

When you run this, you’ll find that the model compiles and the neural architecture search starts. Success!

Conclusion

In this blog post, we discussed NAS for the classification of EO images. Using the work by Palacios Salinas et al. as an example, you have learned how to customize the AutoKeras search space by creating a hyperblock and how to apply transfer learning by including pre-trained weights obtained from training on EO datasets.

The Python notebook for this blog post can be found at https://github.com/JuliaWasala/NAS_with_AutoKeras_EO. The original code by Nelly R. Palacios Salinas can be found on GitHub: https://github.com/palaciosnrps/automl-rs-project. This code allows you to use her methods for NAS for the classification of EO imagery in just a few lines of code.

In the next post, we will go even further into the customization of AutoKeras with another example of our research: NAS for super-resolution. We’re going to cover how to add our own custom metrics, add new model architectures to the search space, and extend AutoKeras to other tasks. Stay tuned for more advanced tutorials on AutoKeras for EO!

References

[1] Jin, H., Song, Q. and Hu, X., 2019, July. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1946-1956).

[2] Palacios Salinas, N.R., Baratchi, M., Rijn, J.N.V. and Vollrath, A., 2021, September. Automated machine learning for satellite data: integrating remote sensing pre-trained models into AutoML systems. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 447-462). Springer, Cham.

[3] Cheng, G., et al.: Remote sensing image scene classification meets deep learning: Challenges, methods, benchmarks, and opportunities. In: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13, 3735–3756 (2020)

[4] Zoph, B., Vasudevan, V., Shlens, J. and Le, Q.V., 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8697-8710).

[5] Gong, X., Chang, S., Jiang, Y. and Wang, Z., 2019. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3224-3234).

[6] Pham, H., Guan, M., Zoph, B., Le, Q. and Dean, J., 2018, July. Efficient neural architecture search via parameters sharing. In International conference on machine learning (pp. 4095-4104). PMLR.

[7] He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

[8] Chollet, F., 2017. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1251-1258).

[9] Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D. and Le, Q., 2020. Rethinking pre-training and self-training. Advances in neural information processing systems, 33, pp.3833-3845.

[10] Cheng, G., Han, J. and Lu, X., 2017. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10), pp.1865-1883.

[11] Helber, P., Bischke, B., Dengel, A. and Borth, D., 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7), pp.2217-2226.

[12] Zhu, X.X., Hu, J., Qiu, C., Shi, Y., Kang, J., Mou, L., Bagheri, H., Häberle, M., Hua, Y., Huang, R. and Hughes, L., 2019. So2Sat LCZ42: A benchmark dataset for global local climate zones classification. arXiv preprint arXiv:1912.12171.

[13] Yang, Y. and Newsam, S., 2010, November. Bag-of-visual-words and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems (pp. 270-279).