Lin et al, 2020

Research Topic

• Category (General): Deep Learning
• Category (Specific): Computer Vision

Paper summary

• Propose a new loss function tackling the extreme class imbalance problem encountered in object detections.
• Reshaping Cross Entropy Loss Function such that it down-weights the loss assigned to well-classified examples.
• Understand if one-stage de-tectors can match or surpass the accuracy of two-stage detectors while running at similar or faster speeds.
• Introduce the RetinaNet to evaluate the effectiveness of the loss, but since the author stated: “Our simple detector achieves top results not based on innovations in network design but due to our novel loss.”, I’m gonna skip the RetinaNet in this recap.

Explain Like I’m 5 (ELI5) 👶👶👶

• Basically it is Balanced Cross Entropy Loss Function but with a tunable modulating factor such that it controls the weight loss of easy/hard samples.

Issues addressed by the paper

• Current state-of-the-art object detectors are based on a two-stage, proposal-driven mechanism, which consistently achieves top accuracy on the challenging COCO benchmark.
• Could a simple one-stage detector achieve similar accuracy?
• Recent work on one-stage detectors yield faster detectors with accuracy within 10-40% relative to state-of-the-art two-stage methods, but takes also a lot of time.
• Class imbalance is addressed in R-CNN-like (one-stage) detectors by a two-stage cascade and sampling heuristics.
• An one-stage detector must process a much larger set of candidate object locations regularly sampled across an image, which makes the model inefficient as the training procedure is still dominated by easily classified background examples.
• Balanced CE loss function: while \(\alpha\) balances the importance of positive/negative examples, it does not differentiate between easy/hard examples.

Approach/Method

• Present a one-stage object detector (RetinaNet) that, for the first time (by the time of publish), matches the state-of-the-art COCO AP.
• To achieve this result:
  • Identify class imbalance during training as the main obstacle impeding one-stage detector from achieving state-of-the-art accuracy,
  • Propose a new loss function that eliminates this barrier.
• Focal Loss:
  • Dynamically scaled cross entropy loss.
  • Automatically down-weight the contribution of easy examples (inliers) during training and rapidly focus the model on hard examples.
  • Focal loss performs the opposite role of a robust loss (e.g: Huber Loss): it focuses training on a sparse set of hard examples.
  • They first defined the probability function like this:
  \[p_t = \left\{ \begin{array}{ll} p & if \ y=1 \\ 1-p & otherwise. \end{array} \right.\]
  • Using this definition, we can define CE loss func like this: \(CE(p,y) = CE(p_t) = -log(p_t)\) for comparision.
  • Balanced Focal Loss’s definition:
  \[FL(p_t) = -\alpha_{t}(1-p_t)^{\gamma}log(p_t)\]
  • When an example is misclassified and \(p_t\) is small, the modulating factor is near 1 and the loss is unaffected. As \(p_t \to 1\), the factor goes to 0 and the loss for well-classified is down-weighted.
  • The focusing parameter \(\gamma\) smoothly adjusts the rate at which easy examples are down-weighted.
  • When \(\gamma = 0\), focal loss is equivalent to cross entropy loss, as \(\gamma\) is increased the effect of the modulating factor is likewise increased.

  

Best practice

• \(\gamma = 2\) works well in practice.
• The benefit of changing \(\gamma\) is much larger, and indeed the best \(\alpha\)’s ranged in just [.25,.75].
• In general \(\alpha\) should be decreased slightly as \(\gamma\) is increased.

Hidden gems💎💎💎

• In practice \(\alpha\) (of Balanced CE) may be set by inverse class frequency or treated as a hyperparameter to set by cross validation.
• One notable property of CE loss, which can be easily seen in the upper plot, is that even examples that are easily classified incur a loss with non-trivial magnitude. When summed over a large number of easy examples, these small loss values can overwhelm the rare class.
• Two-stage detectors are often trained with the CE loss without use of \(\alpha\)-balancing, but address class imbalance through two mechanisms: (1) a two-stage cascade and (2) biased minibatch sampling.
• Feature Pyramid Network augments a standard convolutional network with a top-down pathway and lateral connections so the network efficiently constructs a rich, multi-scale feature pyramid from a single resolution input image.
• Larger backbone networks yield higher accuracy, but also slower inference speeds, likewise for input image scale.

Confusing aspects of the paper

• The different between easy and hard sample?
  • The difference between the two is not obvious from the paper.
  • IMO, easy samples means that the probability \(p_t\) is close to the class (1 for positive, 0 for negative), vice versa for the hard samples.
• Why \(\alpha\)-balancing of CE doesn’t differentiate between easy/ hard examples?
  • This issue is also not verified in the paper.
  • IMO, \(\alpha\) is only used to address the imbalanced class problem (since its value also tuned for the same reason), Focal Loss adds the modular factor which directly affected by the probability that the model suggested and adjust the loss.

Results

• The model achieves top results, outperforming both one-stage and two-stage models.

Conclusions

The author’s conclusions

• Focal loss applies a modulating term to the cross entropy loss in order to focus learning on hard negative examples.
• The approach is simple and highly effective.

Rating

Paper implementation

class FocalLoss(nn.Module):
    def __init__(self, n_class, alpha=None, gamma=2, reduction='mean'):
        super(FocalLoss, self).__init__()
        if alpha is None:
            self.alpha = Variable(torch.ones(n_class, 1))
        else:
            self.alpha = alpha if isinstance(alpha, Variable) else Variable(alpha)

        self.gamma = gamma
        self.n_class = n_class
        self.reduction = reduction

    def forward(self, inputs, targets):
        device = inputs.device
        bs = inputs.size(0)
        n_channel = inputs.size(1)
        probs = F.softmax(inputs)
        class_mask = inputs.data.new(bs, n_channel).fill_(0)
        class_mask = Variable(class_mask)
        ys = targets.view(-1, 1)
        # onehot encoding
        class_mask.scatter_(1, ys.data, 1.)

        if inputs.is_cuda and not self.alpha.is_cuda:
            self.alpha = self.alpha.to(device)

        alphas = self.alpha[ys.data.view(-1)]
        probs = (probs*class_mask).sum(1).view(-1,1)
        log_p = probs.log()
        batch_loss = -alphas*(torch.pow((1-probs), self.gamma))*log_p
        if self.reduction == "mean":
            loss = batch_loss.mean()
        else:
            loss = batch_loss.sum()
        return loss

Cited references and used images from:

• Focal Loss for Dense Object Detection
• Reddit post

Papers needed to be conquered next 👏👏👏

• SIMCLR maybe?

He, Tong Zhang, Zhi Zhang, Hang Zhang, Zhongyue Xie, Junyuan Li, Mu, 2019

Research Topic

• Category (General): Deep Learning
• Category (Specific): Computer Vision

Paper summary

• Synthesize multiple effective methods that are briefly mentioned in litterature, in order to evaluate their impact on the final model accuracy through ablation study.

Issues addressed by the paper

• Find a combination of training procedure and model architecture refinements that improve model accuracy but barely change computational complexity.
• These “tricks” usually are minor ones (such as changing the stride of a convolutional layer), but create a big impact.
• These methods lead to significant accuracy improvement and combining them together can further boost the model accuracy. Transfer learning’s performance of these “boosted” model would be much better too.

Approach/Method

Training baseline

• Use the same training transformation (random sampling, random crop, random flip, hue/saturation/brightness, PCA noise, normalize).
• Only resize in validation set.
• Xavier initialization for weights of both convolutional and fully connected.
• Use NAG (Nesterov), 120 epochs, 8 GPUs, batch-size = 256.
• LR initialized to 0.1 and divided by 10 at the 30th, 60th, and 90th epochs.

Efficient Training

1. Larger training batch-size:
   • Since we’re using mini-batch SGD, for the same number of epochs, training with a large batch size results in a model with degraded validation accuracy compared to the ones trained with smaller batch sizes.
   • Some heuristics the help scale the batch size up like:
     • Linear scaling learning rate: a large batch size reduces the noise in the gradient, so we may increase the learning rate to make a larger progress along the op- posite of the gradient direction.
     • Learning rate warmup: Using a too large learning rate may result in numerical instability, so we gradually increase it.
     • Zero \(\gamma\): we initialize \(\gamma = 0\) for all BN layers that sit at the end of a residual block. Therefore, all residual blocks just return their inputs, mimics network that has less number of layers and is easier to train at the initial stage.
     • No bias decay: Only apply to the regularization to weights to avoid overfitting.
2. Low-precision training:
   • Use FP16 to train since new hardware may have enhanced arithmetic logic unit for lower precision data types.

Model Tweaks

• A model tweak is a minor adjustment to the network architecture (changing the stride of convlution layer).
• Such a tweak barely changes the computational complexity but might have a non-negligible effect on the model accuracy.
• Take a closer look at Resnet architecture:
  • Resnet network consists of an input stem (reduces the input W/H by 4 times and increases its channel size to 64), four subsequent stages and a final output layer.
  • Resnet has a basic block called Bottleneck Block (downsampling block). There are 2 paths in the block:
    • Path A: has three convolutions, whose kernel sizes are 1×1 (stride 2 - halve the input size), 3×3 and 1×1 (output channels x4 the previous 2), respectively.
    • Path B: uses a 1×1 convolution (stride of 2) to create the same output shape as path A, so we can sum outputs of both paths to obtain the output of the downsampling block.

• There are 3 tweaks mentioned is this paper:

• Resnet-B: modifies downsampling block
  • Observation: convolution in path A (original) ignores \(\frac{3}{4}\) of the input feature map because it uses a kernel size 1×1 with a stride of 2.
  • Tweak: Change stride of the first two conv, so no information is ignored.
• Resnet-C: modifies input stem
  • Observation: the computational cost of a convolution is quadratic to the kernel width or height (A 7 × 7 convolution is 5.4 times more expensive than a 3 × 3 convolution).
  • Tweak: replacing the 7×7 convolution with 3 conservative 3 × 3 convolutions, with the first and second convolutions have their output channel of 32 and a stride of 2, while the last convolution uses a 64 output channel.
• Resnet-D: modifies downsampling block
  • Observation: Inspired by Resnet-B, 1×1 convolution in the path B also ignores \(\frac{3}{4}\) of input feature maps.
  • Tweak: Change stride to 1 and add a 2×2 average pooling layer with a stride of 2 before the convolution, since it works well in practice and impacts the computational cost little.

Training Refinements

• Cosine Learning Rate Decay:
  • Method: Same as my previous post.
  • Result: Compared to the step decay, the cosine decay starts to decay the learning since the beginning but remains large until step decay reduces the learning rate by 10x, which potentially improves the training progress.

• Label Smoothing:
  • Method: Same as my previous post.
  • Result: Compared to the softmax, it encourages a finite output from the fully-connected layer and can generalize better.
• Knowledge Distillation: skipped.
• Mixup: Same as my previous post.

Conclusions

Rating

Papers needs to conquer next 👏👏👏

• Focal Loss

Yun, Sangdoo Han, Dongyoon Chun, Sanghyuk Oh, Seong Joon Choe, Junsuk Yoo, Youngjoon

Research Topic

• Category (General): Deep Learning
• Category (Specific): Data Augmentation

Paper summary

• Introduce a new data augmentation method: CutMix, which helps:
  • Generates new samples by cutting and pasting patches within mini-batches, leading to performance boosts in many computer vision tasks.
  • Improves the model robustness against input corruptions
  • Performs better when facing out-of-distribution detection problems.

Explain Like I’m 5 (ELI5) 👶👶👶

Issues addressed by the paper

• Regional cutout may remove informative pixels, in addition to it, these techniques also add noises by replace those cut pixels with black area.
• How can we maximally utilize the deleted regions, while taking advantage of better generalization and localization using regional dropout?
• While certainly improving classification performance, Mixup samples tend to be unnatural.
• Synthesizing training data guides also improves model performance, but introduces costly computations.
• Recently, methods adding noises to the internal features of CNNs or adding extra path to the architecture have been proposed to enhance image classification performance.
• Cutmix has properties such that can compensate all of those mentioned above.

Approach/Method

• Core idea: We have 2 pictures, choose a region (apply to both pictures) in the image that is proportional to the image’s ratio, then swap those regions, leads to a new sample.

    new_image = MASK * image_1 + (1-MASK) * image_2
    new_label = lambda * label_1 + (1-lambda) * label_2
  

• lambda is sampled from symmetric Beta distribution (having same \(\alpha\) and \(\beta\) value), which looks like the following image. This distribution makes sure that the new image will most of the time be close only to the first or the second picture. Only sometimes both images have the same intensity (of course if in a case of multicategorical, other label will be 0).

• MASK has the same WxH like the original image, having full 0s in the region that is gonna be cutout (let’s call it B) and full 1s in the leftover pixels.
• B is a bounding box, having coordinates defined as \(B = (r_x, r_y, r_w, r_h)\) indicating the cropping regions.

• Having these coordinates defined like this would make the cropped area ratio equals to \(1 - \lambda\).

Result

• CutMix overcomes the problem of both Mixup (its samples tend to be unnatural) and regional cutout (removes informative pixels) by replacing the image region with a patch from another training image.
• CutMix incurs only negligible additional cost for training.
• Cutout successfully lets a model focus on less discriminative parts of the object.

• Outperforms other SOTAs using data augmentation during the time of publish.

Best practice

• \(\alpha = 1\) usually performs well.
• Cutout, Mixup, and CutMix require a greater number of training epochs till convergence.
• CutMix achieves the best performance when it is applied on the input images rather than hidden features.
• A new state-of-the-art performance 13.81% (in CIFAR-100) by combining CutMix and ShakeDrop, a regularization that adds noise on intermediate features.

Hidden gems💎💎💎

• Label-smoothing is good when apply together with data augmentation such as Cutout, Mixup, Dropblock.
• Mixup, Manifold Mixup achieve higher accuracies when Cutout is applied on input images.

Conclusions

Rating

Paper implementation

• Using Pytorch Lightning Callback system, inspired by fastai implementation.

    class MixLoss(nn.Module):
        def __init__(self, old_lf, mixup_cb):
            super().__init__()
            self.old_lf = old_lf
            self.mixup_cb = mixup_cb
  
        def forward(self, pred, yb):
            if self.mixup_cb.pl_module.testing: return self.old_lf(pred, yb)
            with NoneReduce(self.old_lf) as lf:
                self.mixup_cb.yb_1 = self.mixup_cb.yb_1.to(pred.device)
                self.mixup_cb.lam = self.mixup_cb.lam.to(pred.device)
                loss = torch.lerp(lf(pred, self.mixup_cb.yb_1), lf(pred,yb), self.mixup_cb.lam)
            return reduce_loss(loss, getattr(self.old_lf, 'reduction', 'mean'))
  
    # Cell
    class CutmixDict(Callback):
        def __init__(self, alpha=1.):
            super().__init__()
            self.distrib = Beta(tensor(alpha), tensor(alpha))
  
        def on_train_start(self, trainer, pl_module):
            assert hasattr(pl_module, 'loss_func'), 'Your LightningModule should have loss_func attribute as your loss function.'
            self.old_lf = pl_module.loss_func
            self.loss_fnc = MixLoss(self.old_lf, self)
            pl_module.loss_func = self.loss_fnc
            self.pl_module = pl_module
  
        def _cutmix(self, batch, logger, log_image=False, pre_fix='train'):
            xb, yb = batch["img"], batch["label"]
            bs = yb.size(0)
            W, H = xb.size(3), xb.size(2)
  
            lam = self.distrib.sample((1,)).squeeze()
            lam = torch.stack([lam, 1-lam])
            self.lam = lam.max()
  
            # Permute the batch
            shuffle = torch.randperm(bs)
            xb_1, self.yb_1 = xb[shuffle], yb[shuffle]
  
            x1, y1, x2, y2 = self.rand_bbox(W, H, self.lam)
            xb[:, :, x1:x2, y1:y2] = xb_1[:, :, x1:x2, y1:y2]
            self.lam = (1 - ((x2-x1) * (y2-y1)) / float(W*H))
  
            if log_image:
                grid = torchvision.utils.make_grid(xb)
                logger.experiment.add_image(pre_fix + '_cutmix', grid)
                grid_g = torchvision.utils.make_grid(xb_1)
                logger.experiment.add_image(pre_fix + '_cut_from', grid_g)
                dif = abs(xb - xb_1)
                grid_d = torchvision.utils.make_grid(dif)
                logger.experiment.add_image(pre_fix + '_dif', grid_d)
            return xb
  
        def on_train_batch_start(self, trainer, pl_module, batch, batch_idx, dataloader_idx):
            x = self._cutmix(batch, trainer.logger)
            batch["img"] = x
  
        def on_validation_start(self, trainer, pl_module):
            pl_module.loss_func = self.old_lf
  
        def on_validation_end(self, trainer, pl_module):
            pl_module.loss_func = self.loss_fnc
  
        def on_fit_end(self, trainer, pl_module):
            pl_module.loss_func = self.old_lf
  
        def rand_bbox(self, W, H, lam):
            cut_rat = torch.sqrt(1. - lam)
            cut_w = (W * cut_rat).type(torch.long)
            cut_h = (H * cut_rat).type(torch.long)
            # uniform
            cx = torch.randint(0, W, (1,))
            cy = torch.randint(0, H, (1,))
            x1 = torch.clamp(cx - cut_w // 2, 0, W)
            y1 = torch.clamp(cy - cut_h // 2, 0, H)
            x2 = torch.clamp(cx + cut_w // 2, 0, W)
            y2 = torch.clamp(cy + cut_h // 2, 0, H)
            return x1, y1, x2, y2
  

Papers needs to conquer next 👏👏👏

• Any recommendation?

Zhang, Hongyi Cisse, Moustapha Dauphin, Yann N.Lopez-Paz, David, 2018

Research Topic

• Category (General): Deep Learning
• Category (Specific): Data Augmentation

Paper summary

• Introduce a new data augmentation method: MixUp, which helps:
  • Improve generalization of neural network architectures.
  • Reduces memorization of corrupt labels.
  • Increases the robustness to adversarial learning.
  • Stabilize the training of GANs.

Explain Like I’m 5 (ELI5) 👶👶👶

• Well this gif sums up pretty well…

Issues addressed by the paper

• As the model is being trained by minimizing the loss/objective function using a known set of training data (Empirical Risk Minimization), it can choose to memorize the training data rather than generalize it, even when in the presence of strong regularization. So the model will perform terribly when evaluating on examples outside the training distribution.
• Normal data augmentation can help model to improve generalization, but it requires costly computations, and also needs human experts to decide how to augment the data.

Approach/Method

• Core idea: Linear interpolate (lerp) two images and its labels together, then do it for the whole batch. Use soft label for better performance. How much to interpolate is used by a number call lambda generated by Beta Distribution.

    new_image = lambda * image_1 + (1-lambda) * image_2
    new_label = lambda * label_1 + (1-lambda) * label_2
  

• lambda is sampled from symmetric Beta distribution (having same \(\alpha\) and \(\beta\) value), which looks like the following image. This distribution makes sure that the new image will most of the time be close only to the first or the second picture. Only sometimes both images have the same intensity (of course if in a case of multicategorical, other label will be 0).

• They tried multiple ways to pick images:
  • Use 2 different dataloader, take one of each then interpolate -> good mix, but need 2 loaders which takes more time to run.
  • Use more than two pictures to mix: non-significant improvement, but increases the computation cost.
  • Use only 1 dataloader, then mixup the batch and its shuffled-version -> may have duplicates, but peforms equally well -> RECOMMEND.

Result

• Reduces the amount of undesirable oscillations when predicting outside the training examples, also makes memorization more difficult to achieve (generalizes better). The author had make tests by training both normal (ERM) and mixup model against randomly corruptled labels:
  • As you can see, the ERM model starts to overfit the corrupted labels when the LR starts to slow down to fine tune.
  • mixup model doesn’t do the same, so when it’s tested against the real label, it still performs fairly well -> not overfitted.
• Leads to decision boundaries that transition linearly from class to class, providing a smoother estimate of uncertainty:
  • In the ERM model, if an adversarial example (class orange) can push through the blue boundary, the model will definately determine that example is class 0.
  • On the other hand, since mixup model produce a smooth boundary, it may still classify that example correctly.

• Significantly improve the robustness of neural networks without hindering the speed of ERM.

Best practice

• \(\alpha \in [0.1, 0.4]\) usually performs well, whereas larger \(\alpha\) may lead to underfitting.
• When using mixup, use larger neural network and also train longer for better result.
• mixup + dropout performs very nicely together (produced SOTA in the paper).

Hidden gems💎💎💎

• For normal data augmentation, one usually uses rotation, translation, cropping, resizing, flipping and random erasing to enforce visually plausible invariances in the model through the training data.

Conclusions

Rating

Paper implementation

• Using Pytorch Lightning Callback system, inspired by fastai implementation.

    class MixLoss(nn.Module):
        def __init__(self, old_lf, mixup_cb):
            super().__init__()
            self.old_lf = old_lf
            self.mixup_cb = mixup_cb
  
        def forward(self, pred, yb):
            if self.mixup_cb.pl_module.testing: return self.old_lf(pred, yb)
            with NoneReduce(self.old_lf) as lf:
                self.mixup_cb.yb_1 = self.mixup_cb.yb_1.to(pred.device)
                self.mixup_cb.lam = self.mixup_cb.lam.to(pred.device)
                loss = torch.lerp(lf(pred, self.mixup_cb.yb_1), lf(pred,yb), self.mixup_cb.lam)
            return reduce_loss(loss, getattr(self.old_lf, 'reduction', 'mean'))
  
    class MixupDict(Callback):
        def __init__(self, alpha=0.4):
            super().__init__()
            self.distrib = Beta(tensor(alpha), tensor(alpha))
  
        def on_train_start(self, trainer, pl_module):
            self.old_lf = pl_module.loss_func
            self.loss_fnc = MixLoss(self.old_lf, self)
            pl_module.loss_func = self.loss_fnc
            self.pl_module = pl_module
  
        def _mixup(self, batch, logger, log_image=False, pre_fix='train'):
            xb, yb = batch["img"], batch["label"]
            bs = yb.size(0)
  
            # Produce "bs" probability for each sample
            lam = self.distrib.sample((bs,)).squeeze()
  
            # Get those probability that >0.5, so that the first img (in the nonshuffle batch) has bigger coeff
            # Which avoid duplication mixup
            lam = torch.stack([lam, 1-lam], 1)
            self.lam = lam.max(1)[0]
  
            # Permute the batch
            shuffle = torch.randperm(bs)
            xb_1, self.yb_1 = xb[shuffle], yb[shuffle]
            nx_dims = len(xb.size())
            weight = unsqueeze(self.lam, n=nx_dims-1)
            x_new = torch.lerp(xb_1, xb, weight=weight)
            if log_image:
                grid = torchvision.utils.make_grid(x_new)
                logger.experiment.add_image(pre_fix + 'mixup', grid)
                grid_g = torchvision.utils.make_grid(xb)
                logger.experiment.add_image(pre_fix + 'norm', grid_g)
                dif = abs(xb - x_new)
                grid_d = torchvision.utils.make_grid(dif)
                logger.experiment.add_image(pre_fix + 'dif', grid_d)
            return x_new
  
        def on_train_batch_start(self, trainer, pl_module, batch, batch_idx, dataloader_idx):
            x = self._mixup(batch, trainer.logger)
            batch["img"] = x
  
        def on_validation_start(self, trainer, pl_module):
            pl_module.loss_func = self.old_lf
  
        def on_validation_end(self, trainer, pl_module):
            pl_module.loss_func = self.loss_fnc
  
        def on_fit_end(self, trainer, pl_module):
            pl_module.loss_func = self.old_lf
  

Papers needs to conquer next 👏👏👏

• Cutmix

Ilya Loshchilov, Frank Hutter, 2016

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

Leslie N. Smith, Nicholay Topin, 2017

Research Topic

• Category (General): Deep Learning
• Category (Specific): Hyper Tuning, Optimization

Paper summary

SGDR

• Partial warm restarts improve rate of convergence, often used in gradient-free optimization.
• Propose a warm restart technique for stochastic gradient descent.
• Study its performance on CIFAR-10/100
• Show that this technique improve its anytime performance when training deep neural network:
  • SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results

Super-convergence

• Introduce “Super-convergence” term: where neural networks can be train and converge much faster than standard training methods.
• Propoce a way to achieve super-convergence: one-cycle policy + large learning rate.
• Use Hessian Free optimization method to produce an estimate of the optimal learning rate.
• Study its performance on CIFAR-10/100, MNIST, Imagenet with various model.
• Show that this phenomenon can also happen when the amount of labeled training data is limited but still boost the model performance.
• Mostly all of these are mentioned in the previous recap, so it will not be discussed more in the sections below.

Explain Like I’m 5 (ELI5) 👶👶👶

SGDR

• Just like a normal person works everyday. You start the day with maximum effort, but then time goes by you feel tired and the productivity reduces. You go home, rest. The next day, you’re recharged and start the cycle once again.

Super-convergence

• As a result of CLR, super-convergence is born.

Issues addressed by the paper

SGDR

• Despite of the existance of advanced optimization methods like Adam, AdaDelta, SOTA result on CIFAR-10/100, ImageNet still based on SGD with momentum, associated with Resnet model.
• Current way to get out of the plateau while using SGD is LR scheduler and L2 regularization.
• They want to break through and produce a new approach to SGD.

Super-convergence

• They found a way to train DNN faster and achieve better performance.
• Large LR regularizes training, so other regularization method should be reduced to maintain optimal balance of optimization
• Hessian-free optimization method estimates optimal LR, demonstrating that large LR find wide, flat minima.

Approach/Method

SGDR

• SGDR simulates a new warm-started run/restart of SGD after \(T_{i}\) epochs are performed.

• Learning rate is calculated by cos annealing function.

• \(\eta\): learning rate.
• \(T_{cur}\): how many epochs passed since the restart.

• \(T_{i}\): how mane epochs for a restart, you can leave it constant or increase over-time.

• How the learning rate looks after training:

Best practice

SGDR

• Start with small \(T_{i}\), then increase it by factor of \(T_{mult}\) at every start.

• Decreate max_lr and min_lr at every new start may increase performance.
• Stop training when current learning rate is equal to min_lr.
• SGDR allows to train larger network.

Results

• SGDR technique helps the author to surpass the SOTA at the time with much faster computational time.

Conclusions

The author’s conclusions

• Our SGDR simulates warm restarts by scheduling the learning rate to achieve competitive results on CIFAR-10 and CIFAR-100 roughly two to four times faster, achieved new state- of-the-art results with SGDR.
• SGDR might also reduce the problem of learning rate selection because the annealing and restarts of SGDR scan / consider a range of learning rate values.

Rating

My Conclusion

• Another technique can be considered when training in new project, should quick test overall before stick to one and go deeper.

Paper implementation

Cos Annealing

def sched_cos(start, end, T_cur, T_i):
    return start + (1 + math.cos(math.pi*(T_cur / T_i))) * (end-start) / 2

SGDR

def step(self, epoch=None):
    if epoch is None and self.last_epoch < 0:
        epoch = 0

    if epoch is None:
        epoch = self.last_epoch + 1
        self.T_cur = self.T_cur + 1
        if self.T_cur >= self.T_i:
            self.T_cur = self.T_cur - self.T_i
            self.T_i = self.T_i * self.T_mult
    else:
        if epoch < 0:
            return
        if epoch >= self.T_0:
            if self.T_mult == 1:
                self.T_cur = epoch % self.T_0
            else:
                n = int(math.log((epoch / self.T_0 * (self.T_mult - 1) + 1), self.T_mult))
                self.T_cur = epoch - self.T_0 * (self.T_mult ** n - 1) / (self.T_mult - 1)
                self.T_i = self.T_0 * self.T_mult ** (n)
        else:
            self.T_i = self.T_0
            self.T_cur = epoch
    self.last_epoch = math.floor(epoch)
    self.lr = sched_cos(self.base_lr, self.max_lr, self.T_cur, self.T_i)

Cited references and used images from:

• https://towardsdatascience.com/https-medium-com-reina-wang-tw-stochastic-gradient-descent-with-restarts-5f511975163
• https://arxiv.org/abs/1608.03983
• Pytorch library

Papers needs to conquer next 👏👏👏

• Mixup
• Cutmix

Lesli N.Smith, 2018

Research Topic

• Category (General): Deep Learning.
• Category (Specific): Hyperparameters Tuning.

Paper summary

• Introduce techniques to set some essential hyper-parameters such as: learning rate, batch size, momentum, weight decay.
• How to examine training/ test loss curve for clues of underfitting, overfitting.
• Introduce a new method of cyclical learning rate: 1cycle policy.
• Discuss about cyclical momentum, cyclical weight decay.
• Produce multiples examples to show the importance of balanced regularization for each dataset/ architecture.

Explain Like I’m 5 (ELI5) 👶👶👶

• Not necessary since this paper mostly tackle best pratices while working on projects.

Issues addressed by the paper

• Grid-search is computationally expensive and time consumming. But a good choice of hyper-parameters is vital for a model to perform well, so how to do grid-search efficiently?
• Underfitting/ overfitting trade-off.

1. Learning rate:
   • Too small: overfitting can occur.
   • Too large: can have regularize but will diverge.
2. Batch size:
   • Suggest that when we comparing batch size, neither maintaining constant #epochs (train the same #epochs for each batch size) nor constant #iterations (train the same #iterations/epoch each batch size) is appropriate:
     • constant #epochs:
       • Computationally efficient, but penalized more since we see a big proportion of samples each time.
     • constant #iterations:
       • Overfit will occur.
   • Batch size directly affect computational time.
3. Momentum:
   • Momentum and learning rate are closely related and its optimal values are dependent on each other.
   • Momentum’s effect on updating the weights is of the same magnitude as the learning rate (used SGD with momentum equation to verify).
4. Weight decay:
   • Weight decay is one form of regularization and it plays an important role in training so its value needs to be set properly.

Approach/Method

• 1cycle policy:
  • Instead of cyclical learning rate using triangular as the previous post, the author suggest to:
    1. Train all epochs in more than one cycle just a small proportion.
    2. In the remaining iterations, the learning rate will decline from base_lr to several orders of magnitude less.
  • Experiments show that this policy allows the accuracy to plateau before the training ends.
• The author show these 6 remarks after multiple researchs and experiments:
  1. The test/validation loss is a good indicator of the network’s convergence and should be examined for clues.
     • Look at the loss curve and also plot generalization error curve (valid_loss - train_loss), one can determine whether the architechture has the capacity to overfit or has too small learning rate (which also leads to overfit)
  2. Achieving the horizontal part of the test loss is the goal of hyperparameter tuning.
     • The horizontal part is the red line.
  3. The amount of regularization must be balanced for each dataset and architecture.
  4. The practitioner’s goal is obtaining the highest performance while minimizing the needed computational time.
  5. Optimal momentum value(s) will improve network training.
  6. Since the amount of regularization must be balanced for each dataset and architecture, the value of weight decay is a key knob to turn for tuning regularization against the regularization from an increasing learning rate.

Best practices

1. Learning rate:
   • Use learning rate range test to find the minimum and maximum learning rate boundaries:
     • Maximum learning rate bound: the maximum value that the model can still converge
     • Minumum learning rate bound:
       1. \(\frac{1}{3};\frac{1}{4}\) of max bound.
       2. \(\frac{1}{10};\frac{1}{20}\) of max bound if using 1cycle.
   • Use 1cycle policy to achive super-convergence (reachs global optima with iterations much less than regulars).
   • Other regularization methods must be reduced to compensate for the regularization effects of large learning rates.
2. Batch size:
   • Small batch sizes add regularization, large batch sizes add less; utilize this while balancing the proper amount of regularization.
   • Often better to use a larger batch size so a larger learning rate can be used (leads to using a larger batch size when using the 1cycle learning rate schedule).
3. Momentum:
   • Short runs with momentum values of 0.99, 0.97, 0.95, and 0.9 will quickly show the best value for momentum.
   • If use 1cycle policy, should use cyclical momentum starting at maximum momentum value and decreasing to a value of 0.8 or 0.85 (performance is almost independent of the minimum momentum value).
   • Decreasing cyclical momentum when the learning rate increases provides an equivalent result to the best constant momentum value.
   • Using cyclical momentum along with the LR range test stabilizes the convergence when using large learning rate values more than a constant momentum does.
4. Weight decay:
   • Should be a constant value.
   • Should use grid search to find a proper value; validation loss early in the training is sufficient for determining a good value.
   • Another option as a grid search for weight decay is to make a single run at a middle value for weight decay and save a snapshot after the loss plateaus. Use this snapshot to restart runs, each with a different value of WD. This can save time in searching for the best weight decay.
   • A complex dataset requires less regularization so test smaller weight decay values, such as \(10^{−4}, 10^{−5}, 10^{−6}, 0\)
   • A shallow architecture requires more regularization so test larger weight decay values, such as \(10^{−2}, 10^{−3}, 10^{−4}\).
   • The optimal weight decay is different if you search with a constant learning rate versus using a learning rate range.

Hidden gems💎💎💎

• Test loss decreases more rapidly during the initial iterations and is then horizontal is an early positive clue indicating that the model will produce a better final accuracy. (Blue curve)
• Learning rates that are too small can exhibit some overfitting behavior.
• There is a maximum speed the learning rate can increase without the training becoming unstable.
• The very large learning rates provided the twin benefits of regularization that prevented overfitting and faster training of the network.
• Set momentum as large as possible without causing instabilities during training.
• Momentum range test is not useful for finding an optimal momentum, you should use grid search.
• Decreasing the momentum while the learning rate increases provides three benefits (by experiments):
  1. a lower minimum test loss.
  2. faster initial convergence.
  3. greater convergence stability over a larger range of learning rates.
• Large momentum helps escape saddle points but can hurt the final convergence, implying that momentum should be reduced at the end of training.
• A good procedure is to test momentum values in the range of 0.9 to 0.99.
• All the general ideas can apply to shallow or deep networks, although the details (i.e., specific values for momentum) varied.

Results

• As the experiment shown, if one can find optimal values for these hyper-parameters, the model would achieve super-convergence, which saves computational cost and time.

  Limitations

• These disciplines are not proved but mostly achieved by experiments. So we can only use this as a guide and apply it to our projects.

Confusing aspects of the paper

• Very straightforward, confusing-free.

Conclusions

Rating

Would read again.

My conclusion

• Good workflow to deal with hyper-parameters.
• Can be use as a reference when start a new projects.

Paper implementation

Cited references and used images from:

• Lesli N.Smith, 2018
• https://sgugger.github.io/the-1cycle-policy.html
• https://github.com/asvcode/1_cycle

Papers needs to conquer next 👏👏👏

• SGDR

Lesli N.Smith, 2017

Research Topic

• Category (General): Deep Learning
• Category (Specific): Hyperparameters (Learning rate) Tuning

Paper summary

• Instead of monotonically decreasing the learning rate as in the traditional way, the author introduced a new method (called Cyclical Learning Rates) to control it, allowing the hyper-parameter to rise and fall systematically during training.
• CLR improves classification accuracy without tuning and does not require any computations.
• The author also showed a way to estimate the bounds for the learning rate to vary while training.

Explain Like I’m 5 (ELI5) 👶👶👶

Imagine you try to draw a picture, there are multiple ways to tackle this:

• You jump straight-in, drawing up to ~80-90% of the picture. However, to refine it, you have to slow down to finish the touches, which very tiring, so you might have to slow down much more until you finish it or abandon it midway. (~ Learning rate schedule)
• You start by drawing the layout, and then you calculate how much effort you should spend on each section. After you finish one, you reevaluate again and continue to draw until the picture is done. (~Adaptive learning rate)
• You start simple, draw a tree today, draw some more trees tomorrow, and draw a sun and a mountain the following day. The gist is that you start slow and progress over time until you reach a particular intensity. Then you slow down again to avoid burnout until you’re at the initial state (finish a cycle), you will restart the cycle again until the picture is done. (~Cyclical Learning Rates)

Issues addressed by the paper

• To train a deep neural network to convergence requires one to experiment with a variety of LR.
• There are already multiple solutions to this. These include learning rate scheduling (e.g., time-based decay, step decay, exponential decay), or adaptive learning rate (e.g., RMSProp, AdaGrad, AdaDelta, Adam), but there are still some drawbacks such as:
  • Learning rate scheduling: Monotonically decreased learning rate, which later proved that it would not help the model escape from the saddle point.
  • Adaptive learning rate: requires high computational cost.

Approach/Method

• Observation: increasing the learning rate might have a short term negative effect and yet achieve a longer-term beneficial effect. (since increasing the learning rate allows more rapid traversal of saddle point plateaus)
• Pick minimum (base_lr) and maximum (max_lr) boundaries, and the learning rate will cyclically vary between these bounds.

• For the cyclical function, triangular(Bartlett) window, parabolic(Welch) window, and sinusoidal(Hanning) window produced equivalent results, which led to adopting a triangular window thanks to its simplicity.

• Other variations:

  • triangular: using triangular window
  • triangular_2: same as triangular but after every cycle, max_lr is halved
  • exp_range: each boundary value declines by exponential factor of current iterations.

• How to find the bounds?
  • Using “LR range test”:
    • Run model for several epochs while letting the learning rate increase linearly between low and high LR values.
    • Notice these 2 points:
      • When accuracy starts to increase. -> base_lr
      • When accuracy slows down, becomes ragged or falls down. -> max_lr

Best practices

• Set stepsize to 2-10 times of #iterations/epoch.
• Best to stop training at the end of the cycle (LR at base_lr and the accuracy peaks)
  • -> Early stopping might not be good for CLR.
• Optimum learning rate is usually within a factor of two of the largest one that converges, and set base_lr = \(\frac{1}{3}\) or \(\frac{1}{4}\) of max_lr.

Results

• CLR helps to model to converge much faster.
• Decay (monotonically decreasing LR)’s result provides evidence that both increasing and decreasing LR are essential.

Limitations

• When using with adaptive learning rate methods, the benefits from CLR are reduced.

Confusing aspects of the paper

• The variables of the code weren’t explained clearly, so I redefine it here:
  • 1 epoch: converted to #iterations (= training_size / batch_size).
  • 1 cycle: LR goes from base_lr -> max_lr -> base_lr.
  • epochCounter: current #iterations (e.g: 1 epoch has 2000 iters, the model’s at epochs 1.5 => epochCounter = 3000).
  • stepsize: #iterations to reach 1/2 cycle.
  • opt.LR: base lr
  • cycle: which cycle the model’s currently in (1-2-…), always starts from 1 (first cycle).
  • x: which part of the half-cycle, in range [0,1].
  • lr: the updated lr, with triangular method.

Conclusions

The author’s conclusions

• All experiments show similar or better accuracy performance when using CLR versus using a fixed learning rate, even though the performance drops at some of the learning rate values within this range.

Rating

Noice

My Conclusion

• This paper is straightforward, well-explained, so I highly recommend new DL-practitioners to try reading it.
• CLR is an impressive technique to control the learning rate. We should try this method when we started a new model or a new dataset, giving a lovely baseline for further optimization.
• CLR is also widely used by kagglers.

Paper implementation

Cyclical Learning Rates

• triangular implementation:
• triangular_2 implementation:
• exp_range implementation:

• Plot:

• Explaination for how x is calculated:
  • \(raw\_x = \frac{current\_iteration}{step\_size}\): current cycle in term of half-cycles (floatting point)
  • \(x’ = raw\_x - 2*cycle\): how many half-cycles left to complete this cycle
  • \(x’ = x’ + 1\): Shift so that the function 0-centered on the y-axis, so that when we take absolute, we can achieve cycles.

Learning Rate Finder

• An implementation of LRFinder using CLR using Callback (highly influenced by fastai):

Cited references and used images from:

• https://www.jeremyjordan.me/nn-learning-rate/
• https://github.com/bckenstler/CLR/

Papers needs to conquer next 👏👏👏

• SGDR
• Fit One Cycle