{"id":41310,"date":"2023-02-27T18:43:54","date_gmt":"2023-02-27T18:43:54","guid":{"rendered":"https:\/\/www.askpython.com\/?p=41310"},"modified":"2023-02-27T18:43:56","modified_gmt":"2023-02-27T18:43:56","slug":"adam-optimizer","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/examples\/adam-optimizer","title":{"rendered":"Adam optimizer: A Quick Introduction"},"content":{"rendered":"\n

Optimization is one of the critical processes in deep learning that helps in tuning the parameters of a model to minimize the loss function. Adam optimizer is one of the widely used optimization algorithms in deep learning that combines the benefits of Adagrad<\/a> and RMSprop<\/a> optimizers. <\/p>\n\n\n\n

In this article, we will discuss the Adam optimizer, its features, and an easy-to-understand example of its implementation in Python using the Keras library.<\/a><\/p>\n\n\n\n

What is Adam Optimizer and how it works?<\/h2>\n\n\n\n

Adam stands for Adaptive Moment Estimation. It is an optimization algorithm that was introduced by Kingma and Ba<\/a> in their 2014 paper. The algorithm computes the adaptive learning rates for each parameter and stores the first and second moments of the gradients.<\/p>\n\n\n\n

Adam optimizer is an extension of the stochastic gradient descent (SGD) algorithm that updates the learning rate adaptively. The Adam optimizer updates the parameters of the model using the first and second moments of the gradients. The first moment is the mean of the gradients, and the second moment is the uncentered variance of the gradients. <\/p>\n\n\n\n

The algorithm computes the adaptive learning rates for each parameter and uses the first and second moments of the gradients to adapt the learning rate. This helps in providing a different learning rate for each parameter and hence more precise parameter updates.<\/p>\n\n\n\n

The working of Adam optimizer can be summarized in the following steps:<\/p>\n\n\n\n

    \n
  1. Initialize the learning rate and the model weights.<\/li>\n\n\n\n
  2. Compute the gradients of the model with respect to the loss function using backpropagation<\/a>.<\/li>\n\n\n\n
  3. Compute the moving average of the gradient and the squared gradient.<\/li>\n\n\n\n
  4. Compute the bias-corrected moving averages.<\/li>\n\n\n\n
  5. Update the model weights using the bias-corrected moving averages.<\/li>\n<\/ol>\n\n\n\n

    The Adam optimizer updates the learning rate adaptively, depending on the gradient’s moving average and the squared gradient’s moving average. The moving average is computed for each parameter, and the learning rate is updated accordingly. This helps in providing a different learning rate for each parameter, which is useful in case some parameters are more sensitive than others.<\/p>\n\n\n\n

    Example of Adam Optimizer<\/h2>\n\n\n\n

    Let’s understand this with an easy to understand example:<\/p>\n\n\n\n

    Minimize the value of function x^3 – 2*x<\/strong>^2 + 2. The manual computation looks something like this:<\/p>\n\n\n\n

    \"Calculating
    Calculating The Minima Manually<\/figcaption><\/figure>\n\n\n\n

    Adam Optimizer Implementation in Python<\/h2>\n\n\n\n

    Now let’s see how to use Adam optimizer for computing the same:<\/p>\n\n\n

    \nimport tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Define the function\ndef f(x):\n    return x**3 - 2*x**2 + 2\n\n# Define the Adam optimizer\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.05,  epsilon=1e-07,)\n# Define the starting point for optimization\nx = tf.Variable(0.001)\n\n# Define a list to store the history of x values\nx_values = []\n\n# Define the number of optimization steps\nnum_steps = 100\n\n# Perform the optimization\nfor i in range(num_steps):\n    with tf.GradientTape() as tape:\n        # Calculate the value of the function and record the gradient\n        y = f(x)\n        gradient = tape.gradient(y, x)\n    # Use the Adam optimizer to update the value of x\n    optimizer.apply_gradients([(gradient, x)])\n    # Record the current value of x\n    x_values.append(x.numpy())\n\n# Print the optimized value of x and the value of the function at that point\nprint("Optimized value of x:", x.numpy())\nprint("Value of the function at the optimized point:", f(x.numpy()))\n# Plot the function and the optimization path\nx_plot = np.linspace(-2, 2, 500)\ny_plot = f(x_plot)\nplt.plot(x_plot, y_plot, label='Function')\nplt.plot(x_values, [f(x) for x in x_values], label='Optimization path', marker='o')\nplt.xlabel('x')\nplt.ylabel('y')\nplt.title('Function and Optimization Path')\nplt.legend()\nplt.show()\n<\/pre><\/div>\n\n\n
    The output looks like this:<\/strong><\/p>\n\n\n\n
    \"Adam
    Adam Optimizer Example 1<\/figcaption><\/figure>\n\n\n\n
    Now that we’ve understood the working of Adam with an example, let’s also know about how it is different from other optimizers.<\/p>\n\n\n\n
    Advantages of Adam over other optimizers<\/h2>\n\n\n\n
    \"Comparsion
    Comparison Of Adam to Other Optimization Algorithms
    Taken from Adam paper<\/a><\/figcaption><\/figure>\n\n\n\n
    Let’s see how is this different from other optimizers:<\/p>\n\n\n\n