{"id":35924,"date":"2022-10-30T08:58:03","date_gmt":"2022-10-30T08:58:03","guid":{"rendered":"https:\/\/www.askpython.com\/?p=35924"},"modified":"2023-01-29T09:30:46","modified_gmt":"2023-01-29T09:30:46","slug":"sigmoid-activation-function","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/examples\/sigmoid-activation-function","title":{"rendered":"The sigmoid activation function in Python"},"content":{"rendered":"\n<p>If you&#8217;re learning about neural networks, chances are high that you have come across the term <strong>activation function<\/strong>. In neural networks, an activation function decides whether a particular neuron will be activated or not. Activation functions take the weighted summation of the nodes as input and perform some mathematical computation, depending on the activation function, and output a value that decides whether a neuron will be activated or not.<\/p>\n\n\n\n<p>They are many activation functions like <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/www.askpython.com\/python\/examples\/relu-function\" data-type=\"post\" data-id=\"19314\">ReLu<\/a><\/li><li><a href=\"https:\/\/www.askpython.com\/python\/examples\/calculating-softmax\" data-type=\"post\" data-id=\"18458\">Softmax<\/a><\/li><li><a href=\"https:\/\/www.askpython.com\/python\/examples\/tanh-activation-function\" type=\"post\" id=\"35856\">tanh<\/a><\/li><li><a href=\"https:\/\/www.askpython.com\/python\/numpy-linear-algebraic-functions\" data-type=\"post\" data-id=\"13883\">Linear<\/a><\/li><li><a href=\"https:\/\/www.askpython.com\/python-modules\/training-gan-in-pytorch\" data-type=\"post\" data-id=\"14678\">Leaky ReLu<\/a><\/li><li>Sigmoid<\/li><\/ul>\n\n\n\n<p>In this tutorial, we will be learning about the <strong>sigmoid activation function<\/strong>. So let&#8217;s begin!<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">What is the sigmoid function &#8211; The math behind it<\/h2>\n\n\n\n<p>Sigmoid is a <strong>non-linear <\/strong>activation function. It is mostly used in models where we need to predict the probability of something. As probability exists in the value range of 0 to 1, hence the <strong>range of sigmoid <\/strong>is also from<strong> 0 to 1<\/strong>, both inclusive.<br>Let&#8217;s have a look at the equation of the sigmoid function.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"602\" height=\"102\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-1.png\" alt=\"Sigmoid Equation \" class=\"wp-image-35890\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-1.png 602w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-1-300x51.png 300w\" sizes=\"auto, (max-width: 602px) 100vw, 602px\" \/><figcaption>Sigmoid Equation <\/figcaption><\/figure>\n\n\n\n<p>Sigmoid is usually denoted using the Greek symbol sigma.  So, we can also write<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"311\" height=\"71\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-equation-1.png\" alt=\"Sigmoid Equation 1\" class=\"wp-image-35971\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-equation-1.png 311w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-equation-1-300x68.png 300w\" sizes=\"auto, (max-width: 311px) 100vw, 311px\" \/><figcaption>Sigmoid Equation 1<\/figcaption><\/figure>\n\n\n\n<p>In the above equation, &#8216;<strong>e<\/strong>&#8216; is Euler&#8217;s number. Its value is approximately <strong>2.718<\/strong>. Similarly, <\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"333\" height=\"68\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1.png\" alt=\"Sigmoid -x Eq 1\" class=\"wp-image-35972\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1.png 333w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1-300x61.png 300w\" sizes=\"auto, (max-width: 333px) 100vw, 333px\" \/><figcaption>Sigmoid -x Equation 1<\/figcaption><\/figure>\n\n\n\n<p>In fact, we can derive a relation between the above two equations as<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"307\" height=\"73\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation.png\" alt=\"Sigmoid X And -X Relation\" class=\"wp-image-35973\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation.png 307w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation-300x71.png 300w\" sizes=\"auto, (max-width: 307px) 100vw, 307px\" \/><figcaption>Sigmoid x And -x Relation<\/figcaption><\/figure>\n\n\n\n<p>We can also prove this relation as shown below:<\/p>\n\n\n\n<p><strong>LHS:<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"333\" height=\"68\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1.png\" alt=\"LHS equation 1\" class=\"wp-image-35972\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1.png 333w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-1-300x61.png 300w\" sizes=\"auto, (max-width: 333px) 100vw, 333px\" \/><figcaption>LHS Equation 1<\/figcaption><\/figure>\n\n\n\n<p>It can also be written as<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"382\" height=\"78\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-4-2.png\" alt=\"LHS equation 2\" class=\"wp-image-36047\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-4-2.png 382w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-eq-4-2-300x61.png 300w\" sizes=\"auto, (max-width: 382px) 100vw, 382px\" \/><figcaption>LHS Equation 2<\/figcaption><\/figure>\n\n\n\n<p><strong>RHS:<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"441\" height=\"66\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-1.png\" alt=\"RHS Equation 1\" class=\"wp-image-36014\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-1.png 441w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-1-300x45.png 300w\" sizes=\"auto, (max-width: 441px) 100vw, 441px\" \/><figcaption>RHS Equation 1<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"407\" height=\"86\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-2.png\" alt=\"RHS Equation 2\" class=\"wp-image-36016\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-2.png 407w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/rhs-eq-2-300x63.png 300w\" sizes=\"auto, (max-width: 407px) 100vw, 407px\" \/><figcaption>RHS Equation 2<\/figcaption><\/figure>\n\n\n\n<p>Therefore, LHS=RHS<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"567\" height=\"87\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation-2.png\" alt=\"Sigmoid x And -x Relation Equation 2\" class=\"wp-image-36017\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation-2.png 567w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-x-and-x-relation-2-300x46.png 300w\" sizes=\"auto, (max-width: 567px) 100vw, 567px\" \/><figcaption>Sigmoid x And -x Relation Equation 2<\/figcaption><\/figure>\n\n\n\n<p>Hence, we have proved the relation.<\/p>\n\n\n\n<p>Another property of the sigmoid activation function is that it is <strong>differentiable<\/strong>. Let us see how we can differentiate it.<\/p>\n\n\n\n<p>Differentiating <em>sigmoid equation 1<\/em> we get<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"767\" height=\"92\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-1-1.png\" alt=\"Sigmoid Differentiation Equation 1 \" class=\"wp-image-36044\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-1-1.png 767w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-1-1-300x36.png 300w\" sizes=\"auto, (max-width: 767px) 100vw, 767px\" \/><figcaption>Sigmoid Differentiation Equation 1 <\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"412\" height=\"88\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-2.png\" alt=\"Sigmoid Differentiation Equation2\" class=\"wp-image-36041\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-2.png 412w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-diff-eq-2-300x64.png 300w\" sizes=\"auto, (max-width: 412px) 100vw, 412px\" \/><figcaption>Sigmoid Differentiation Equation 2<\/figcaption><\/figure>\n\n\n\n<p>So, from <em>Sigmoid Equation 1<\/em>, <em>Sigmoid x And -x Relation Equation 2<\/em> and <em>Sigmoid Differentiation Equation 2<\/em>, we can write<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"417\" height=\"78\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-1.png\" alt=\"Sigmoid And Its Differentiation Equation 1\" class=\"wp-image-36050\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-1.png 417w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-1-300x56.png 300w\" sizes=\"auto, (max-width: 417px) 100vw, 417px\" \/><figcaption>Sigmoid And Its Differentiation Equation 1<\/figcaption><\/figure>\n\n\n\n<p>Or, <\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"406\" height=\"98\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-2.png\" alt=\"Sigmoid And Its Differentiation Equation 2\" class=\"wp-image-36051\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-2.png 406w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-and-its-diff-eq-2-300x72.png 300w\" sizes=\"auto, (max-width: 406px) 100vw, 406px\" \/><figcaption>Sigmoid And Its Differentiation Equation 2<\/figcaption><\/figure>\n\n\n\n<p>Phew! That&#8217;s a lot of maths! Now, let us have a look at the graph of the sigmoid function.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Sigmoid graph using Python Matplotlib<\/h2>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\n#importing the required libraries\nfrom math import exp\nfrom matplotlib import pyplot as plt \n\n#defining the sigmoid function \ndef sigmoid(x):\n    return 1\/(1+exp(-x))\n\n#input \ninput = &#x5B;]\nfor x in range(-5, 5):\n    input.append(x)\n    \n#output\noutput = &#x5B;]\nfor ip in input:\n    output.append(sigmoid(ip))\n    \n#plotting the graph\nplt.plot(input, output)\nplt.title(&quot;Sigmoid activation function&quot;)\nplt.grid()\n#adding labels to the axes\nplt.xlabel(&quot;x&quot;)\nplt.ylabel(&quot;sigmoid(x)&quot;)\nplt.scatter(&#x5B;0], &#x5B;0.5], color=&quot;red&quot;, zorder=5) \nplt.show()\n<\/pre><\/div>\n\n\n<p> <strong>Output:<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"523\" height=\"353\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-plot.png\" alt=\"Sigmoid Plot\" class=\"wp-image-36077\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-plot.png 523w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-plot-300x202.png 300w\" sizes=\"auto, (max-width: 523px) 100vw, 523px\" \/><figcaption>Sigmoid Plot<\/figcaption><\/figure>\n\n\n\n<p>The above plot leads us to a few properties of the sigmoid function. They are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>S-shaped:<\/strong> The graph of <code>sigmoid<\/code> is S-shaped just like the graph of <code>tanh<\/code> activation function.<\/li><li><strong>Domain:<\/strong> The domain of <code>sigmoid<\/code> is (-\u221e, +\u221e).<\/li><li><strong>Continuous:<\/strong> The <code>sigmoid<\/code> function is continuous everywhere.<\/li><li>The <code>sigmoid<\/code> function is <strong>monotonically increasing<\/strong>.<\/li><li><strong>sigmoid(0)= 0.5<\/strong><\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Relation between sigmoid and tanh <\/h2>\n\n\n\n<p>We have previously discussed the <a href=\"https:\/\/www.askpython.com\/python\/examples\/tanh-activation-function\" type=\"post\" id=\"35856\">tanh activation<\/a> function in our tutorial. <\/p>\n\n\n\n<p>The equation of tanh is:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"656\" height=\"106\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-eq-2.png\" alt=\"tanh Equation\" class=\"wp-image-35888\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-eq-2.png 656w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-eq-2-300x48.png 300w\" sizes=\"auto, (max-width: 656px) 100vw, 656px\" \/><figcaption>tanh Equation<\/figcaption><\/figure>\n\n\n\n<p>And, <\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"632\" height=\"102\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-2.png\" alt=\"Sigmoid Eq 2\" class=\"wp-image-35891\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-2.png 632w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/sigmoid-eq-2-300x48.png 300w\" sizes=\"auto, (max-width: 632px) 100vw, 632px\" \/><figcaption>Sigmoid(2x) Equation<\/figcaption><\/figure>\n\n\n\n<p>These two functions are related:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"708\" height=\"103\" src=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-sigmoid-relation-eq.png\" alt=\"Tanh Sigmoid Relation Equation\" class=\"wp-image-35892\" srcset=\"https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-sigmoid-relation-eq.png 708w, https:\/\/www.askpython.com\/wp-content\/uploads\/2022\/10\/tanh-sigmoid-relation-eq-300x44.png 300w\" sizes=\"auto, (max-width: 708px) 100vw, 708px\" \/><figcaption>Tanh Sigmoid Relation Equation<\/figcaption><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p>Let&#8217;s have a quick recap: The sigmoid activation function is non-linear, monotonic, S-shaped, differentiable, and continuous. That&#8217;s all! We have learned about the sigmoid activation function and also its properties. <\/p>\n\n\n\n<p>Hope you found this tutorial helpful. Do check out more such tutorials related to Python <a href=\"https:\/\/www.askpython.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a>.  <\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you&#8217;re learning about neural networks, chances are high that you have come across the term activation function. In neural networks, an activation function decides whether a particular neuron will be activated or not. Activation functions take the weighted summation of the nodes as input and perform some mathematical computation, depending on the activation function, [&hellip;]<\/p>\n","protected":false},"author":46,"featured_media":36053,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-35924","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-examples"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/35924","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/users\/46"}],"replies":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/comments?post=35924"}],"version-history":[{"count":0,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/35924\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media\/36053"}],"wp:attachment":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media?parent=35924"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/categories?post=35924"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/tags?post=35924"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}