{"id":11601,"date":"2020-12-31T05:40:00","date_gmt":"2020-12-31T05:40:00","guid":{"rendered":"https:\/\/www.askpython.com\/?p=11601"},"modified":"2026-09-29T16:51:21","modified_gmt":"2026-09-29T16:51:21","slug":"logistic-regression-from-scratch","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/examples\/logistic-regression-from-scratch","title":{"rendered":"Logistic Regression From Scratch in Python [Algorithm Explained]"},"content":{"rendered":"\n

The objective of this tutorial is to implement our own Logistic Regression from scratch. This is going to be different from our previous tutorial on the same topic where we used built-in methods to create the function.<\/p>\n\n\n\n

Logistic regression is a classic method mainly used for Binary Classification problems. even though it can be used for multi-class classification problems with some modification, in this article we will perform binary classification.<\/p>\n\n\n\n

Implementing Logistic Regression from Scratch<\/h2>\n\n\n\n

Step by step we will break down the algorithm to understand its inner working and finally will create our own class.<\/p>\n\n\n\n

Step-1: Understanding the Sigmoid function<\/h3>\n\n\n\n

The sigmoid function in logistic regression returns a probability value that can then be mapped to two or more discrete classes. Given the set of input variables, our goal is to assign that data point to a category (either 1 or 0). The sigmoid function outputs the probability of the input points belonging to one of the classes. <\/p>\n\n\n

\n#Defining a sigmoid function\ndef sigmoid(z):\n    op = 1\/(1 + np.exp(-z)) \n    return op\n<\/pre><\/div>\n\n\n
Step-2: The Loss Function<\/h3>\n\n\n\n
The loss function consists of parameters\/weights, when we say we want to optimize a loss function by this we simply refer to finding the best values of the parameters\/weights.<\/p>\n\n\n\n
The loss function for Logistic Regression is defined as:<\/p>\n\n\n\n
\"Loss
Loss Function<\/figcaption><\/figure><\/div>\n\n\n
\n#Loss Function\n\ndef loss(h, y):\n    return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()\n<\/pre><\/div>\n\n\n
Step-3: Gradient descent<\/h3>\n\n\n\n
The Gradient descent is just the derivative of the loss function with respect to its weights.<\/p>\n\n\n\n
We get this after we find find the derivative of the loss function:<\/p>\n\n\n\n
\"Gradient
Gradient Of Loss Function<\/figcaption><\/figure><\/div>\n\n\n
\n#Gradient_descent\n\ndef gradient_descent(X, h, y):\n    return np.dot(X.T, (h - y)) \/ y.shape[0]\n<\/pre><\/div>\n\n\n
The weights are updated by subtracting the derivative (gradient descent) times the learning rate. Updating the weights:<\/p>\n\n\n\n
\"Updating
Updating Weights<\/figcaption><\/figure><\/div>\n\n\n\n
Here – alpha<\/strong> is the learning rate.<\/p>\n\n\n\n
Putting it all together<\/h2>\n\n\n\n
Let’s create a class to compile the steps mentioned above. Here’s the complete code for implementing Logistic Regression from scratch. We have worked with the Python numpy module<\/a> for this implementation. <\/p>\n\n\n
\n#import required modules\nimport numpy as np\n\nclass LogisticRegression:\n    def __init__(self,x,y):      \n        self.intercept = np.ones((x.shape[0], 1))  \n        self.x = np.concatenate((self.intercept, x), axis=1)\n        self.weight = np.zeros(self.x.shape[1])\n        self.y = y\n        \n    #Sigmoid method\n    def sigmoid(self, x, weight):\n        z = np.dot(x, weight)\n        return 1 \/ (1 + np.exp(-z))\n    \n    #method to calculate the Loss\n    def loss(self, h, y):\n        return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()\n    \n    #Method for calculating the gradients\n    def gradient_descent(self, X, h, y):\n        return np.dot(X.T, (h - y)) \/ y.shape[0]\n\n    \n    def fit(self, lr , iterations):\n        for i in range(iterations):\n            sigma = self.sigmoid(self.x, self.weight)\n            \n            loss = self.loss(sigma,self.y)\n\n            dW = self.gradient_descent(self.x , sigma, self.y)\n            \n            #Updating the weights\n            self.weight -= lr * dW\n\n        return print('fitted successfully to data')\n    \n    #Method to predict the class label.\n    def predict(self, x_new , treshold):\n        x_new = np.concatenate((self.intercept, x_new), axis=1)\n        result = self.sigmoid(x_new, self.weight)\n        result = result >= treshold\n        y_pred = np.zeros(result.shape[0])\n        for i in range(len(y_pred)):\n            if result[i] == True: \n                y_pred[i] = 1\n            else:\n                continue\n                \n        return y_pred\n            \n<\/pre><\/div>\n\n\n
To implement the Algorithm we defined a fit<\/code> method which requires the learning rate and the number of iterations as the input arguments.<\/p>\n\n\n\n
The above class can be initialized by providing the input data and the target values.<\/p>\n\n\n\n
Now, it’s time to test our implementation.<\/h3>\n\n\n
\nfrom sklearn.datasets import load_breast_cancer\n\n#Loading the data\ndata = load_breast_cancer()\n\n#Preparing the data\nx = data.data\ny = data.target\n\n#creating the class Object\nregressor = LogisticRegression(x,y)\n\n#\nregressor.fit(0.1 , 5000)\n\n\ny_pred = regressor.predict(x,0.5)\n\nprint('accuracy -> {}'.format(sum(y_pred == y) \/ y.shape[0]))\n<\/pre><\/div>\n\n\n
Output:<\/strong><\/p>\n\n\n
\nfitted successfully to data\naccuracy -> 0.9209138840070299\n<\/pre><\/div>\n\n\n
Our implemented model achieved accuracy of 92%, not bad.<\/p>\n\n\n\n
 You can find the notebook for this tutorial here<\/a> on my GitHub Repository. <\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
This article was all about implementing a Logistic Regression Model from scratch to perform a binary classification task. We also unfold the inner working of the regression algorithm by coding it from 0. <\/p>\n\n\n\n
Till we meet next time. Happy Learning!<\/p>\n","protected":false},"excerpt":{"rendered":"
The objective of this tutorial is to implement our own Logistic Regression from scratch. This is going to be different from our previous tutorial on the same topic where we used built-in methods to create the function. Logistic regression is a classic method mainly used for Binary Classification problems. even though it can be used […]<\/p>\n","protected":false},"author":16,"featured_media":68434,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-11601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-examples"],"_links":{"self":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/11601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/comments?post=11601"}],"version-history":[{"count":0,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/11601\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media\/68434"}],"wp:attachment":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media?parent=11601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/categories?post=11601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/tags?post=11601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}