{"id":9523,"date":"2026-04-18T12:00:00","date_gmt":"2026-04-18T12:00:00","guid":{"rendered":"https:\/\/www.askpython.com\/?p=9523"},"modified":"2026-04-18T03:39:46","modified_gmt":"2026-04-18T03:39:46","slug":"principal-component-analysis","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/examples\/principal-component-analysis","title":{"rendered":"Principal Component Analysis from Scratch in Python"},"content":{"rendered":"

Principal component analysis, or PCA in short, is famously known as a dimensionality reduction technique.<\/p>\n

It has been around since 1901 and is still used as a predominant dimensionality reduction method in machine learning and statistics. PCA is an unsupervised statistical method.<\/p>\n

In this article, we will have some intuition about PCA and will implement it by ourselves from scratch using Python and NumPy.<\/p>\n

Why use PCA in the first place?<\/h2>\n

To support the cause of using PCA let\u2019s look at one example.<\/p>\n

Suppose we have a dataset<\/strong> having two variables and 10 data points. If we were asked to visualize the data points, we could do it very easily. The result is very interpretable as well.<\/p>\n

\n\n\n\n\n
X1<\/td>\n2<\/td>\n8<\/td>\n1<\/td>\n4<\/td>\n22<\/td>\n15<\/td>\n25<\/td>\n29<\/td>\n4<\/td>\n2<\/td>\n<\/tr>\n
X2<\/td>\n3<\/td>\n6<\/td>\n2<\/td>\n6<\/td>\n18<\/td>\n16<\/td>\n20<\/td>\n23<\/td>\n6<\/td>\n4<\/td>\n<\/tr>\n<\/tbody>\n<\/table>
Example Data points<\/figcaption><\/figure>\n
\"Plotting
Plotting Data On Two Dimensions<\/strong><\/figcaption><\/figure>\n

Now, if we try to increase the number of variables, it gets almost impossible for us to imagine a dimension higher than three.\u00a0<\/p>\n

This problem we face when analyzing higher-dimensional datasets is what is commonly referred to as \u201cThe curse of dimensionality<\/em><\/strong>\u201d. This term was first coined by\u00a0Richard E. Bellman.<\/p>\n

Principal Component analysis reduces high dimensional data to lower dimensions while retaining the original information and capturing the maximum variability of the dataset. Data visualization is the most common application of PCA. PCA is also used to make the training of an algorithm faster by reducing the number of dimensions of the data, which improves prediction efficiency.<\/p>\n

Implementation of PCA with Python<\/h2>\n

To grasp the maximum intuition from the content given below, we assume you must know a little bit about linear algebra and matrices<\/a>. If not, then we highly encourage you to watch the Linear algebra series of 3Blue1Brown<\/a> on YouTube by Grant Sanderson, to get a refresher of the concepts, as it will prove to be very beneficial in your Machine Learning journey ahead.<\/p>\n

We can think of Principal Component analysis to be like fitting an n-dimensional ellipsoid to the data so that each axis (ax) of the ellipsoid represents a principal component. The larger the principal component axis, the larger the variability in data it represents.<\/p>\n

\"Fitting
Example: Fitting An Ellipse To Data<\/strong><\/figcaption><\/figure>\n

 Steps to implement PCA in Python<\/h2>\n
#Importing required libraries\nimport numpy as np<\/code><\/pre>\n
1. Subtract the mean of each variable<\/h3>\n
Subtract the mean of each variable from the dataset so that the dataset is centered on the origin, with values ranging from min to max around zero. Doing this proves to be very helpful when calculating the covariance matrix.<\/p>\n
#Generate a dummy dataset.\nX = np.random.randint(10,50,100).reshape(20,5) \n# mean Centering the data  \nX_meaned = X - np.mean(X , axis = 0)<\/code><\/pre>\n
Data generated by the above code has dimensions (20,5) i.e. 20 examples and 5 variables for each example. The number of features in this sample dataset is 5. We calculated the mean of each variable and subtracted that from every row of the respective column.<\/p>\n
2. Calculate the Covariance Matrix<\/h3>\n
Calculate the Covariance Matrix of the mean-centered data. You can know more about the covariance matrix in this really informative Wikipedia article here<\/a>. <\/p>\n
The covariance matrix is a square matrix denoting the covariance and correlation structure of the elements with each other. The covariance of an element with itself is nothing but just its Variance, which is the square of the standard deviation.<\/p>\n
That\u2019s why the diagonal elements of a covariance matrix are just the variance of the elements.<\/p>\n
# calculating the covariance matrix of the mean-centered data.\ncov_mat = np.cov(X_meaned , rowvar = False)<\/code><\/pre>\n
We can easily calculate the covariance Matrix using numpy.cov( )<\/code> a method. The default value for rowvar<\/code> is set to True<\/code>, remember to set it to False<\/code> to get the covariance matrix in the required dimensions.<\/p>\n
3. Compute the Eigenvalues and Eigenvectors<\/h3>\n
Now, compute the Eigenvalues and Eigenvectors for the calculated Covariance matrix. The Eigenvectors of the Covariance matrix we get are Orthogonal to each other and each vector represents a new axis called a principal axis.<\/p>\n
A Higher Eigenvalue corresponds to the most variance and higher variability in that direction. Hence, the principal axis with the higher Eigenvalue will be an axis capturing higher variability in the data.<\/p>\n
Orthogonal means the vectors are mutually perpendicular to each other. Eigenvalues and vectors seem to be very scary until we get the idea and concepts behind it. <\/p>\n
#Calculating Eigenvalues and Eigenvectors of the covariance matrix\neigen_values , eigen_vectors = np.linalg.eigh(cov_mat)<\/code><\/pre>\n
NumPy linalg.eigh( )<\/code> method returns the eigenvalues and eigenvectors of a complex Hermitian or a real symmetric matrix.<\/p>\n
4. Sort Eigenvalues in descending order<\/h3>\n
Sort the Eigenvalues in descending order along with their corresponding eigenvectors.<\/strong><\/p>\n
Remember, each column in the eigenvalue vector-matrix<\/strong> corresponds to a principal component, so arranging them in descending order of their Eigenvalue will automatically arrange the principal components in descending order of their variability.<\/p>\n
Hence the first column in our rearranged Eigen vector-matrix will be the first principal component that captures the highest variability.<\/p>\n
#sort the eigenvalues in descending order\nsorted_index = np.argsort(eigen_values)[::-1]\n\nsorted_eigenvalue = eigen_values[sorted_index]\n#similarly sort the eigenvectors \nsorted_eigenvectors = eigen_vectors[:,sorted_index]<\/code><\/pre>\n
np.argsort returns an array of indices of the same shape, similar to how lambda x: x would work in sorting operations.<\/p>\n
5. Select a subset from the rearranged Eigenvalue matrix<\/h3>\n
Select a subset from the rearranged Eigenvalue matrix as per our need i.e. number_comp = 2. This means we selected the first two principal components. <\/p>\n
# select the first n eigenvectors, n is desired dimension\n# of our final reduced data.\n\nn_components = 2 #you can select any number of components.\neigenvector_subset = sorted_eigenvectors[:,0:n_components]<\/code><\/pre>\n
n_components = 2 means our final data should be reduced to just 2 variables. if we change it to 3 then we get our data reduced to 3 variables.<\/p>\n
6. Transform the data<\/h3>\n
Finally, perform a linear transformation of the data by having a dot product between the Transpose of the Eigenvector subset and the Transpose of the mean-centered data. By transposing the outcome of the dot product, the result we get is the data transformed into fewer dimensions from higher dimensions.<\/p>\n
#Transform the data \nX_reduced = np.dot(eigenvector_subset.transpose(),X_meaned.transpose()).transpose()<\/code><\/pre>\n
The final dimensions of X_reduced will be ( 20, 2 ), and originally the data was of higher dimensions ( 20, 5 ).<\/p>\n
Now we can visualize our data with the available tools we have. Hurray! Mission accomplished.<\/p>\n
Complete Code for Principal Component Analysis in Python<\/h2>\n
Now, let\u2019s just combine everything above by making a function and try our Principal Component analysis from scratch on an example.<\/p>\n
import numpy as np\n\ndef PCA(X , num_components):\n    \n    #Step-1\n    X_meaned = X - np.mean(X , axis = 0)\n    \n    #Step-2\n    cov_mat = np.cov(X_meaned , rowvar = False)\n    \n    #Step-3\n    eigen_values , eigen_vectors = np.linalg.eigh(cov_mat)\n    \n    #Step-4\n    sorted_index = np.argsort(eigen_values)[::-1]\n    sorted_eigenvalue = eigen_values[sorted_index]\n    sorted_eigenvectors = eigen_vectors[:,sorted_index]\n    \n    #Step-5\n    eigenvector_subset = sorted_eigenvectors[:,0:num_components]\n    \n    #Step-6\n    X_reduced = np.dot(eigenvector_subset.transpose() , X_meaned.transpose() ).transpose()\n    \n    return X_reduced<\/code><\/pre>\n
We defined a function implementing the PCA algorithm that accepts a data matrix and the number of components as input arguments.<\/p>\n
We’ll use the IRIS dataset<\/a> as our sample dataset and apply our PCA function to it<\/p>\n
import pandas as pd\n\n#Get the IRIS dataset\nurl = \"https:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/iris\/iris.data\"\ndata = pd.read_csv(url, names=['sepal length','sepal width','petal length','petal width','target'])\n\n#prepare the data\nx = data.iloc[:,0:4]\n\n#prepare the target\ntarget = data.iloc[:,4]\n\n#Applying it to PCA function\nmat_reduced = PCA(x , 2)\n\n#Creating a Pandas DataFrame of reduced Dataset\nprincipal_df = pd.DataFrame(mat_reduced , columns = ['PC1','PC2'])\n\n#Concat it with target variable to create a complete Dataset\nprincipal_df = pd.concat([principal_df , pd.DataFrame(target)] , axis = 1)<\/code><\/pre>\n
Important Tip:<\/strong> Data standardization is crucial wherever necessary before applying any ML algorithm to it. In the above code, we did not standardize our data, but we did so while implementing PCA.<\/p>\n
Let’s plot our results using the seaborn<\/a> and matplotlib<\/a> libraries.<\/p>\n
import seaborn as sb\nimport matplotlib.pyplot as plt\n\nplt.figure(figsize = (6,6))\nsb.scatterplot(data = principal_df , x = 'PC1',y = 'PC2' , hue = 'target' , s = 60 , palette= 'icefire')<\/code><\/pre>\n
\"Reduced
Reduced Dimension Plot<\/strong><\/figcaption><\/figure>\n
That’s it! It worked perfectly. <\/p>\n
Conclusion<\/h2>\n
In this article, we learned about PCA, how PCA works, and implemented PCA using NumPy<\/a>. Happy learning!<\/p>\n","protected":false},"excerpt":{"rendered":"
Principal component analysis, or PCA in short, is famously known as a dimensionality reduction technique. It has been around since 1901 and is still used as a predominant dimensionality reduction method in machine learning and statistics. PCA is an unsupervised statistical method. In this article, we will have some intuition about PCA and will implement […]<\/p>\n","protected":false},"author":6,"featured_media":65273,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-9523","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-examples"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/9523","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/comments?post=9523"}],"version-history":[{"count":0,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/9523\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media\/65273"}],"wp:attachment":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media?parent=9523"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/categories?post=9523"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/tags?post=9523"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}