Linear Regression from Scratch

Batch gradient descent linear regression implemented from first principles — no ML frameworks, full convergence diagnostics.

A complete linear regression implementation built from the math up: batch gradient descent, cost tracking, and convergence diagnostics, with a CLI and plots. No scikit-learn — the point is to show the mechanics, not import them.

Anyone can call .fit(). This exists to demonstrate understanding of what .fit() is doing: the gradient, the update rule, why the learning rate matters, and how to tell whether the thing actually converged.

🔵 Complete — a foundations / from-scratch reference piece.

This implements batch gradient descent for simple (univariate) linear regression:

1. Initialize slope m = 0 and intercept b = 0
2. Forward pass — compute predictions: ŷ = mx + b
3. Cost — compute Mean Squared Error: MSE = (1/n) Σ(y − ŷ)²
4. Gradients — compute partial derivatives ∂MSE/∂m and ∂MSE/∂b
5. Update — m ← m − α·(∂MSE/∂m) and b ← b − α·(∂MSE/∂b)
6. Repeat until MSE < tolerance or max_iter is reached

where α is the learning rate.

Test_Dataset.csv is a synthetic 10-point dataset with two columns (x, y):

The data closely follows y ≈ 100x with minor noise (two rows are swapped), producing an analytical least-squares solution of approximately slope = 98.79, intercept = 6.67.

Requirements: Python 3.8+

# 1. (Recommended) create and activate a virtual environment
python -m venv venv
source venv/bin/activate        # macOS / Linux
venv\Scripts\activate           # Windows

# 2. Install dependencies
pip install -r requirements.txt

python LinearReg.py --file Test_Dataset.csv

With explicit hyperparameters:

python LinearReg.py --file Test_Dataset.csv --lr 0.01 --tol 1e-3 --max-iter 10000

python LinearReg.py --xvals 1,2,3,4,5 --yvals 2,4,6,8,10 --lr 0.01 --tol 1e-3

--file and --xvals are mutually exclusive — you must use one or the other.

Terminal:

Converged at iteration 847, MSE=0.000931
Result: slope = 98.7879, intercept = 6.6667

Chart 1 — Regression Fit: A scatter plot of the input data overlaid with the fitted line y = mx + b. The equation is printed in the legend.

Chart 2 — Cost vs. Iteration: A line plot of MSE at each gradient descent step. A healthy run shows MSE decreasing monotonically toward the convergence threshold.

Python · NumPy · Matplotlib