Python Urllib Module

Last Updated : 17 Jul, 2026

urllib module is a built-in Python package used for working with URLs (Uniform Resource Locators). It provides utilities to fetch web pages, parse URLs, handle URL-related errors, and work with robots.txt files. The urllib package is commonly used for basic web requests and URL manipulation.

Installation

Before using urllib module, install it using below command in cmd or terminal:

import urllib.request

urllib Modules

The urllib package is divided into the following modules, each serving a specific purpose:

  • urllib.request: Opens URLs and retrieves data from web servers.
  • urllib.parse: Parses, joins, and encodes URLs.
  • urllib.error: Handles exceptions raised during URL operations.
  • urllib.robotparser: Reads and interprets robots.txt files.

urllib.request

The urllib.request module is used to open URLs and retrieve data from web servers. The urlopen() function sends a request to the specified URL and returns the server's response.

Python 
import urllib.request

response = urllib.request.urlopen("https://example.com")
print(response.status)
print(response.read()[:200])

Output

200
b'<!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,"><meta name="viewport" content="width=device-width, initial-scale=1"><style>body{background:#eee;width:6'

Explanation:

  • Imports the urllib.request module.
  • Opens the specified URL using urlopen().
  • Prints the HTTP status code.
  • Displays the first few bytes of the webpage content.

urllib.parse

The urllib.parse module provides functions for parsing, creating, and modifying URLs. It is commonly used to split a URL into its components or combine components back into a complete URL.

Python 
from urllib.parse import urlparse

url = "https://example.com/products/item?id=101"
parsed_url = urlparse(url)

print(parsed_url)
print(parsed_url.scheme)
print(parsed_url.netloc)
print(parsed_url.path)
print(parsed_url.query)

Output

ParseResult(scheme='https', netloc='example.com', path='/products/item', params='', query='id=101', fragment='')
https
example.com
/products/item
id=101

Explanation:

  • Parses the given URL into its individual components.
  • Displays the protocol (scheme), domain (netloc), path, and query string.

Common Functions in urllib.parse

  • urlparse(): Splits a URL into its components.
  • urlunparse(): Combines URL components into a complete URL.
  • urlsplit(): Similar to urlparse() but excludes parameters.
  • urlunsplit(): Reconstructs a URL created by urlsplit().
  • urldefrag(): Removes the fragment (#section) from a URL.

urllib.error

The urllib.error module contains exception classes that help handle errors while opening URLs. The two most commonly used exceptions are:

  • URLError: Raised when a URL cannot be reached due to network or address-related issues.
  • HTTPError: Raised when the server returns an HTTP error such as 404, 403, or 500.

Example: Handling URL Errors

Python 
import urllib.request
import urllib.error

try:
    response = urllib.request.urlopen("https://invalid-example-url.com")
    print(response.read())

except urllib.error.URLError as e:
    print("URL Error:", e.reason)

Output

URL Error: [Errno 11001] getaddrinfo failed

Explanation:

  • Attempts to open the given URL.
  • If the URL cannot be reached, a URLError exception is raised.
  • Prints the reason for the error instead of stopping the program.

Example: Handling HTTP Errors

Python 
import urllib.request
import urllib.error

try:
    urllib.request.urlopen("https://example.com/nonexistentpage")

except urllib.error.HTTPError as e:
    print("HTTP Error:", e.code)

Output

HTTP Error: 404

Explanation:

  • Tries to access a web page.
  • If the server returns an HTTP error, the program catches the HTTPError.
  • Prints the HTTP status code (such as 404 or 403).

urllib.robotparser

The urllib.robotparser module is used to read and interpret a website's robots.txt file. It helps determine whether a web crawler is allowed to access a specific URL.

Python 
import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://example.com/"))

Output

True

Explanation:

  • Creates a RobotFileParser object.
  • Loads the website's robots.txt file.
  • Reads the crawling rules.
  • Checks whether all user agents (*) are allowed to access the specified URL.
Comment

Explore