{"id":64292,"date":"2025-05-03T13:20:39","date_gmt":"2025-05-03T13:20:39","guid":{"rendered":"https:\/\/www.askpython.com\/?p=64292"},"modified":"2025-11-19T13:21:53","modified_gmt":"2025-11-19T13:21:53","slug":"python-beautifulsoup-web-scraping-example","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/python-beautifulsoup-web-scraping-example","title":{"rendered":"Python BeautifulSoup Web Scraping Example"},"content":{"rendered":"\n

I was working on a Python project the other day when I needed to grab some product prices from an online store. Going through pages manually would have taken hours, but Python made it simple – just a few lines of code with BeautifulSoup and requests, and I had all the data I needed in minutes.<\/p>\n\n\n\n

Web scraping automates the process of extracting data from websites. When you visit a webpage, your browser receives HTML content from a server. Web scraping tools follow the same process – they send HTTP requests, receive HTML responses, and parse that data for specific information.<\/p>\n\n\n\n

Web scraping turns manual data collection into automated workflows, saving developers countless hours of repetitive work. Python’s BeautifulSoup library makes this process straightforward by providing intuitive methods to navigate HTML structures and extract desired content.<\/em><\/p><\/blockquote><\/figure>\n\n\n\n

Setting Up Your BeautifulSoup Environment<\/h2>\n\n\n\n

Before you start scraping websites, you’ll need three essential libraries:<\/p>\n\n\n

\npip3 install requests\npip3 install beautifulsoup4\npip3 install html5lib\n\n<\/pre><\/div>\n\n\n
The requests<\/code> library handles HTTP requests to websites. BeautifulSoup parses HTML content, while html5lib acts as the parsing engine that creates a tree structure from raw HTML.<\/p>\n\n\n
\nimport requests\nfrom bs4 import BeautifulSoup\n\n<\/pre><\/div>\n\n\n
These imports give you everything needed to start extracting web data.<\/p>\n\n\n\n
Making Your First Request with Python bs4<\/h2>\n\n\n\n
Every web scraping project starts with fetching a webpage’s HTML content:<\/p>\n\n\n
\nURL = "https:\/\/example.com"\nresponse = requests.get(URL)\nprint(response.content)\n\n<\/pre><\/div>\n\n\n
This code sends a GET request to the specified URL and stores the server’s response. The response.content<\/code> contains the raw HTML of the webpage.<\/p>\n\n\n\n
Sometimes servers block automated requests. Adding a user agent header mimics a real browser:<\/p>\n\n\n
\nheaders = {\n    'User-Agent': 'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36'\n}\nresponse = requests.get(URL, headers=headers)\n\n<\/pre><\/div>\n\n\n
Parsing HTML with BeautifulSoup<\/h2>\n\n\n\n
Raw HTML isn’t very useful on its own. BeautifulSoup transforms it into a navigable structure:<\/p>\n\n\n
\nsoup = BeautifulSoup(response.content, 'html5lib')\nprint(soup.prettify())\n\n<\/pre><\/div>\n\n\n
The prettify()<\/code> method displays the HTML with proper indentation, making it readable:<\/p>\n\n\n
\n<html>\n <body>\n  <div class="product">\n   <h2>Product Name<\/h2>\n   <span class="price">$29.99<\/span>\n  <\/div>\n <\/body>\n<\/html>\n\n<\/pre><\/div>\n\n\n
BeautifulSoup creates a parse tree where you can search for specific elements using tag names, classes, IDs, or attributes.<\/p>\n\n\n\n
Finding Elements<\/h2>\n\n\n\n
BeautifulSoup offers two main methods for locating elements:<\/p>\n\n\n\n
The find() Method<\/h3>\n\n\n\n
Returns the first matching element:<\/p>\n\n\n
\n# Find first div with class 'product'\nproduct = soup.find('div', class_='product')\n\n# Find element by ID\nheader = soup.find('div', id='header')\n\n# Find by multiple attributes\nspecial_item = soup.find('div', attrs={'class': 'product', 'data-sale': 'true'})\n\n<\/pre><\/div>\n\n\n
The find_all() Method<\/h3>\n\n\n\n
Returns all matching elements as a list:<\/p>\n\n\n
\n# Find all product divs\nproducts = soup.find_all('div', class_='product')\n\n# Find all links\nlinks = soup.find_all('a')\n\n# Limit results\nfirst_five_products = soup.find_all('div', class_='product', limit=5)\n\n<\/pre><\/div>\n\n\n
Extracting Data<\/h2>\n\n\n\n
Once you’ve found elements, you can extract their content and attributes:<\/p>\n\n\n
\nproduct = soup.find('div', class_='product')\n\n# Get text content\nproduct_name = product.h2.text\nprice = product.find('span', class_='price').text\n\n# Get attributes\nimage_url = product.img['src']\nproduct_link = product.a['href']\n\n# Handle missing elements safely\ndescription = product.find('p', class_='description')\nif description:\n    desc_text = description.text\nelse:\n    desc_text = "No description available"\n\n<\/pre><\/div>\n\n\n
Reading the Parse Tree<\/h2>\n\n\n\n
BeautifulSoup lets you move through HTML elements using dot notation:<\/p>\n\n\n
\n# Access nested elements\ncontainer = soup.find('div', class_='container')\nfirst_product = container.div\nproduct_title = first_product.h2.text\n\n# Navigate siblings\nnext_product = first_product.find_next_sibling('div')\nprevious_product = first_product.find_previous_sibling('div')\n\n# Navigate parents\nproduct_section = first_product.parent\n\n<\/pre><\/div>\n\n\n
Scraping Product Data<\/h2>\n\n\n\n
Let’s build a complete scraper that extracts product information. I’m using a sample website with clean HTML for demonstration but you can apply the methods similarly to most websites. <\/p>\n\n\n
\nimport requests\nfrom bs4 import BeautifulSoup\nimport csv\n\ndef scrape_products(url):\n    headers = {'User-Agent': 'Mozilla\/5.0'}\n    response = requests.get(url, headers=headers)\n    soup = BeautifulSoup(response.content, 'html5lib')\n    \n    products = []\n    \n    # Find all product containers\n    for item in soup.find_all('div', class_='product-item'):\n        product = {}\n        \n        # Extract product details\n        product['name'] = item.find('h3', class_='product-name').text.strip()\n        product['price'] = item.find('span', class_='price').text.strip()\n        product['url'] = item.find('a')['href']\n        \n        # Handle optional fields\n        rating = item.find('div', class_='rating')\n        product['rating'] = rating.text.strip() if rating else 'No rating'\n        \n        products.append(product)\n    \n    return products\n\n# Scrape and save data\nproducts = scrape_products('https:\/\/example-shop.com\/products')\n\n# Save to CSV\nwith open('products.csv', 'w', newline='', encoding='utf-8') as file:\n    writer = csv.DictWriter(file, fieldnames=['name', 'price', 'url', 'rating'])\n    writer.writeheader()\n    writer.writerows(products)\n\n<\/pre><\/div>\n\n\n
Sample output (products.csv):<\/p>\n\n\n
\nname,price,url,rating\nWireless Headphones,$79.99,\/products\/wireless-headphones,4.5 stars\nSmart Watch,$199.99,\/products\/smart-watch,4.8 stars\nBluetooth Speaker,$49.99,\/products\/bluetooth-speaker,No rating\n\n<\/pre><\/div>\n\n\n
Handling Dynamic Content<\/h2>\n\n\n\n
Many modern websites load content dynamically. BeautifulSoup works with static HTML only. For dynamic content, you’ll need to:<\/p>\n\n\n\n
    \n
  1. Check the Network tab in browser developer tools<\/li>\n\n\n\n
  2. Find API endpoints that return JSON data<\/li>\n\n\n\n
  3. Use requests to call these APIs directly<\/li>\n<\/ol>\n\n\n
    \n# Example: Scraping from an API\napi_url = "https:\/\/example.com\/api\/products"\nresponse = requests.get(api_url)\nproducts = response.json()\n\nfor product in products['items']:\n    print(f"{product['name']}: ${product['price']}")\n\n<\/pre><\/div>\n\n\n
    Best Practices<\/h2>\n\n\n\n
    Web scraping requires responsible behavior:<\/p>\n\n\n\n
      \n
    1. Respect robots.txt<\/strong>: Check if the website allows scraping<\/li>\n\n\n\n
    2. Add delays<\/strong>: Space out requests to avoid overwhelming servers<\/li>\n\n\n\n
    3. Handle errors gracefully<\/strong>: Websites change, elements disappear<\/li>\n\n\n\n
    4. Cache responses<\/strong>: Store data locally to minimize requests<\/li>\n<\/ol>\n\n\n
      \nimport time\nfrom requests.exceptions import RequestException\n\ndef safe_scrape(url, delay=1):\n    try:\n        response = requests.get(url, timeout=10)\n        response.raise_for_status()\n        time.sleep(delay)  # Be polite\n        return BeautifulSoup(response.content, 'html5lib')\n    except RequestException as e:\n        print(f"Error scraping {url}: {e}")\n        return None\n\n<\/pre><\/div>\n\n\n
      Common Pitfalls<\/h2>\n\n\n\n
      Watch out for these issues:<\/p>\n\n\n\n
        \n
      • Class names change<\/strong>: Websites update their HTML structure<\/li>\n\n\n\n
      • IP blocking<\/strong>: Too many requests trigger security measures<\/li>\n\n\n\n
      • Legal concerns<\/strong>: Some sites prohibit scraping in their terms<\/li>\n\n\n\n
      • Encoding issues<\/strong>: Handle special characters properly<\/li>\n<\/ul>\n\n\n
        \n# Handle encoding\nsoup = BeautifulSoup(response.content, 'html5lib', from_encoding='utf-8')\n\n# Deal with special characters\ntext = element.text.encode('ascii', 'ignore').decode('ascii')\n\n<\/pre><\/div>\n\n\n
        Web scraping opens up countless possibilities for data collection and automation. Start with simple projects, respect website policies, and gradually tackle more complex scraping challenges as you gain experience with BeautifulSoup’s powerful features.<\/p>\n","protected":false},"excerpt":{"rendered":"
        I was working on a Python project the other day when I needed to grab some product prices from an online store. Going through pages manually would have taken hours, but Python made it simple – just a few lines of code with BeautifulSoup and requests, and I had all the data I needed in […]<\/p>\n","protected":false},"author":6,"featured_media":64296,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-64292","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/64292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/comments?post=64292"}],"version-history":[{"count":0,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/posts\/64292\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media\/64296"}],"wp:attachment":[{"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/media?parent=64292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/categories?post=64292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.askpython.com\/wp-json\/wp\/v2\/tags?post=64292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}