Alright, I won’t waste time explaining why you need this guide. Instead, I’ll keep it simple and show you the easiest ways to get the data you need from any website using a Python script.

There are basically four main approaches:

1. Basic request & parsing. You write a script using a library like requests, urllib, or http.client to fetch the page, then parse the result using tools like BeautifulSoup or lxml. The downside? This won’t work on sites that rely heavily on JavaScript.
2. Simulating a real user. If the first method fails, you’ll need something more advanced — like browser automation. Libraries such as Selenium, Pyppeteer, or Playwright let you control a real browser, so you can interact with JavaScript-heavy pages.
3. Using a dedicated scraping framework. This is where Scrapy comes in. Technically, it could fit into one of the first two categories, but since it’s a self-sufficient framework designed specifically for web scraping, it deserves its own mention.
4. The easiest option — using an API. Some services scrape the data for you, so all you need to do is send a request and process the response.

Before we jump into examples, just a heads-up: if you’re using one of the first three methods, you’ll have to deal with captchas, proxies, and other stuff websites use to block scrapers. If that sounds like too much trouble, just go with an API that does the work for you. My personal recommendation? HasData. But yeah, you probably guessed that.

Alright, enough intro. Let’s get to the actual guide.

Requests & BeautifulSoup

The easiest way to start scraping something is with the Requests library and BeautifulSoup. It works well, but honestly, it’s only good for the simplest and most static websites, which are getting rarer these days. Still, it’s a great way to get your feet wet with scraping.

So, to make things easier, I uploaded a scraping script using Requests and BeautifulSoup on Google Colab. This way, you don’t even need to install Python on your computer to run it.

But if you want to run it on your own machine, let’s dive in. Generally, both Requests and BeautifulSoup come pre-installed, but if you’re using virtual environments, you might need to install them yourself with this command:

pip install requests beautifulsoup4

After that, create a new file and import them into your script like this:

import requests
from bs4 import BeautifulSoup

Then, you’ll need to set the URL for the page you want to scrape:

url = "https://example.com"

Next, just go to that link and parse the data using CSS selectors:

response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.text, "html.parser")
title = soup.find_all("h1")

And that’s pretty much it! If you want a slightly more complex example, check out the Google Colab notebook. I added headers (just make sure to change the User Agents to something current), and I also included a check to see if the request was successful. If it is, we gather all the links from the page; if not, we print out an error message.

Overall, it looks like this:

Example with Scrapy

Let’s talk about the Scrapy framework. To be honest, I’m not a huge fan, but it’s definitely worth mentioning because it’s a solid framework for scraping. First things first, let’s install Scrapy:

pip install scrapy

Next, we’ll create a new project:

scrapy startproject scrapy_example
cd scrapy_example
scrapy genspider titles_scrapy_example example.com

It should look something like this:

Then, open the file of the spider you just created (Scrapy calls them spiders) and edit it like this:

def parse(self, response):
    titles = response.css("h1").extract()
    yield {'titles': titles}

Right here:

Or, if you prefer using XPath, you can do it this way:

def parse(self, response):
    titles = response.xpath("//h1/text()").extract()
    yield {'titles': titles}

Just a quick note:

You can have multiple spiders in your project, not just one. To run your entire project with all its spiders, use this command:

scrapy crawl titles_scrapy_example

Just make sure to replace it with your project name. It looks like this:

A Bit About Headless Browsers

So, let’s talk about my favorite topic — headless browsers. I’ll start with the less popular ones and work my way to the one I’ll use for an example. First up is Playwright. It’s not super famous, but I’ve heard that testers really know their stuff when it comes to it 😉

Then there’s Pyppeteer, which is basically a wrapper for the popular NodeJS tool, Puppeteer. It’s not super popular in Python, but if you want to make your script asynchronous, this is where you’d want to go.

And lastly, the big one that you probably guessed I’d talk about — Selenium. There are times when even Selenium struggles, like when you’re trying to scrape data from Ahrefs. But no worries, there’s SeleniumBase with its cool UC mode that does a great job of hiding the fact that it’s a bot.

But today, that’s not really what we’re focusing on. If you’re interested in that, just let me know, and I’ll definitely write about it later. For now, let’s stick with a simple example of launching a web driver, navigating to a page, and scraping some data.

First, as always, you need to install it:

pip install selenium

Then, import Selenium and all the necessary modules:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

Now, let’s create a driver object, go to a webpage, and get some data:

url = "https://example.com"

chrome_options = Options() 
driver = webdriver.Chrome(options=chrome_options)

driver.get(url)

paragraphs = driver.find_elements(By.CSS_SELECTOR, 'p')
for paragraph in paragraphs:
    print(paragraph.text)

And don’t forget to close the web driver at the end:

driver.quit()

So, that’s how it looks:

The Tastiest Part — A Web Scraping API Example

If you’re as lazy as I am, then the easiest way is to use a ready-made API that fetches all the data for you and returns it in a nice, structured way. As you’ve probably guessed, the example here will be using HasData’s API.

Sure, we could scrape some random website ourselves, but honestly, that’s not very exciting. If you’re curious about scraping a specific site, feel free to drop a comment or message me on LinkedIn — I might write about it.

Anyway, back to the point. I want to show how to use an API with an example of the SERP API. This feels especially relevant after Google’s update on January 16.

I’ve also uploaded a working script to Google Colaboratory, so you can check it out there right away.

We only need the requests library and a HasData API key. First, we import the library and set the endpoint:

import requests

base_url = "https://api.hasdata.com/scrape/google/serp"

Then, we define the parameters, including the HasData API key:

query = "Coffee"
location = "Austin,Texas,United+States"
device_type = "desktop"
api_key = "YOUR-API-KEY"

Next, we set the headers, make the request, and get the data:

url = f"{base_url}?q={query}&location={location}&deviceType={device_type}"
headers = {
    'Content-Type': 'application/json',
    'x-api-key': api_key
}
response = requests.get(url, headers=headers)

data = response.json()

Finally, we print the data:

print(data)

For convenience, I’ve uploaded a version in Colab Research that saves the data as well:

Wrapping Up

If you want more details, examples, or explanations about these libraries, check out HasData — I’ve already written about how to get started with Python and compared different web scraping libraries.

Or just drop a comment about what you’d like to read next, and I’ll keep it in mind for my next article.