Python Web Scraping Basics

What is Web Scraping?

Web scraping means programmatically fetching a page's raw HTML and then parsing that markup to extract the specific data you need, rather than reading it visually in a browser.

This tutorial uses BeautifulSoup for parsing -- install it first with pip install beautifulsoup4.

Example: What is Web Scraping?

# pip install beautifulsoup4
from bs4 import BeautifulSoup

html = "<html><body><h1>Title</h1></body></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.h1.text)

Finding Tags and Titles

BeautifulSoup's .find(tag_name) locates the first matching HTML element anywhere in the parsed document and gives you access to its text and attributes, which is the basic building block for pulling out things like a page's main heading or title.

Example: Finding Tags and Titles

from bs4 import BeautifulSoup

html = "<html><head><title>My Page</title></head></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.find("title").text)  # locates the first <title> element and reads its text

Filtering with Classes

Since real-world pages rely heavily on CSS classes to distinguish elements that share the same tag, .find(tag, class_=name) lets you narrow a search to only elements carrying a specific class -- essential for scraping structured content like article titles or price tags that share a tag name with unrelated elements.

Example: Filtering with Classes

from bs4 import BeautifulSoup

html = '<div class="price">$10</div><div>Other</div>'
soup = BeautifulSoup(html, "html.parser")
print(soup.find("div", class_="price").text)  # narrows the search to a specific CSS class

Finding Multiple Elements

.find_all(tag) returns every matching element on the page as a list, rather than stopping at the first match like .find() does.

Use it whenever you need all instances of something -- every link, every list item, every row in a table -- not just one.

Example: Finding Multiple Elements

from bs4 import BeautifulSoup

html = "<ul><li>One</li><li>Two</li></ul>"
soup = BeautifulSoup(html, "html.parser")
items = soup.find_all("li")  # returns every matching element, not just the first
print([item.text for item in items])

Extracting Link Attributes

Links live in <a> tags, and the actual destination URL is stored in the tag's href attribute, accessed like a dictionary key: link[href].

Forgetting this and trying to read the link's visible text as if it were the URL is a common scraping mistake.

Example: Extracting Link Attributes

from bs4 import BeautifulSoup

html = '<a href="https://example.com">Visit</a>'
soup = BeautifulSoup(html, "html.parser")
link = soup.find("a")
print(link["href"])  # the destination URL lives in the href attribute