Python Web Scraping Basics
In this page:
What is Web Scraping?
Web scraping means programmatically fetching a page's raw HTML and then parsing that markup to extract the specific data you need, rather than reading it visually in a browser. This tutorial uses BeautifulSoup for parsing -- install it first with pip install beautifulsoup4.
Example: What is Web Scraping?
# pip install beautifulsoup4
from bs4 import BeautifulSoup
html = "<html><body><h1>Title</h1></body></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.h1.text)Finding Tags and Titles
BeautifulSoup's .find(tag_name) locates the first matching HTML element anywhere in the parsed document and gives you access to its text and attributes, which is the basic building block for pulling out things like a page's main heading or title.
Example: Finding Tags and Titles
from bs4 import BeautifulSoup
html = "<html><head><title>My Page</title></head></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.find("title").text)Filtering with Classes
Since real-world pages rely heavily on CSS classes to distinguish elements that share the same tag, .find(tag, class_=name) lets you narrow a search to only elements carrying a specific class -- essential for scraping structured content like article titles or price tags that share a tag name with unrelated elements.
Example: Filtering with Classes
from bs4 import BeautifulSoup
html = '<div class="price">$10</div><div>Other</div>'
soup = BeautifulSoup(html, "html.parser")
print(soup.find("div", class_="price").text)Finding Multiple Elements
.find_all(tag) returns every matching element on the page as a list, rather than stopping at the first match like .find() does. Use it whenever you need all instances of something -- every link, every list item, every row in a table -- not just one.
Example: Finding Multiple Elements
from bs4 import BeautifulSoup
html = "<ul><li>One</li><li>Two</li></ul>"
soup = BeautifulSoup(html, "html.parser")
items = soup.find_all("li")
print([item.text for item in items])Extracting Link Attributes
Links live in <a> tags, and the actual destination URL is stored in the tag's href attribute, accessed like a dictionary key: link[href]. Forgetting this and trying to read the link's visible text as if it were the URL is a common scraping mistake.
Example: Extracting Link Attributes
from bs4 import BeautifulSoup
html = '<a href="https://example.com">Visit</a>'
soup = BeautifulSoup(html, "html.parser")
link = soup.find("a")
print(link["href"])Chapter Quiz — Complete all 14 topics to unlock
0/14 topics done
Complete these topics first:
- Python NumPy Introduction
- Python NumPy Arrays
- Python Pandas Introduction
- Python Pandas DataFrame
- Python Matplotlib Basics
- Python Data Visualization
- Python Statistics Module
- Python CSV & Data Analysis
- Python requests Module
- Python JSON & APIs
- Python Web Scraping Basics
- Python Flask Introduction
- Python Django Introduction
- Python MongoDB