Web Scraping with Python: BeautifulSoup Tutorial

steps

1. Create a Python script (scraper.py): Open your text editor or IDE and create a new Python file called scraper.py.

2. Import necessary libraries: At the beginning of the file, import the required Python modules:

   pythonCopy

   from bs4 import BeautifulSoup
   import requests
   import csv

3. Fetch an HTML file or use a local file for testing:

   • For live websites: Use the requests library to fetch the website's HTML content.
   • For local HTML files: Open and read the file into your script.

4. Load HTML content into BeautifulSoup: To parse and work with the HTML:

   pythonCopy

   with open("sample.html", "r") as html_file:
       content = html_file.read()
   soup = BeautifulSoup(content, 'lxml')

5. Identify the elements to scrape:

   • Use soup.find() or soup.find_all() to locate elements by HTML tag names or CSS class names.
   • For example:
   pythonCopy

   titles = soup.find_all('h5', class_='course-title')

6. Extract data: Extract the text or attributes from the HTML elements:

   pythonCopy

   for title in titles:
       print(title.text)

Each step above builds towards crafting a workable scraping script.

```python
from bs4 import BeautifulSoup
import requests
import csv

# Fetch the HTML of the webpage
url = 'http://example.com/sample-webpage'
response = requests.get(url)
html_content = response.text

# Parse the HTML content with BeautifulSoup
soup = BeautifulSoup(html_content, 'lxml')

# Extract data - e.g., course titles and prices
courses = soup.find_all('div', class_='course-card')
data = []
for course in courses:
    title = course.find('h5', class_='course-title').text.strip()
    price = course.find('span', class_='course-price').text.strip()
    data.append([title, price])

# Write the scraped data to a CSV file
with open('courses.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(['Title', 'Price'])
    writer.writerows(data)

print("Data saved to courses.csv.")
```

steps

1. Structure your data: Create a list of lists where each sublist represents a row of data:

   pythonCopy

   data = [['Python Basics', '$20'], ['Django Web Development', '$25']]

2. Write the data to CSV: Use Python's built-in csv module:

   pythonCopy

   import csv
   
   with open('courses.csv', 'w', newline='') as file:
       writer = csv.writer(file)
       writer.writerow(['Title', 'Price'])  # Header row
       writer.writerows(data)

3. Verify CSV output: Check the saved courses.csv file to ensure the data is formatted correctly.