Web Scraping Basics — fetch, parse, and pipeline the web (without getting blocked)

"Sometimes the data you need lives behind a messy HTML page, not a shiny API. This is where web scraping turns chaos into a dataset."

---

Why this matters (and how it builds on what you already know)

You’ve already learned how to work with files and formats and to parse JSON and XML. Great — scraping is often the first step that generates those same JSON/XML files. Think of scraping as the gardener who collects the flowers (raw HTML) so you can press them into neat JSON herbarium specimens for your models (remember the Deep Learning Foundations module?). Web scraping turns human-facing web pages into machine-ready training data.

This guide shows practical, ethical, and deployable basics: how to fetch HTML or JSON endpoints, how to parse them, how to handle dynamic content, and how to make scraping part of a robust pipeline.

---

Quick roadmap

• Fetch: requests, headers, respecting robots.txt
• Parse: BeautifulSoup, CSS selectors, extracting JSON endpoints
• Dynamic content: Playwright / Selenium or API sniffing
• Pagination, rate limiting, and retries
• Storage: CSV, JSON Lines, databases
• Deploy: Docker, cron / serverless / Airflow, monitoring

---

1) Fetching HTML safely and politely

Micro explanation: The web server is a person too — be respectful.

Key points:

• Check robots.txt and site terms first. Robots.txt is instructions, not legal cover-all, but follow it.
• Use a sensible User-Agent. Don’t pretend to be a browser unless necessary.
• Rate-limit your requests and add jitter.
• Use retries with exponential backoff for transient failures.

Example (Python requests):

import requests
from time import sleep

headers = {"User-Agent": "MyDataScienceBot/1.0 (+https://example.com/)")}

for i in range(5):
    resp = requests.get("https://example.com/listing/page/1", headers=headers, timeout=10)
    if resp.status_code == 200:
        html = resp.text
        break
    sleep(2 ** i + (0.1 * i))  # exponential backoff with jitter

---

2) Parsing HTML — BeautifulSoup, but make it human

Micro explanation: HTML is nested text; think of it like a messy city map.

BeautifulSoup gives you a map and a flashlight.

Example: extract titles and prices

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")

items = []
for card in soup.select(".product-card"):
    title = card.select_one(".title").get_text(strip=True)
    price = card.select_one(".price").get_text(strip=True)
    items.append({"title": title, "price": price})

Pro tip: prefer CSS selectors (select, select_one) — they read like CSS and are easier to tweak.

---

3) Often there’s JSON hiding in plain sight

Many modern sites render data by fetching JSON endpoints. Open the Network tab, reload — magic. If there's a /api/items?page=1 returning JSON, use that instead of scraping HTML.

Example:

resp = requests.get("https://example.com/api/items?page=1", headers=headers)
data = resp.json()
# Now data is already structured — we skip HTML parsing entirely.

This ties directly to the JSON/XML parsing topic you studied — once you fetch JSON, the same parsing and validation techniques apply.

---

4) Dynamic content: when HTML is just a shell

If the page relies on JavaScript to populate content, you have options:

• Use a headless browser (Playwright or Selenium) to render JS.
• Find the underlying API the page calls and hit that instead (cleaner).
• Use network sniffing or DevTools to discover the endpoints.

Short Playwright example (fast and modern):

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dynamic")
    html = page.content()
    browser.close()

Use headless browsers sparingly — they’re heavier and slower. Prefer API endpoints when possible.

---

5) Pagination, rate limits, and pro tips

• Detect pagination by links or API page parameters.
• Respect server limits: max 1-2 requests/sec for smaller sites; use proxies for high-volume scraping.
• Rotate User-Agents and IPs only when necessary and ethically justified.
• Cache responses locally (ETag / Last-Modified) to avoid re-downloading unchanged content.

Retry pattern (requests + backoff) is your friend.

---

6) Storage & data engineering handoff

After extraction, save in clean, structured forms your pipeline expects:

• JSON Lines (.jsonl) for records streaming
• CSV for tabular content
• Store raw HTML or raw JSON for reproducibility
• Load into a database (Postgres, MongoDB) or object store (S3)

Example: write JSON Lines

import json
with open("data.jsonl", "w", encoding="utf-8") as f:
    for item in items:
        f.write(json.dumps(item, ensure_ascii=False) + "\n")

This output is exactly what your model training pipeline likes: cleaned, versioned records ready for feature extraction and batching into PyTorch datasets (linking to your Deep Learning Foundations work).

---

7) Deployment: make your scraper a citizen of production

Micro explanation: scraping in production = solid ops + good ethics.

Options:

• Cron in a Docker container for small jobs
• Cloud functions (AWS Lambda, Google Cloud Functions) for event-driven scraping
• Airflow or Prefect for complex DAGs and retries
• Kubernetes + CronJobs for scale

Add monitoring and alerts: track failures, 4xx/5xx spikes, and data drift (if extracted fields suddenly vanish).

---

8) Legal & ethical checklist (don’t skip this)

• Read site terms and copyright rules.
• Prefer public APIs when provided.
• Never attempt to bypass paywalls or authentication improperly.
• Rate-limit and avoid scraping sensitive personal data.
• When in doubt, ask the site owner.

---

Quick summary — what to remember

• Scraping is the bridge between messy HTML and structured JSON/CSV your models want.
• Prefer official APIs or JSON endpoints over HTML parsing.
• Use BeautifulSoup for static pages, Playwright for dynamic rendering when needed.
• Be polite: obey robots.txt, throttle requests, and implement retries and caching.
• Store data as JSONL/CSV or in a DB and integrate with your training pipeline (hello, PyTorch datasets).
• Deploy thoughtfully: Docker, serverless, or Airflow, with monitoring and error handling.

"Good scraping is boring: slow, polite, logged, and well-tested. Bold scraping gets you blocked."

---

Next steps (practical exercises)

1. Pick a public site with an API (or an open HTML page). Extract 100 items into a JSONL file.
2. Build a small Docker image that runs your scraper and writes to S3 (or local storage).
3. Hook it into a scheduler and add basic retry/alerting.

This will connect what you’ve learned about file formats and JSON parsing to real-world data acquisition — the very same data that can train the models you learned to build in Deep Learning Foundations.

---

Key takeaways: be curious, be careful, and automate responsibly. Happy scraping!