Python on Web Scraping: 15 Proven Strategies to Master Data Extraction Like a Pro

What’s Inside?

Introduction to Python on Web Scraping

Python on web scraping has become the gold standard for automated data collection. Whether you are gathering e-commerce prices, research data, or market trends, Python provides a smooth and efficient ecosystem. But “scraping” isn’t just about pulling text; it’s about doing it at scale, legally, and without getting blocked.

But what exactly is web scraping?

What is Web Scraping?

Web scraping is the automated process of extracting data from websites. Instead of manual copy-pasting, a script retrieves the HTML content and pulls specific pieces of information like text, images, or links. It replaces tedious manual copying with high-speed, accurate data collection for use cases like price monitoring, market research, and AI training. 

The video above explains the basis of web scraping. Just think of web scraping as a digital assistant that reads web pages for you fast and without getting tired.

How Python on Web Scraping Works Step-by-Step

Web scraping involves four main steps:

1. Sending HTTP Requests

A script or scraper sends an HTTP request to the target website’s server asking for page content. The web server responds with the page’s HTML code, the raw markup that contains all the visible content, metadata, and structure you see in a browser.

2. Parsing HTML Content

The HTML is parsed into a DOM tree by the scraper. To enable the scraper to navigate to particular items, this stage generates a blueprint of the page’s headings, paragraphs, tables, and links. Imagine it like a map showing the locations of everything before you begin extracting.

3. Extracting Structured Data

Specific tags, IDs, or classes are identified and extracted. The scraper finds the precise items with the desired data using selectors such as XPath expressions or CSS selectors. Product prices, contact information, article text, and other items may be pulled. Only pertinent data is extracted by the scraper; advertisements, navigation menus, and other superfluous markup are removed.

4. Storing the Data

There are frequently formatting errors, unnecessary whitespace, or encoding problems in raw extracted data. This data is cleaned by the scraper, which also converts data types, normalizes text, and removes HTML tags before storing it in an organized manner. This last stage turns untidy online material into datasets that are ready for analysis. The data is then stored in CSV files, JSON format or databases like SQLite or PostgreSQL.

Key Components of Web Scrapping

There are three main components of web scraping which are:

  1. Web Scrapers/Bots: Software that automatically visits websites.
  2. Target Content: The information being collected, such as product prices, contact info, or reviews.
  3. Structured Output: Data organized into a usable format, often CSV or SQL database. 

Why Python is the Top Choice for Scraping?

Python remains a top-three language globally according to Stack Overflow because it offers:

Common Applications

There are various use cases where web scrapping can be used which include but are not limited to the following:

The Growth of Web Data Extraction

The global big data market is expected to exceed $655 billion by 2029, growing at a compound annual growth rate (CAGR) of over 13%, according to market research reports. A significant portion of this data originates from web-based sources.

Companies rely heavily on automated data extraction for:

In fact, studies show that over 60% of businesses use web scraping tools in some capacity for market intelligence.

15 Proven Strategies for Python Web Scraping

Strategy 1: Choose the Right Library

Comparison chart of BeautifulSoup, Scrapy, Selenium for web scraping

Not every project needs a heavy browser. There are various Python libraries you can explore with the most popular ones listed below:

BeautifulSoup and Requests (The Dynamic Duo)

BeautifulSoup is a parsing library that makes navigating HTML easy. It allows developers to extract specific tags, classes, and attributes from web pages. It is best for beginners for small-scale scraping projects and mostly static pages.

The Requests library handles HTTP requests. It allows you to retrieve web pages simply by sending GET or POST requests. Example usage includes fetching webpage content, handling headers and managing sessions.

This combination is the gold standard for beginners and simple, static data extraction.

Scrapy (The Powerhouse Framework)

Scrapy is a powerful scraping framework built for large-scale projects. It’s fast, asynchronous, and highly customizable. It is the go-to framework for enterprise-level, large-scale crawling best for:

Unlike the libraries above, Scrapy is designed for high-performance crawling. 

Playwright (The Modern Browser)

Developed by Microsoft, Playwright is a modern automation tool that controls a real browser (Chromium, Firefox, or WebKit). 

Selenium (The Established Veteran)

Selenium automates browser interaction. It is especially useful for scraping JavaScript-heavy websites that load content dynamically. It is the original browser automation tool and remains widely used despite newer competitors. 

Selenium is best for legacy enterprise systems, projects requiring specific older browser versions (like Internet Explorer), or when you need the largest possible pool of community support. Its key advantage is that it has a massive community and a decade of tutorials and plugins (like undetected-chromedriver) to help bypass anti-bot protections.

However, there is a trade off. It is generally slower and more resource-heavy than Playwright because its architecture requires more overhead to communicate with the browser.

Which Library Should you Use?

Library Best ForComplexityPerformance
Requests + BS4Static pages / BeginnersLowFast (for single pages)
ScrapyLarge-scale crawlingHighBlazing Fast (Async)
PlaywrightModern JS-heavy sitesMediumFast & Stable
SeleniumBrowser automation / LegacyMediumSlower / Resource-heavy

While BeautifulSoup is beginner-friendly, mastering a framework like Scrapy takes more time. If you’re wondering where this fits into your overall coding journey, check out our guide on how long it actually takes to master Python for different career paths.

Strategy 2: Check the Robots.txt File

Robots.txt observation when web scraping

The robots.txt file is the “polite doorman” of a website. It resides in the root directory (e.g., example.com/robots.txt) and follows the Robots Exclusion Protocol (REP) to give directives to automated agents. 

1. Key Directives to Look For

When you open a robots.txt file, you will encounter several critical fields:

Automating Compliance with Python

You don’t have to check these rules manually for every URL. Python’s standard library includes the urllib.robotparser module, which handles this automatically. Check out the example code given below:

from urllib.robotparser import RobotFileParser

# Initialize the parser
rp = RobotFileParser()
rp.set_url("https://www.example.com/robots.txt")
rp.read()

# Check if your bot is allowed to crawl a specific page
url_to_scrape = "https://www.example.com/products"
user_agent = "MyPythonScraper"

if rp.can_fetch(user_agent, url_to_scrape):
    print("Safe to scrape!")
else:
    print("Scraping restricted by robots.txt.")

Why It Matters: Ethics and Legal Risks

While robots.txt is technically a voluntary guideline rather than a legally binding contract, ignoring it carries significant risks: 

Strategy 3: Use Asynchronous Requests

Standard scraping is often synchronous, meaning the script sends one request and “blocks” (idles) while waiting for the server to respond before moving to the next task. In contrast, asynchronous scraping allows the program to send hundreds of requests almost simultaneously, processing each response the moment it arrives. 

1. The Performance Leap

The primary bottleneck in web scraping is I/O-bound latency the time spent waiting for a remote server to send data. 

Core Python Tools for Async Scraping

To master this strategy, you must move beyond the standard requests library, which does not natively support async operations. 

Implementation Best Practices

Example Snapshot: Async with aiohttp
import asyncio
import aiohttp

async def fetch_page(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_page(session, url) for url in urls]
        # Execute all requests concurrently
        pages = await asyncio.gather(*tasks)
        return pages

# Run the async scraper
urls = ["https://example.com/page1", "https://example.com/page2"]
results = asyncio.run(main(urls))

Strategy 4: Rotate User-Agents

User agent rotation

Every time your script sends a request, it includes a User-Agent (UA) string—a “digital ID card” that tells the server your browser version, operating system, and device type. By default, libraries like requests identify themselves as python-requests/2.x.x, which is a massive red flag for anti-bot systems.

1. Why Rotation is Mandatory?

If a server sees 1,000 requests in 60 seconds all coming from the exact same “Chrome 114 on Windows 10” ID, it will quickly flag that traffic as non-human. Rotation mimics a diverse group of users (some on iPhones, others on MacBooks or Androids), making your traffic patterns look natural and organic.

2. Automating with fake-useragent

Manually maintaining a list of UA strings is tedious because they become outdated quickly. The fake-useragent library solves this by providing a real-time database of the most common strings.

Example Implementation:
from fake_useragent import UserAgent
import requests

ua = UserAgent()

# Generate a random, high-quality User-Agent
headers = {'User-Agent': ua.random}

response = requests.get("https://example.com", headers=headers)
print(f"Scraping as: {headers['User-Agent']}")

3. Pro-Level Best Practices

4. The Hierarchy of Trust

When rotating, prioritize Desktop Chrome and Safari strings, as these are the most common. Avoid using mobile User-Agents if you are scraping a site that has a drastically different mobile layout (m.example.com), as this might break your CSS selectors.

Strategy 5: Implement Proxy Rotation

If User-Agents are your digital ID card, your IP Address is your home address. Even with a fake ID, if a server sees 5,000 requests coming from the same “house” in five minutes, it will slam the door and blacklist your IP. Proxy rotation solves this by routing your requests through a pool of intermediate servers, making it appear as though the traffic is coming from thousands of different locations worldwide.

1. Types of Proxies: Choose Your Armor

Not all proxies are created equal. Depending on your target, you’ll need to choose the right level of “stealth”:

2. How Rotation Works in Python

You can manage a proxy pool manually by passing a dictionary to your request, or use a “Backconnect Proxy” service that provides a single entry point and rotates the IP for you automatically.

Manual Rotation Example:
import requests
import random

proxy_pool = [
    "http://proxy1.com:8001",
    "http://proxy2.com:8002",
    "http://proxy3.com:8003",
]

def get_data(url):
    # Pick a random proxy from the list
    proxy = {"http": random.choice(proxy_pool), "https": random.choice(proxy_pool)}
    try:
        response = requests.get(url, proxies=proxy, timeout=5)
        return response.text
    except:
        print("Proxy failed, trying another...")

3. Pro-Level Strategies

4. Integration with Scrapy

For heavy-duty projects, use scrapy-proxies or a middleware provided by services like Bright Data or Oxylabs. These allow you to scale to millions of requests without writing custom rotation logic.

Strategy 6: Handle Dynamic Content with Headless Browsers

Many modern websites are built using frameworks like React, Vue, or Angular. These sites don’t send data in the initial HTML; instead, they load a blank template and use JavaScript to “fetch” the content after the page opens. Traditional scrapers see only an empty page. To extract this data, you need a browser that can actually execute JavaScript—but you don’t need the visual clutter.

1. What is “Headless” Mode?

A Headless Browser is a web browser without a Graphical User Interface (GUI). It does everything a normal browser does—renders CSS, executes JavaScript, and handles cookies—but it runs in the background.

2. Implementing Headless Selenium

In older versions of Selenium, setting up headless mode was clunky. In the latest versions, it is a simple flag.

Example Implementation:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Setup Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless=new") # The magic line
chrome_options.add_argument("--disable-gpu")  # Recommended for Windows

# Initialize driver
driver = webdriver.Chrome(options=chrome_options)

driver.get("https://www.dynamic-site.com")
print(f"Page Title: {driver.title}")

# Now you can scrape content that was generated by JavaScript
content = driver.find_element("id", "dynamic-list").text
driver.quit()

3. Pro-Level Best Practices

4. When to Use (and When to Skip)

Strategy 7: Use CSS Selectors over XPath

When you scrape a page, you need a way to tell your script exactly which element to grab. The two primary languages for this are XPath (XML Path Language) and CSS Selectors. While XPath is technically more “powerful” (it can navigate backwards up the tree), CSS Selectors are the preferred choice for professional developers building scalable scrapers.

1. The Speed and Readability Advantage

CSS selectors are the native language of web browsers and front-end developers.

2. Simpler Syntax for Common Tasks

Most scraping involves targeting IDs, classes, and attributes. CSS selectors make this effortless:

3. When to Make an Exception (The XPath Edge)

While CSS is the “pro” choice for 90% of tasks, you should keep XPath in your back pocket for two specific scenarios:

  1. Text-Based Searching: XPath can find an element based on the text it contains (e.g., //button[text()="Submit"]). CSS cannot do this.
  2. Parent Navigation: XPath allows you to move up the tree (e.g., “Find the price tag, then find the container it lives in”). CSS only moves down or sideways.

4. Pro-Level Implementation

In Python’s BeautifulSoup or Scrapy, using CSS selectors is straightforward:

BeautifulSoup Example:
# Using CSS Selector (Clean & Fast)
price = soup.select_one(".product-card .price").text

# Using XPath (Verbose)
# price = soup.find_element(By.XPATH, "//div[@class='product-card']//span[@class='price']").text
Scrapy Example:
# Pro tip: Scrapy allows you to chain them
title = response.css("h1.title::text").get()

Strategy 8: Add Random Delays

Computers are fast—too fast. A human takes several seconds to read a page, move the mouse, and click a link. A Python script can request 50 pages in a single second. This “hammering” effect is a massive red flag for Web Application Firewalls (WAFs) and can lead to an instant IP ban. Adding random delays (also known as “jitter”) is the simplest way to make your scraper feel human.

1. The Danger of Static Delays

Many beginners use a fixed delay like time.sleep(2). While this is better than nothing, it creates a perfectly consistent “heartbeat” pattern in the server logs. Modern anti-bot systems look for this mathematical regularity. To be a pro, your delays must be stochastic (randomly determined).

2. Implementing “Jitter” with Python

Using random.uniform() allows you to set a range, ensuring that every pause is slightly different.

Example Implementation:
import time
import random
import requests

urls = ["://site.com", "://site.com", "://site.com"]

for url in urls:
    response = requests.get(url)
    # Extract your data here...
    
    # Wait between 1.5 and 4.8 seconds
    wait_time = random.uniform(1.5, 4.8)
    print(f"Waiting for {wait_time:.2f} seconds...")
    time.sleep(wait_time)

3. Pro-Level Best Practices

4. Why it Matters for Ethics

Beyond avoiding bans, random delays are part of Scraping Etiquette. Sending too many requests too fast consumes the target website’s bandwidth and CPU, which can slow down the site for real human users. Being a “pro” means getting the data without breaking the source.

Strategy 9: Leverage “Requests” Sessions

When you use a standard requests.get(), Python opens a new connection to the server, fetches the data, and then immediately slams the connection shut. To the website, every single request looks like a brand-new visitor who just cleared their cache and cookies. This is inefficient, slow, and highly suspicious.

By using requests.Session(), you create a “persistence layer” that mimics a real browser session.

1. The Performance Boost: Connection Pooling

The “secret sauce” of a Session is TCP Connection Re-use.

2. Handling State: Cookies and Logins

Real users have “state.” If you log into a website on page A, the server gives you a cookie so you stay logged in on page B.

3. Pro-Level Implementation

Instead of calling requests.get(), you wrap your logic inside a with statement to ensure the session is cleaned up afterward.

Example Implementation:
import requests

# Create the session object
with requests.Session() as session:
    # Set headers once for the entire session
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)...',
        'Accept-Language': 'en-US,en;q=0.9'
    })

    # Step 1: Log in (the session saves the login cookie)
    login_data = {'user': 'my_username', 'pass': 'my_password'}
    session.post('https://example.com/login', data=login_data)

    # Step 2: Scrape protected data (the session sends the cookie back)
    response = session.get('https://example.com/dashboard')
    print(response.text)

4. Why Pros Use It

Strategy 10: Clean Data with Pandas

Selenium automating browser interaction for dynamic web scraping

Websites are designed for human eyes, not databases. When you scrape, you’ll often find “dirty” data: extra spaces, inconsistent date formats, hidden HTML characters (like \n or \t), and duplicate entries. If you don’t clean this immediately, your analysis or machine learning model will fail. Pandas is the industry-standard library for turning this mess into a clean, structured DataFrame.

1. Why “Scrape-and-Clean” is the Pro Workflow

Don’t wait until the end of a 10,000-page crawl to start cleaning. Pros clean data in-flight or immediately after a batch is finished. This allows you to catch errors early—for example, if a website changes its layout and you start scraping “None” values, Pandas can alert you instantly.

2. Common Data “Gunk” and the Pandas Fix

Here are the three most common issues you’ll face and how to solve them in one or two lines of code:

3. Pro-Level Implementation

Instead of saving to a CSV and then opening it, load your list of dictionaries directly into a Pandas DataFrame.

Example Implementation:
import pandas as pd

# Imagine this is the data you pulled with BeautifulSoup
scraped_data = [
    {"product": " Laptop ", "price": "$1,200", "date": "Jan 1, 2024"},
    {"product": "Laptop ", "price": "$1,200", "date": "2024-01-01"}, # Duplicate!
    {"product": " Smartphone", "price": "$800", "date": "2024/01/02"}
]

df = pd.DataFrame(scraped_data)

# 1. Strip whitespace
df['product'] = df['product'].str.strip()

# 2. Clean 'price' to be a number (remove '$' and ',')
df['price'] = df['price'].replace(r'[\$,]', '', regex=True).astype(float)

# 3. Standardize dates
df['date'] = pd.to_datetime(df['date'])

# 4. Remove duplicates
df.drop_duplicates(subset=['product', 'price'], inplace=True)

print(df)

4. Exporting Like a Pro

Once cleaned, Pandas makes it easy to move data to its final home. Whether it’s a PostgreSQL database, an Excel file, or a JSON blob, you can do it in a single command.

Strategy 11: Monitor for Layout Change

“Brittle” scripts are the hallmark of a beginner. If your code assumes an element will always be there, it will eventually crash and lose hours of progress. Professional scrapers are built with defensive programming—they expect things to go wrong and are designed to fail gracefully while alerting the developer.

1. The “Try-Except” Safety Net

Never let a missing element kill your entire process. Wrap your extraction logic in try-except blocks. This ensures that if one product on a page of fifty is missing a “Description” tag, the script logs the error and moves on to the forty-nine that do work.

Example Implementation:
import logging

# Set up a log file to track "missing" elements
logging.basicConfig(filename='scraper_errors.log', level=logging.ERROR)

def extract_product_data(element):
    try:
        # Attempt to find the price
        price = element.select_one(".price-tag").text
        return price
    except AttributeError:
        # The class ".price-tag" wasn't found! 
        logging.error(f"Layout Change Detected: Could not find price on {url}")
        return "N/A" # Default value so the script keeps running

2. Automated “Health Checks”

Don’t wait for the script to finish to find out the data is empty. Pros implement Validation Checks at the start of a run.

3. Pro-Level Monitoring Tools

If you are scraping at an enterprise level, you don’t just want a log file—you want a dashboard.

4. Why it Matters for Reliability

Monitoring isn’t just about fixing bugs; it’s about Data Integrity. There is nothing worse than running a scraper for 12 hours only to realize the website changed its HTML in hour one, leaving you with a CSV full of empty rows.

Strategy 12: Use an API First

The first rule of web scraping is: Don’t scrape if you don’t have to. Before you spend hours writing complex CSS selectors or rotating proxies, check if the website provides an official API (Application Programming Interface). An API is a door specifically built for developers to access data in a clean, structured, and legal way.

Before you start writing complex scrapers, remember that many platforms offer a direct way to access data. We’ve broken down exactly how to navigate this choice in our article on leveraging Python for Web Scraping and APIs, which explains why an API is often the faster, more stable route.

1. Why APIs Beat Scraping Every Time?

2. How to Find “Hidden” and Official APIs

3. Working with APIs in Python

Python makes API interaction incredibly simple. You don’t need BeautifulSoup or Selenium; you only need requests.

Example Implementation:
import requests

# Example: Fetching data from a public API
api_url = "https://api.coingecko.com"

response = requests.get(api_url)

if response.status_code == 200:
    data = response.json() # Automatically converts JSON to a Python dictionary
    price = data['bitcoin']['usd']
    print(f"The current price of Bitcoin is: ${price}")
else:
    print("API request failed.")

4. Pro-Level Best Practices

Strategy 13: Store Data in Structured Formats

The value of a scraper isn’t in the code; it’s in the output. Beginners often make the mistake of simply printing results to the console or saving everything into a single, messy text file. Professional scraping requires choosing a storage format that matches the shape of your data and the needs of your end-user.

1. Choosing the Right Format

2. Why “In-Memory” Storage is Dangerous

Never keep all your scraped data in a Python list until the end of the script. If the scraper crashes on page 99 of 100, you lose everything.

3. Pro-Level Implementation with Python

Python’s built-in libraries make saving data nearly effortless.

Example: Saving to JSON (Nested Data)
import json

data = {
    "product": "Gaming Laptop",
    "specs": {"RAM": "16GB", "Storage": "512GB SSD"},
    "reviews": [{"user": "Alice", "score": 5}, {"user": "Bob", "score": 4}]
}

with open('products.json', 'a') as f:
    json.dump(data, f, indent=4)
Example: Saving to SQLite (Scalable Data)
import sqlite3

# Connect to database (creates it if it doesn't exist)
conn = sqlite3.connect('scraped_data.db')
cursor = conn.cursor()

# Create a table
cursor.execute('''CREATE TABLE IF NOT EXISTS products 
                  (name TEXT, price REAL)''')

# Insert data safely
cursor.execute("INSERT INTO products VALUES (?, ?)", ("Laptop", 1200.00))
conn.commit()
conn.close()

4. Best Practices for Data Integrity4. Best Practices for Data Integrity

Strategy 14: Bypass CAPTCHAs Responsibly

CAPTCHAs (Completely Automated Public Turing test to tell Computers and Humans Apart) are the ultimate “speed bumps” of the web. They are triggered when a website’s security system detects non-human behavior—like moving too fast, using a “blacklisted” datacenter IP, or having inconsistent browser headers.

1. The Best Strategy: Prevention

The “pro” way to handle CAPTCHAs is to never trigger them in the first place. If you are seeing CAPTCHAs, your previous 13 strategies need tuning.

2. Using 3rd-Party Solver Services

When a CAPTCHA is unavoidable (e.g., on a login page), pros use Solver APIs. These services use either advanced AI or human-in-the-loop workers to solve the puzzle and return a “token” that your script can submit to the website.

3. Pro-Level Implementation (Selenium/Playwright)

In a browser-based scraper, you can use specialized “Stealth” plugins that hide the properties that CAPTCHAs look for (like the navigator.webdriver flag).

Example using Playwright Stealth:
from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    
    # Apply stealth to hide bot signatures
    stealth_sync(page)
    
    page.goto("https://www.google.com")
    # The site is now less likely to present a challenge
    browser.close()

4. The “Ethical” in Responsible

Strategy 15: Stay GDPR Compliant

The General Data Protection Regulation (GDR) is the “Final Boss” of web scraping. It doesn’t matter if your code is perfect; if you scrape Personally Identifiable Information (PII)—such as names, home addresses, private emails, or even IP addresses—of individuals in the EU without a valid legal basis, you risk massive fines.

1. The Golden Rule: Public vs. Personal

Web scraping is generally legal for publicly available, non-personal data (like stock prices, product descriptions, or weather data). The moment you touch data that can identify a specific human being, the legal landscape shifts.

2. Establishing a “Legal Basis”

Under GDPR, you cannot just take data because it’s “there.” You need one of these six justifications, but for scrapers, only two usually apply:

3. Pro-Level Compliance Checklist

To keep your project (and your clients) safe, follow these implementation steps:

4. Beyond the EU: Global Privacy Laws

GDPR is the gold standard, but a pro stays aware of other regional laws:

Conclusion

Growth statistics of web data extraction industry

Python web scraping continues to dominate the world of automated data collection. With powerful libraries, flexible frameworks, and a supportive community, Python remains the top choice for developers and businesses alike.

As the digital economy grows, mastering web scraping will only become more valuable. By following best practices, respecting legal boundaries, and optimizing performance, you can unlock enormous insights from publicly available web data.

Mastering Python on web scraping is about more than just writing code; it’s about building resilient, respectful, and efficient systems. However, you can also do efficient data extraction with other programming languages such as Ruby through web scraping. As the data economy grows, these 15 strategies will help you unlock massive insights from the public web.

For Mac users, explore the best web scraping software mac tools to run your scripts efficiently.

Frequently Asked Questions

Is web scraping legal?

Yes, as long as the data is public and you aren’t violating the site’s Terms of Service or personal privacy laws (GDPR/CCPA).

What is the best library for beginners?

BeautifulSoup combined with Requests is ideal.

Can Python scrape JavaScript websites?

Yes, by using tools like Selenium or Playwright which execute the JavaScript before extracting the data.

How many pages ca I scrape per hour?

With Scrapy and a good proxy pool, you can scrape thousands of pages per hour, but always be mindful of the target server’s load.

Do I need proxies?

For large-scale scraping, yes.

Is scraping better than APIs?

APIs are preferred when available, but scraping is useful when APIs do not exist.

Tagged , , , , , , , , ,