A product-search page is designed for browsing, not for producing a clean table. Titles wrap, sponsored listings interrupt the grid, prices have several formats, and part of the page may appear after the first HTML response. A useful extraction script must handle these details without quietly fabricating complete-looking records.
AlgoFly's Selenium and Python video uses Amazon product search as the setting for a step-by-step data-collection tutorial. The published description covers browser setup, navigation, dynamic content, extraction, and saving records, but does not link a maintained code repository. This companion focuses on a reusable collection design. The example selectors belong to a hypothetical page you control; they are not advertised as current Amazon selectors.
Define the record before launching a browser
Start with the research question. If you need displayed prices for a set of queries, define which marketplace, location, currency, time, and result depth belong to the observation. A query string alone is not enough to reproduce what a browser saw.
Keep a listing identifier when one is available, the source URL, title, displayed price text, normalized price if parsing succeeds, and a timestamp. Separate raw text from normalized values so a parser can be fixed later without recollecting every page.
Use collection methods you are authorized to use. Where a suitable licensed feed or official interface is available for the task, evaluate it before maintaining a browser scraper. This workflow does not include CAPTCHA solving or evading access controls.
Wait for the condition that makes extraction possible
A page-load event does not guarantee that the result grid has finished rendering. Selenium's explicit waits let a script wait for a relevant condition, such as the first product card appearing. That is more precise than a long fixed sleep, which can be unnecessarily slow on one run and insufficient on another.
Keep browser creation and cleanup together. Use a predictable viewport and record any settings that affect the returned page. The following is an illustrative fixture-page example; replace its URL and selectors only for a source you are permitted to collect.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
driver.get("http://localhost:8000/research-products.html")
cards = WebDriverWait(driver, 15).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, "[data-product]"))
)
rows = []
for card in cards:
title = card.find_element(By.CSS_SELECTOR, "[data-title]").text.strip()
prices = card.find_elements(By.CSS_SELECTOR, "[data-price]")
rows.append({
"listing_id": card.get_attribute("data-product"),
"title": title,
"price_text": prices[0].text.strip() if prices else None,
})
finally:
driver.quit()
Missing data should survive as missing data
Some cards do not contain a visible price. Others represent a bundle, a promotion, or an unavailable item. Do not catch every extraction exception and insert zero. Zero looks like a remarkable discount and can contaminate every later analysis.
Distinguish an optional field from a broken layout. An absent optional rating can yield null; a page with no recognizable cards should be recorded as a collection failure. Store an error reason and enough page context to investigate, while avoiding unnecessary personal information in captured artifacts.
Normalize whitespace and URLs carefully. Removing every non-digit character from a price string loses decimal separators and currency meaning. A parser should know the locale and reject formats it cannot interpret rather than guessing.
Keep collection bounded and deduplicate by identity
Set a maximum number of pages and a clear stop condition. Reaching a page without a next-page control is different from a timeout. If a source uses infinite scrolling, define the maximum scroll count and detect whether new listing identifiers were actually added.
Deduplicate using stable listing identity, not only title. Two products can share a name, and one product can appear as both a sponsored and an organic result. If placement matters to the research, retain each occurrence with its rank and placement type while linking it to the same listing.
Make retries bounded. Repeatedly hammering a failed page does not make the dataset more reliable. Record failures for a later controlled retry and stop when the source denies access.
Keep the collection run auditable
A newline-delimited JSON file is a convenient starting point because each record can preserve both normalized and raw fields. Add a run identifier and the observed timestamp to every record. Write results as they are collected so a browser crash does not erase a long run.
For price-history work, append observations rather than updating one “current price” field. Keep query, page, location context, and listing identifier attached. Our companion article on retail price history explains why those dimensions matter once the data reaches a dashboard.
If you later add a database, preserve the same provenance. A fast query cannot compensate for a record whose marketplace or collection date has been lost.
Test parsers on saved, permitted fixtures
Split browser navigation from data extraction and normalization. Test the parser against a small fixture containing a normal item, an unavailable item, a malformed price, and duplicate listings. This reveals changes without making live requests for every code edit.
Monitor counts of missing fields and unparsed prices between runs. A sudden change can indicate a layout update rather than a real market event. Treat the browser script as a collector with observable failures, not a guarantee that every visible card becomes a valid product record.
The video's useful starting point is browser automation. The lasting value comes from the surrounding choices: precise scope, explicit waits, honest missing values, bounded work, and enough context to explain every row after the browser window closes.