Scrape E-commerce Sites at the Lowest Layer That Works
When scraping product pages, a 200 response doesn't guarantee useful data, as key information often loads via JavaScript. The crucial question is not "which scraper?" but "where is the data located?" Many e-commerce pages use client-side rendering, fetching data via backend API calls. A good starting point is to inspect network requests in DevTools for JSON responses containing product details.A rough workflow involves manually loading the page, filtering network requests for Fetch/XHR, searching response bodies for known product data, copying the request as cURL, and then minimizing headers to find the essential ones. Some sites allow direct cURL calls, while others require session cookies, geo headers, or CSRF tokens.Shopify stores are often easier, as many expose their catalog through /products.json, providing product titles, variants, prices, and image URLs. This method is faster and more robust than parsing HTML and avoids CSS selector failures from theme changes. However, some Shopify stores block this endpoint or limit the exposed data.When direct JSON endpoints aren't available, or JavaScript is needed for token generation, rendering the page with a headless browser like Playwright is necessary. Playwright can capture API responses or extract data from the DOM after JavaScript execution. This approach is slower and requires managing issues like timeouts, CAPTCHAs, and region inference.Full browser automation is reserved for interactive states like login-protected prices, wishlists, or checkout processes that demand clicks and persistent sessions. However, this is the most resource-intensive and prone to issues like slower jobs, increased CAPTCHA risk, and browser fingerprints. The decision tree prioritizes public JSON endpoints, then internal API calls, followed by headless browser rendering, and finally full browser automation. Avoid starting with full browser automation unless absolutely necessary, as it becomes expensive and flaky at scale.
cURLcalls, while others require session cookies, geo headers, or CSRF tokens.Shopify stores are often easier, as many expose their catalog through/products.json, providing product titles, variants, prices, and image URLs. This method is faster and more robust than parsing HTML and avoids CSS selector failures from theme changes. However, some Shopify stores block this endpoint or limit the exposed data.When direct JSON endpoints aren't available, or JavaScript is needed for token generation, rendering the page with a headless browser like Playwright is necessary. Playwright can capture API responses or extract data from the DOM after JavaScript execution. This approach is slower and requires managing issues like timeouts, CAPTCHAs, and region inference.Full browser automation is reserved for interactive states like login-protected prices, wishlists, or checkout processes that demand clicks and persistent sessions. However, this is the most resource-intensive and prone to issues like slower jobs, increased CAPTCHA risk, and browser fingerprints. The decision tree prioritizes public JSON endpoints, then internal API calls, followed by headless browser rendering, and finally full browser automation. Avoid starting with full browser automation unless absolutely necessary, as it becomes expensive and flaky at scale.