Reverse Engineering LEGO Product Data with Custom Scrapers

Introduction

Ever wanted to extract rich product data and gallery images from a website for research, analysis, or automation purposes? This post walks you through a simple, yet powerful, Playwright-powered Scrapy configuration that navigates LEGO product pages and collects key information. While the project focuses on LEGO sets, the underlying principles can be applied to a wide range of e-commerce or product-driven platforms.

Important Disclaimer: The techniques discussed here tread into "grey hat" territory. We're showcasing them for educational and research purposes only. Always check the legal and ethical guidelines before scraping or reverse-engineering any site. Respect website policies, consider rate limits, and avoid using these methods for malicious or unethical activities.

The Spiders: Two Approaches to Extraction

We use two Scrapy spiders with Playwright integration for browser automation. Playwright helps load dynamic content that a standard HTML request might miss.

The Product Spider: lego_spider

The first spider focuses on capturing core product details:

• Product ID & Name: Extracted from the page's DOM elements.
• Images: Grabs up to five product images.
• Recommended Age, Parts, and Minifigure Count: Useful product attributes for collectors.
• Description & Specifications: Detailed textual info about the set.

Key Features:

• Playwright Integration: Ensures JavaScript-powered content is fully rendered.
• CSS Selectors: Targeting specific data attributes (data-test) makes the extraction reliable and less prone to design changes.
• JSON Output: Ideal for integrating the collected data into further pipelines or storage systems.

Code Snippet:

Copyimport scrapy
import json
from lego_scraper.consts import PRODUCT_LINKS
from scrapy_playwright.page import PageMethod


class LegoSpider(scrapy.Spider):
    name = "lego_spider"
    allowed_domains = ["lego.com"]
    start_urls = PRODUCT_LINKS

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                callback=self.parse,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_load_state", 'domcontentloaded'),
                    ],
                },
            )

    def close(self, reason):
        self.crawler.engine.downloader.close()

    def parse(self, response):
        product_id = response.css("div[data-test='item-value'] span > span::text").get()
        product_name = response.css("h1[data-test='product-overview-name'] span::text").get()
        images = response.css("#product-media-1 source::attr(srcset)").getall()

        # Limit to 5 images
        max_images = 5
        image_fields = {}
        for idx in range(max_images):
            field_name = f'image_{idx + 1}'
            image_fields[field_name] = images[idx] if idx < len(images) else ''

        recommended_age = response.css("div[data-test='ages-value'] span > span::text").get()
        number_of_parts = response.css("div[data-test='pieces-value'] span > span::text").get()
        number_of_minifigures = response.css("div[data-test='minifigures-value'] span > span::text").get()

        description_elements = response.css(
            "div[data-test='pdp-specifications-accordion-content-child'] p::text, "
            "div[data-test='pdp-specifications-accordion-content-child'] ul li::text"
        ).getall()
        description = " ".join(description_elements).strip()

        product = {
            "product-id": product_id,
            "product url": response.url,
            "product name": product_name,
            "recommended age": recommended_age,
            "number of parts": number_of_parts,
            "number of mini-figures": number_of_minifigures if number_of_minifigures else "0",
            "description": description,
            **image_fields
        }
        print(f"Got product - {product_name}\n{json.dumps(product, indent=4, ensure_ascii=False)}")
        yield product

The Gallery Spider: gallery_spider

The second spider focuses on extracting all available gallery images by navigating to a galleryOpen query parameter.

Key Features:

• Image-Focused Extraction: Scrapes up to 20 gallery images for each product.
• URL Manipulation: Appends ?galleryOpen=true&galleryView=0 to product URLs to reveal additional gallery assets.
• Filter Non-LEGO and Irrelevant Images: Avoids collecting noise or unrelated content.

Code Snippet:

Copyimport scrapy
from lego_scraper.consts import PRODUCT_LINKS
from scrapy_playwright.page import PageMethod

class GallerySpider(scrapy.Spider):
    name = "gallery_spider"
    gallery_query = '?galleryOpen=true&galleryView=0'
    start_urls = PRODUCT_LINKS

    def start_requests(self):
        for url in self.start_urls:
            gallery_url = f"{url + self.gallery_query}"
            yield scrapy.Request(
                gallery_url,
                callback=self.parse,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_load_state", 'domcontentloaded'),
                    ],
                },
            )

    def parse(self, response):
        product_url = response.url.rsplit(self.gallery_query, 1)[0]
        product_path = product_url.split('/')[-1]
        product_id = product_path.split('-')[-1]

        images = response.css('img::attr(src)').getall()
        _filtered_images = [img for img in images if img.startswith('https://www.lego.com/cdn/')]
        _filtered_urls = [img for img in _filtered_images if not img.startswith('/_next/static/')]
        filtered_urls = [img for img in _filtered_urls if 'youtube' not in img]

        print(f"Extracted {len(filtered_urls)} images from {product_url} || {product_id}")
        print(f"Gallery images: {filtered_urls}")

        max_images = 20
        image_fields = {}
        for idx in range(max_images):
            field_name = f'image_{idx + 1}'
            image_fields[field_name] = images[idx] if idx < len(images) else ''

        yield {
            'product_id': product_id,
            'product_url': product_url,
            **image_fields
        }

How it Works

1. Initialization: Both spiders load product links from PRODUCT_LINKS, a constants file that lists the LEGO product pages you want to scrape.
2. Playwright Integration: Each request adds a playwright meta tag, instructing Scrapy to use Playwright to render the page fully before parsing.
3. Content Extraction: Using CSS selectors, both spiders gather the needed data and yield structured dictionaries. These can be exported as JSON or fed into databases.
4. Filtering & Transformation: The code filters out irrelevant images and ensures consistent field naming. A uniform data structure makes subsequent data processing much more manageable.

Use Cases

• Market Research: Compare product details, pricing, and images across multiple LEGO sets or even different e-commerce platforms.
• Inventory Management: Sync your store's product info with official site data for accuracy.
• Data Visualization: Build analytics dashboards or feed data into machine learning models for insights into product lines, rarity, or pricing trends.

Limitations & Ethical Considerations

• Ethical Use: Before scraping, check the target website's terms of service. Some websites explicitly prohibit crawling.
• Load & Performance: Playwright adds overhead to each request. Avoid hammering a website with too many parallel requests.
• Legality: Scraping without permission can lead to legal consequences, especially in jurisdictions with strict anti-scraping laws.

Improvements & Refactoring

1. Modularized Selectors: Extract all CSS selectors into a single selectors.py file to simplify maintenance and updates.
2. Async/Await & Parallelization: Consider using Playwright's async capabilities for more efficient scraping, but ensure respectful load on the target site.
3. Robust Error Handling: Implement try/except blocks and logging to catch broken selectors, missing attributes, or unexpected page structures.
4. Configurable Limits: Make max_images configurable via settings or command-line arguments, rather than hardcoding them.
5. Data Validation: Implement schema validation using tools like pydantic to ensure data consistency and catch anomalies early.

Conclusion

The presented scraping setup is a foundation. From here, you can tailor, optimize, and scale the approach as needed. For research, inventory management, or complex data analysis, having robust scraping capabilities provides a powerful toolkit for understanding products and markets—just remember to use these methods ethically, responsibly, and within legal bounds.