Enhancing LEGO Scraping with MCP Servers for Coordinated Orchestration

Introduction

In a previous post, we explored how to scrape LEGO product data and gallery images using Playwright-powered Scrapy spiders. While that setup works well for a handful of pages, what happens when you need to scale up, manage multiple scraping sessions, or dynamically update selectors without redeploying your code?

Enter the MCP (Modular Control Platform) server: a dedicated backend service that centralizes orchestration tasks, stores configuration data, and provides a stable API for your scraping infrastructure. With an MCP server, you can easily adjust selectors, rotate proxies, and even control how Playwright sessions are handled—all without modifying the spiders directly.

What is an MCP Server?

An MCP server acts as a "control plane" for your scraping setup. Think of it as a microservice that:

• Stores CSS and XPath selectors used by the spiders, allowing dynamic updates if LEGO changes their site structure.
• Manages Playwright settings (like timeouts, concurrency, or browser type) to ensure efficient scraping sessions.
• Logs requests, responses, and errors, enabling better debugging and monitoring.
• Centralizes configuration so that multiple spiders or even separate scraping projects can share consistent guidelines.

Integrating MCP with Your LEGO Scraper

In the previous LEGO scraping project, we hardcoded selectors and configurations directly into the spider code. With MCP integration, your spiders now fetch these details at runtime via an API call.

This lets you react quickly to site changes: if LEGO alters their HTML structure, simply update the MCP server's configuration and all connected spiders pick up the new selectors on their next run—no redeployment needed.

Example MCP Server Setup

Below is a minimal FastAPI-based MCP server that returns configuration data for our LEGO scrapers. It includes:

• A `/selectors` endpoint to fetch updated CSS selectors.
• A `/playwright-config` endpoint to fetch browser and timeout settings.
• A simple in-memory store for demonstration purposes (you can connect a database for production use).

Code Snippet:

Copyfrom fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Selectors(BaseModel):
    product_id: str
    product_name: str
    images: str
    recommended_age: str
    number_of_parts: str
    number_of_minifigures: str
    description_block: str

class PlaywrightConfig(BaseModel):
    headless: bool
    timeout_ms: int
    browser: str

# Mock data for demonstration
current_selectors = Selectors(
    product_id="div[data-test='item-value'] span > span::text",
    product_name="h1[data-test='product-overview-name'] span::text",
    images="#product-media-1 source::attr(srcset)",
    recommended_age="div[data-test='ages-value'] span > span::text",
    number_of_parts="div[data-test='pieces-value'] span > span::text",
    number_of_minifigures="div[data-test='minifigures-value'] span > span::text",
    description_block="div[data-test='pdp-specifications-accordion-content-child'] p::text, div[data-test='pdp-specifications-accordion-content-child'] ul li::text"
)

current_playwright_config = PlaywrightConfig(
    headless=True,
    timeout_ms=10000,
    browser="chromium"
)

@app.get("/selectors", response_model=Selectors)
def get_selectors():
    return current_selectors

@app.get("/playwright-config", response_model=PlaywrightConfig)
def get_config():
    return current_playwright_config

Using MCP Data in Your Spiders

Adjusting your existing scraping code is simple: before parsing the page, fetch selectors and config from the MCP server. This can be done once at startup, cached in a local variable, or refreshed periodically if you anticipate frequent changes.

Modified Spider Snippet:

Copyimport scrapy
import json
import requests
from scrapy_playwright.page import PageMethod

# Fetch configuration from MCP server
MCP_URL = "http://localhost:8000"
selectors = requests.get(f"{MCP_URL}/selectors").json()
playwright_config = requests.get(f"{MCP_URL}/playwright-config").json()

class LegoSpiderMCP(scrapy.Spider):
    name = "lego_spider_mcp"
    allowed_domains = ["lego.com"]
    start_urls = ["https://www.lego.com/en-us/product/optimus-prime-10302"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                callback=self.parse,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_load_state", 'domcontentloaded'),
                    ],
                },
            )

    def parse(self, response):
        product_id = response.css(selectors["product_id"]).get()
        product_name = response.css(selectors["product_name"]).get()
        images = response.css(selectors["images"]).getall()

        recommended_age = response.css(selectors["recommended_age"]).get()
        number_of_parts = response.css(selectors["number_of_parts"]).get()
        number_of_minifigures = response.css(selectors["number_of_minifigures"]).get()

        description_elements = response.css(selectors["description_block"]).getall()
        description = " ".join(description_elements).strip()

        product = {
            "product-id": product_id,
            "product url": response.url,
            "product name": product_name,
            "recommended age": recommended_age,
            "number of parts": number_of_parts,
            "number of mini-figures": number_of_minifigures or "0",
            "description": description,
            "images": images[:5]  # limit to 5 images
        }

        print(json.dumps(product, indent=4, ensure_ascii=False))
        yield product

Benefits of MCP Integration

• Dynamic Adjustments: No more re-deployment when LEGO changes their page structure. Update selectors in MCP and keep scraping.
• Consistent Configuration: Multiple spiders use the same config, ensuring a uniform approach across projects.
• Scalability: As your scraping needs grow, MCP servers can distribute configurations to multiple scraping nodes.
• Enhanced Monitoring: MCP can log requests, maintain metadata, and provide debugging endpoints for quick diagnostics.

Ethical & Legal Considerations

As always, ensure you have permission to scrape. The MCP server makes your operations more efficient, but it doesn't relieve you of the responsibility to follow the target site's terms, respect rate limits, and comply with relevant laws and guidelines.

Expanding the MCP Concept

1. Versioned Configurations: Store multiple selector sets and switch between them easily.
2. Authentication & Access Control: Secure the MCP API so only authorized crawlers can access it.
3. Analytics & Dashboards: Track how often selectors fail, monitor scraping performance, and alert on anomalies.
4. Custom Playwright Hooks: Serve advanced Playwright scripts (like cookies or authentication steps) dynamically from MCP.

Conclusion

Integrating an MCP server into your LEGO scraping workflow transforms a static, brittle process into a dynamic, robust system. By externalizing configuration, selectors, and Playwright settings, you gain the agility to adapt quickly, scale seamlessly, and maintain a healthy scraping operation that can withstand the inevitable changes of real-world websites.