Building a Custom Fetch MCP Server for LLM-Ready Web Content

Introduction

When working with Large Language Models (LLMs), one common challenge is feeding them web content in a clean, consistent format. While you could use simple HTTP requests or existing scraping tools, building a dedicated MCP server for this task offers several advantages:

Centralized content fetching and processing
Consistent markdown output format
Handling of JavaScript-heavy sites through Playwright
Reusable across different LLM applications

Setting Up the Project

Let's create a new MCP server using the TypeScript template:

npx @modelcontextprotocol/create-server fetch-server
cd fetch-server
npm install playwright turndown

We'll use Playwright for web scraping and Turndown for converting HTML to Markdown.

Implementing the Server

Here's our implementation in src/index.ts:

#!/usr/bin/env node
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import {
  CallToolRequestSchema,
  ErrorCode,
  ListToolsRequestSchema,
  McpError,
} from '@modelcontextprotocol/sdk/types.js';
import { chromium } from 'playwright';
import TurndownService from 'turndown';

// Initialize Turndown for HTML to Markdown conversion
const turndown = new TurndownService({
  headingStyle: 'atx',
  codeBlockStyle: 'fenced'
});

// Custom rules for better markdown output
turndown.addRule('removeScripts', {
  filter: ['script', 'style', 'iframe', 'noscript'],
  replacement: () => ''
});

turndown.addRule('preserveLinks', {
  filter: 'a',
  replacement: (content, node: any) => {
    const href = node.getAttribute('href');
    return href ? `[${content}](${href})` : content;
  }
});

class FetchServer {
  private server: Server;
  private browser: any;

  constructor() {
    this.server = new Server(
      {
        name: 'fetch-server',
        version: '0.1.0',
      },
      {
        capabilities: {
          tools: {},
        },
      }
    );

    this.setupToolHandlers();
    
    // Error handling
    this.server.onerror = (error) => console.error('[MCP Error]', error);
    process.on('SIGINT', async () => {
      await this.cleanup();
      process.exit(0);
    });
  }

  private async initBrowser() {
    if (!this.browser) {
      this.browser = await chromium.launch();
    }
  }

  private async cleanup() {
    if (this.browser) {
      await this.browser.close();
    }
    await this.server.close();
  }

  private setupToolHandlers() {
    this.server.setRequestHandler(ListToolsRequestSchema, async () => ({
      tools: [
        {
          name: 'fetch_and_process',
          description: 'Fetch a webpage and convert it to markdown for LLM processing',
          inputSchema: {
            type: 'object',
            properties: {
              url: {
                type: 'string',
                description: 'URL to fetch'
              },
              waitForSelector: {
                type: 'string',
                description: 'Optional CSS selector to wait for before processing',
              },
              excludeSelectors: {
                type: 'array',
                items: { type: 'string' },
                description: 'Optional CSS selectors to remove from content',
              },
              maxLength: {
                type: 'number',
                description: 'Maximum length of processed content (default: 100000)',
              }
            },
            required: ['url']
          }
        }
      ]
    }));

    this.server.setRequestHandler(CallToolRequestSchema, async (request) => {
      if (request.params.name !== 'fetch_and_process') {
        throw new McpError(
          ErrorCode.MethodNotFound,
          `Unknown tool: ${request.params.name}`
        );
      }

      const args = request.params.arguments as {
        url: string;
        waitForSelector?: string;
        excludeSelectors?: string[];
        maxLength?: number;
      };

      try {
        await this.initBrowser();
        const page = await this.browser.newPage();

        // Set viewport and user agent
        await page.setViewport({ width: 1280, height: 800 });
        await page.setExtraHTTPHeaders({
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
        });

        // Navigate and wait
        await page.goto(args.url, { waitUntil: 'networkidle' });
        if (args.waitForSelector) {
          await page.waitForSelector(args.waitForSelector);
        }

        // Remove unwanted elements
        if (args.excludeSelectors) {
          for (const selector of args.excludeSelectors) {
            await page.$$eval(selector, (elements) => 
              elements.forEach(el => el.remove())
            );
          }
        }

        // Extract content
        const content = await page.evaluate(() => {
          // Remove hidden elements
          document.querySelectorAll('[hidden], [style*="display: none"]')
            .forEach(el => el.remove());
          
          return document.body.innerHTML;
        });

        await page.close();

        // Convert to markdown
        let markdown = turndown.turndown(content);

        // Clean up the markdown
        markdown = markdown
          .replace(/\n{3,}/g, '\n\n')  // Remove excess newlines
          .replace(/!\[.*?\]\(.*?\)/g, '')  // Remove images
          .trim();

        // Truncate if needed
        const maxLength = args.maxLength || 100000;
        if (markdown.length > maxLength) {
          markdown = markdown.slice(0, maxLength) + '\n\n[Content truncated...]';
        }

        return {
          content: [
            {
              type: 'text',
              text: markdown
            }
          ]
        };

      } catch (error) {
        if (error instanceof Error) {
          return {
            content: [
              {
                type: 'text',
                text: `Error processing URL: ${error.message}`
              }
            ],
            isError: true
          };
        }
        throw error;
      }
    });
  }

  async run() {
    const transport = new StdioServerTransport();
    await this.server.connect(transport);
    console.error('Fetch MCP server running on stdio');
  }
}

const server = new FetchServer();
server.run().catch(console.error);

Key Features

Our fetch MCP server includes several important features:

Playwright Integration: Handles JavaScript-rendered content and complex web applications
Smart Content Processing: Removes unwanted elements, handles hidden content
Configurable Waiting: Can wait for specific elements to load
Clean Markdown Output: Consistent formatting for LLM processing
Resource Management: Proper browser cleanup and error handling

Using the Server

After building and installing the server, you can use it like this:

// Example usage in an LLM application
const response = await use_mcp_tool({
  server_name: "fetch-server",
  tool_name: "fetch_and_process",
  arguments: {
    url: "https://example.com/article",
    waitForSelector: "article",
    excludeSelectors: [
      ".comments",
      ".ads",
      "nav",
      "footer"
    ],
    maxLength: 50000
  }
});

Advanced Configuration

You can enhance the server further with these additions:

Rate Limiting: Add delays between requests to respect website limits
Caching: Store processed content to avoid repeated fetches
Custom Processing: Add site-specific content extraction rules
Error Recovery: Implement retries and fallback strategies

Ethical Considerations

When building web scraping tools, always:

Respect robots.txt and site terms of service
Implement rate limiting to avoid overwhelming servers
Consider caching to reduce unnecessary requests
Be transparent about your bot's identity through user agents

Conclusion

A custom fetch MCP server provides a robust foundation for feeding web content to LLMs. By centralizing the fetching and processing logic, you ensure consistent, clean input for your language models while maintaining scalability and reusability across different projects.