A Model Context Protocol (MCP) server for web content scanning and analysis. This server provides tools for fetching, analyzing, and extracting information from web pages.

Features

• Page Fetching: Convert web pages to Markdown for easy analysis
• Link Extraction: Extract and analyze links from web pages
• Site Crawling: Recursively crawl websites to discover content
• Link Checking: Identify broken links on web pages
• Pattern Matching: Find URLs matching specific patterns
• Sitemap Generation: Generate XML sitemaps for websites

Available Tools

1. fetch_page

   • Fetches a web page and converts it to Markdown
   • Parameters:
     • url (required): URL of the page to fetch
     • selector (optional): CSS selector to target specific content

2. extract_links

   • Extracts all links from a web page with their text
   • Parameters:
     • url (required): URL of the page to analyze
     • baseUrl (optional): Base URL to filter links

3. crawl_site

   • Recursively crawls a website up to a specified depth
   • Parameters:
     • url (required): Starting URL to crawl
     • maxDepth (optional, default: 2): Maximum crawl depth

4. check_links

   • Checks for broken links on a page
   • Parameters:
     • url (required): URL to check links for

5. find_patterns

   • Finds URLs matching a specific pattern
   • Parameters:
     • url (required): URL to search in
     • pattern (required): Regex pattern to match URLs against

6. generate_sitemap

   • Generates a simple XML sitemap
   • Parameters:
     • url (required): Root URL for sitemap
     • maxUrls (optional, default: 100): Maximum number of URLs to include

Example Usage with Claude Desktop

1. Configure the server in your Claude Desktop settings:

1. Use the tools in your conversations:

Error Handling

The server implements comprehensive error handling:

• Invalid parameters
• Network errors
• Content parsing errors
• URL validation

All errors are properly formatted according to the MCP specification.