The scraper tool is an asynchronous utility that leverages the Oxylabs Web Scraper API to fetch and process content from a given URL. This tool is designed to handle various scraping scenarios efficiently, with flexible options for parsing and rendering web pages.

Key Features

1. URL Scraping

   • Fetches the content of a specified URL using Oxylabs' API.

2. HTML Parsing

   • If parsing is enabled, the content is returned in a processed format, suitable for handling by downstream systems like language models (LLMs).
   • If parsing is disabled, the tool strips unnecessary HTML tags and converts the cleaned content into a Markdown file for easier readability and compatibility.

3. JavaScript Rendering

   • Offers an option to render pages using a headless browser, which ensures dynamic content (e.g., JavaScript-rendered pages) is included in the output.
   • Supports the HTML rendering option or skipping rendering entirely.

Examples on how to query Claude or other LLM

• Could you scrape https://oxylabs.io page?
• Scrape https://www.amazon.de/-/en/Smartphone-Contract-Function-Manufacturer-Exclusive/dp/B0CNKD651V with parse enabled.
• Scrape https://www.amazon.de/-/en/gp/bestsellers/beauty/ref=zg_bs_nav_beauty_0 with parse and render enabled.