QuantumScraper is a blazing fast scraper specifically built to crawl the entire web at an extremely fast pace.
This is not your average, general-purpose scraper. QuantumScraper is focused on speed and efficiency, delivering web-scale crawls at lightning-fast speeds.
Features 🌟
Ultra-fast HTTP networking with fasthttp
Built-in NSQD server for URL queuing
Advanced URL management and storage using BadgerDB
Setup
Requirements
Go
AWS CLI (Optional)
URL Starter Data Preparation
To start off the scraper, we're initializing the scraper's state with the URLs of the CommonCrawl.
While we don't need this, we're doing to speedup the scraper. We could start from the root domains priovided by Domains Index , Zonefiles, Domains Monitor or similar.
Alternative: Setup your AWS CLI, login to your AWS account. Then download latest columnar CommonCrawl Index. (This might incur bandwidth costs)