mirroring-web-crawler

mirroring-web-crawler

A recursive, mirroring web crawler as a command-line tool that accepts a starting URL and a destination directory. The crawler will then download the page at the URL, save it in the destination directory, and then recursively proceed to any valid links in this page.

Dependencies

• Go Version- go1.20
• Go package- net/http
• Go package- golang.org/x/sync v0.1.0

Features

• Fetch the resource at a target url
• Parse the retrieve target resource to extract all children <a> links
• Writes downloaded resource into provided directory
• Skip fetching and processing files where not necessary such as when they were already downloaded
• Configurable concurrent processing
• Support interruption and resume capabilities

Enhancements

• Use a goroutine pool backed by a queue which provides a better concurrency model and usage of resources.
• Add support for resuming partially downloaded files which can happen when the crawler is interrupted

Code

Compile:

go build

Usage:

Usage of ./mirroring-web-crawler:
  -concurrency int
        number of concurrent crawlers (default 10)
  -dir string
        target directory to save crawledState files
  -interval int
        the number of milliseconds to wait between crawls (default 1000)
  -overwrite
        overwrite download of files
  -url string
        target url to crawl

Example:

./mirroring-web-crawler -url https://www.scrapingbee.com -dir ./scrapping-bee -interval 10 -concurrency 100

Output:

 % ./mirroring-web-crawler -url https://www.scrapingbee.com -dir ./scrapping-bee -interval 10 -concurrency 100
starting crawler on: 'https://www.scrapingbee.com'
fetching 'https://www.scrapingbee.com'
writing to 'scrapping-bee/index.html'
extracted valid child url: 'https://www.scrapingbee.com/index.xml'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.png'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.svg'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/main.min.a09f1f7d5c32eba3a323bc3c39fca98dc62a83bad52faf6e0c62e7c5285cab6a.css'
extracted valid child url: 'https://www.scrapingbee.com/main.min.a09f1f7d5c32eba3a323bc3c39fca98dc62a83bad52faf6e0c62e7c5285cab6a.css'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/#pricing'
extracted valid child url: 'https://www.scrapingbee.com/#faq'
extracted valid child url: 'https://www.scrapingbee.com/blog/'
extracted valid child url: 'https://www.scrapingbee.com/features/screenshot/'
extracted valid child url: 'https://www.scrapingbee.com/features/google/'
extracted valid child url: 'https://www.scrapingbee.com/features/data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/features/javascript-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/features/make/'
extracted valid child url: 'https://www.scrapingbee.com/tutorials'
extracted valid child url: 'https://www.scrapingbee.com/documentation/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/'
extracted valid child url: 'https://www.scrapingbee.com/features/data-extraction'
extracted valid child url: 'https://www.scrapingbee.com/features/javascript-scenario'
extracted valid child url: 'https://www.scrapingbee.com/features/screenshot'
extracted valid child url: 'https://www.scrapingbee.com/features/google'
extracted valid child url: 'https://www.scrapingbee.com/features/make'
extracted valid child url: 'https://www.scrapingbee.com/documentation/#javascript-execution'
extracted valid child url: 'https://www.scrapingbee.com/documentation/data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/js-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/#Pricing'
extracted valid child url: 'https://www.scrapingbee.com/journey-to-one-million-arr/'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/#about-us'
extracted valid child url: 'https://www.scrapingbee.com/journey-to-one-million-arr/'
extracted valid child url: 'https://www.scrapingbee.com/blog/'
extracted valid child url: 'https://www.scrapingbee.com/rebranding/'
extracted valid child url: 'https://www.scrapingbee.com/affiliates/'
extracted valid child url: 'https://www.scrapingbee.com/curl-converter/'
extracted valid child url: 'https://www.scrapingbee.com/terms-and-conditions/'
extracted valid child url: 'https://www.scrapingbee.com/privacy-policy/'
extracted valid child url: 'https://www.scrapingbee.com/gdpr/'
extracted valid child url: 'https://www.scrapingbee.com/data-processing-agreement/'
extracted valid child url: 'https://www.scrapingbee.com/#features'
extracted valid child url: 'https://www.scrapingbee.com/#pricing'
extracted valid child url: 'https://www.scrapingbee.com/crawlera-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/luminati-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/smartproxy-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/netnut-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scraperapi-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scrapingbee-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-competitor-monitoring/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scrape-content-google-sheet/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-stock-price-slack/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-job-data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/webscraping-questions/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-without-getting-blocked/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-tools/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-free-proxy-list-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-mobile-4g-proxy-provider-webscraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scraping-vs-crawling/'
extracted valid child url: 'https://www.scrapingbee.com/blog/rotating-proxies/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-101-with-python/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-php/'
extracted valid child url: 'https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-ruby/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-javascript/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-r/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-csharp/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-c++/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-elixir/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-perl/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-rust/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-go/'
extracted valid child url: 'https://www.scrapingbee.com/images/logo.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/hero_illustration.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_headless.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/mike.png'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_rendering.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/russel.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_proxies.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/dominic.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon01.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/data.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/js.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/picture-1.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/search.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/nocode.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/anton.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/maxime_2.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/andy.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/jean.jpg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/team/kevin.png'
extracted valid child url: 'https://www.scrapingbee.com/images/team/pierre.png'
extracted valid child url: 'https://www.scrapingbee.com/images/team/etienne.png'
extracted valid child url: 'https://www.scrapingbee.com/images/logo-white.svg'
extracted valid child url: 'https://www.scrapingbee.com/scripts/app-landing.js'
crawler has finished crawling: 'https://www.scrapingbee.com'
starting crawler on: 'https://www.scrapingbee.com/images/team/kevin.png'
fetching 'https://www.scrapingbee.com/images/team/kevin.png'
writing to 'scrapping-bee/images/team/kevin.png'
crawler has finished crawling: 'https://www.scrapingbee.com/images/team/kevin.png'
starting crawler on: 'https://www.scrapingbee.com/images/icons/nocode.svg'
fetching 'https://www.scrapingbee.com/images/icons/nocode.svg'
writing to 'scrapping-bee/images/icons/nocode.svg'
crawler has finished crawling: 'https://www.scrapingbee.com/images/icons/nocode.svg'
starting crawler on: 'https://www.scrapingbee.com/documentation/js-scenario/'
fetching 'https://www.scrapingbee.com/documentation/js-scenario/'
writing to 'scrapping-bee/documentation/js-scenario.html'
extracted valid child url: 'https://www.scrapingbee.com/index.xml'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.png'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.svg'
extracted valid child url: 'https://www.scrapingbee.com/documentation/js-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/main.min.a09f1f7d5c32eba3a323bc3c39fca98dc62a83bad52faf6e0c62e7c5285cab6a.css'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/google/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/proxy-mode/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/js-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/tutorials/'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/documentation'
extracted valid child url: 'https://www.scrapingbee.com/documentation#json_response'
extracted valid child url: 'https://www.scrapingbee.com/documentation/#json_response'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/#about-us'
extracted valid child url: 'https://www.scrapingbee.com/journey-to-one-million-arr/'
extracted valid child url: 'https://www.scrapingbee.com/blog/'
extracted valid child url: 'https://www.scrapingbee.com/rebranding/'
extracted valid child url: 'https://www.scrapingbee.com/affiliates/'
extracted valid child url: 'https://www.scrapingbee.com/curl-converter/'
extracted valid child url: 'https://www.scrapingbee.com/terms-and-conditions/'
extracted valid child url: 'https://www.scrapingbee.com/privacy-policy/'
extracted valid child url: 'https://www.scrapingbee.com/gdpr/'
extracted valid child url: 'https://www.scrapingbee.com/data-processing-agreement/'
extracted valid child url: 'https://www.scrapingbee.com/#features'
extracted valid child url: 'https://www.scrapingbee.com/#pricing'
extracted valid child url: 'https://www.scrapingbee.com/crawlera-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/luminati-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/smartproxy-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/netnut-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scraperapi-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scrapingbee-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-competitor-monitoring/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scrape-content-google-sheet/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-stock-price-slack/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-job-data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/webscraping-questions/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-without-getting-blocked/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-tools/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-free-proxy-list-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-mobile-4g-proxy-provider-webscraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scraping-vs-crawling/'
extracted valid child url: 'https://www.scrapingbee.com/blog/rotating-proxies/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-101-with-python/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-php/'
extracted valid child url: 'https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-ruby/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-javascript/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-r/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-csharp/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-c++/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-elixir/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-perl/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-rust/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-go/'
extracted valid child url: 'https://www.scrapingbee.com/images/logo.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/logo-white.svg'
extracted valid child url: 'https://www.scrapingbee.com/scripts/hljs.min.js'
extracted valid child url: 'https://www.scrapingbee.com/scripts/app-doc.min.js'
crawler has finished crawling: 'https://www.scrapingbee.com/documentation/js-scenario/'
starting crawler on: 'https://www.scrapingbee.com/#features'
fetching 'https://www.scrapingbee.com/#features'
writing to 'scrapping-bee.html'
extracted valid child url: 'https://www.scrapingbee.com/index.xml'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.png'
extracted valid child url: 'https://www.scrapingbee.com/images/favico.svg'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/main.min.a09f1f7d5c32eba3a323bc3c39fca98dc62a83bad52faf6e0c62e7c5285cab6a.css'
extracted valid child url: 'https://www.scrapingbee.com/main.min.a09f1f7d5c32eba3a323bc3c39fca98dc62a83bad52faf6e0c62e7c5285cab6a.css'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/#pricing'
extracted valid child url: 'https://www.scrapingbee.com/#faq'
extracted valid child url: 'https://www.scrapingbee.com/blog/'
extracted valid child url: 'https://www.scrapingbee.com/features/screenshot/'
extracted valid child url: 'https://www.scrapingbee.com/features/google/'
extracted valid child url: 'https://www.scrapingbee.com/features/data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/features/javascript-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/features/make/'
extracted valid child url: 'https://www.scrapingbee.com/tutorials'
extracted valid child url: 'https://www.scrapingbee.com/documentation/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/'
extracted valid child url: 'https://www.scrapingbee.com/features/data-extraction'
extracted valid child url: 'https://www.scrapingbee.com/features/javascript-scenario'
extracted valid child url: 'https://www.scrapingbee.com/features/screenshot'
extracted valid child url: 'https://www.scrapingbee.com/features/google'
extracted valid child url: 'https://www.scrapingbee.com/features/make'
extracted valid child url: 'https://www.scrapingbee.com/documentation/#javascript-execution'
extracted valid child url: 'https://www.scrapingbee.com/documentation/data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/js-scenario/'
extracted valid child url: 'https://www.scrapingbee.com/documentation/#Pricing'
extracted valid child url: 'https://www.scrapingbee.com/journey-to-one-million-arr/'
extracted valid child url: 'https://www.scrapingbee.com/'
extracted valid child url: 'https://www.scrapingbee.com/#about-us'
extracted valid child url: 'https://www.scrapingbee.com/journey-to-one-million-arr/'
extracted valid child url: 'https://www.scrapingbee.com/blog/'
extracted valid child url: 'https://www.scrapingbee.com/rebranding/'
extracted valid child url: 'https://www.scrapingbee.com/affiliates/'
extracted valid child url: 'https://www.scrapingbee.com/curl-converter/'
extracted valid child url: 'https://www.scrapingbee.com/terms-and-conditions/'
extracted valid child url: 'https://www.scrapingbee.com/privacy-policy/'
extracted valid child url: 'https://www.scrapingbee.com/gdpr/'
extracted valid child url: 'https://www.scrapingbee.com/data-processing-agreement/'
extracted valid child url: 'https://www.scrapingbee.com/#features'
extracted valid child url: 'https://www.scrapingbee.com/#pricing'
extracted valid child url: 'https://www.scrapingbee.com/crawlera-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/luminati-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/smartproxy-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/netnut-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scraperapi-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/scrapingbee-alternative/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-competitor-monitoring/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scrape-content-google-sheet/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-stock-price-slack/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/nocode-amazon/'
extracted valid child url: 'https://www.scrapingbee.com/blog/no-code-job-data-extraction/'
extracted valid child url: 'https://www.scrapingbee.com/webscraping-questions/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-without-getting-blocked/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-tools/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-free-proxy-list-web-scraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/best-mobile-4g-proxy-provider-webscraping/'
extracted valid child url: 'https://www.scrapingbee.com/blog/scraping-vs-crawling/'
extracted valid child url: 'https://www.scrapingbee.com/blog/rotating-proxies/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-101-with-python/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-php/'
extracted valid child url: 'https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-ruby/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-javascript/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-r/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-csharp/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-c++/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-elixir/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-perl/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-rust/'
extracted valid child url: 'https://www.scrapingbee.com/blog/web-scraping-go/'
extracted valid child url: 'https://www.scrapingbee.com/images/logo.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/hero_illustration.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_headless.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/mike.png'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_rendering.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/russel.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/landing/feature_proxies.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/dominic.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon01.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/data.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/js.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/picture-1.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/search.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/nocode.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/anton.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/maxime_2.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/andy.jpeg'
extracted valid child url: 'https://www.scrapingbee.com/images/testimonials/jean.jpg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/icons/icon-info-grey.svg'
extracted valid child url: 'https://www.scrapingbee.com/images/team/kevin.png'
extracted valid child url: 'https://www.scrapingbee.com/images/team/pierre.png'
extracted valid child url: 'https://www.scrapingbee.com/images/team/etienne.png'
extracted valid child url: 'https://www.scrapingbee.com/images/logo-white.svg'
extracted valid child url: 'https://www.scrapingbee.com/scripts/app-landing.js'
crawler has finished crawling: 'https://www.scrapingbee.com/#features'
starting crawler on: 'https://www.scrapingbee.com/images/icons/nocode.svg'
content for target url 'https://www.scrapingbee.com/images/icons/nocode.svg' found under file path: 'scrapping-bee/images/icons/nocode.svg'
starting crawler on: 'https://www.scrapingbee.com/#pricing'
content for target url 'https://www.scrapingbee.com/#pricing' found under file path: 'scrapping-bee.html'
^C
crawler has been interrupted

Tests:

go test