Tool for scraping the work URLs off of any AO3 page. Capable of navigating the
depths of index pages. Designed for use with the
FanFicFare extension for Calibre.
Notes for AO3 Maintainers
This crawler uses the user-agent string legowerewolf-ao3scraper/[commit].
The crawler enforces a maximum request rate of 1 request per 10 seconds.
It will also obey Retry-After headers if they are set in the response.
I am more than happy to make changes if requested.
Arguments
-url (string) URL to start crawling from
-delay (int) Delay between requests in seconds (default 10)
-login (string) Login credentials in the form of username:password
-pages (int) Number of pages to crawl (default 1)
-progress (boolean) Show progress bar (default true, disable with
-progress=false)
-series (boolean) Include series in the crawl (default true, disable with
-series=false)
-version (boolean) Show version and exit (default false)
-outputFile (string) Write the list of collected works to a file instead of
the terminal output.