webDownloader

webDownloader v0.1

Golang implementation of a method of downloading websites - built for our Senior Project Peer to Peer Web Cache.

This project is ment for education purposes as a tool in order to develop a peer to peer web cache capability. The repository to that one is currently private, though this tool seems universally useful so I felt should be seperate and public.

Usage: webDownloader {Website URL}

Will contain more options in potential future releases

To-Do

• Downloads HTML Webpages
• Saves them in an organized matter
• Identifies assets
• Downloads Assets
• Organizes and saves assets
• Replaces source location with local path
• Index's and saves table of downloaded webpages
• Connects with the P2P Senior Project Peer To Peer Webcache
• Handles robots.txt to avoid downloading unauthorized information
• Can take in multiple options to handle either the full webpage, specific webpage, directory of webpages etc.