download_tools

Concurrent downloading tools in Go

Here is one of my firsts projects working with the Go programming language. I wanted to write some simple programs taking advantage of goroutines, channels, and concurrent programming in Go. This is somewhat easier and more native to do in Go compared to say Python where you have to use asyncio and aiohttp and manage more of the blocking/non-blocking code yourself.

It seems here that I/O is still very much the bottle neck, trying to write the responses from the urls to disk appears to be the bottleneck, for plain text urls in a CDN I get 20x speedup with roughly 100 workers optimal, urls such as baidu with long latency and RTT can see a 200x improvement in performance!

Benchmarks with Slow Url (High RTT) and no File I/O

Here we can see a 200x speedup over a simple synchronous single worker