distchan

distchan

Package distchan enables Go channels to be used for distributed computation. Improved stable version by covrom.

Why?

While Go's concurrency story around single-process data pipelines is great, its ability to distribute workloads across multiple machines is relatively lacking. There are several options here (notably glow and gleam), but they lack the type-safety and ease-of-use that Go's built-in channels provide.

How?

In a nutshell: standard net primitives, Gob encoding, and reflection.

Architecturally, the package assumes a client-server model of workload distribution. One server can have as many clients connected to it as you want, and work will be distributed across them using a simple round-robin algorithm.

Example

As a simple example, let's say that capitalizing letters in a string is very computationally expensive, and you want to distribute that work across a number of nodes. First, you'll need to create a server in charge of defining the work to be done:

package main

import (
	"log"
	"net"

	"github.com/covrom/distchan"
)

func main() {
	ln, err := net.Listen("tcp", "localhost:5678")
	if err != nil {
		log.Fatal(err)
	}

	var (
		out    = make(chan string)
		in     = make(chan string)
		server = distchan.NewServer(ln, out, in)
	)

	server.Start()
	server.WaitUntilReady() // wait until we have at least one worker available

	go producer(out)

	for s := range in {
		println(s)
	}
}

func producer(out chan<- string) {
	// send strings to be capitalized to out
	out <- "hello world"
	// don't forget to close the channel! this is how all connected
	// clients know that there's no more work coming.
	close(out)
}

Then you'll need to create a client, or worker. It's similarly easy to get wired up, so you can focus on the hard part: capitalizing strings:

package main

import (
	"log"
	"net"
	"strings"

	"github.com/covrom/distchan"
)

func main() {
	conn, err := net.Dial("tcp", "localhost:5678") // must be able to connect to the server
	if err != nil {
		log.Fatal(err)
	}

	var (
		out = make(chan string)
		in  = make(chan string)
	)

	client := distchan.NewClient(conn, out, in).Start()

	// Loop over all input from the server...
	for input := range in {
		capitalized := strings.ToUpper(input)
		// ...and send the results back.
		out <- capitalized
	}

	close(out)
	<-client.Done()
}

Check out the example folder for more examples.