gengen

gengen - The Generator Generator

Gengen is a tool and a library for creating and using Python-style Generators in Go.

Gengen is still a work in progress (see Known Issues).

What Are Generators

Generators are a simple and straightforward way to create Iterators.

Iterators implement the following interface:

type Iterator[T any] interface {
    Next() bool
    Value() T
    Error() error
}

And are used as follows:

// First, we create a new iterator
iterator := NewIterator()

// Then we use the `Next()` method to iterate over it
for iterator.Next() {
	// And the `Value()` method to get the value from it
	value := iterator.Value()
	fmt.Println(value)
}

// Finally, we use the `Error()` method to tell whether the
// iterator is truly exhausted, or whether we stopped iteration
// due to an error.
if iterator.Error() != nil {
	panic(iterator.Error())
}

Iterators allow us to iterate lazy data, generating or fetching what we need on demand. They are great to use, but are difficult to write and maintain. This is where Generators come in.

Generator Syntax

Generators are created using a Generator-Function:

// Generator-functions return a `type gengen.Generator[T]`
func Range(stop int) gengen.Generator[int] {
	for i := 0; i < stop; i++ {
		// Use `func gengen.Yield(value T)` to yield values
		gengen.Yield(i)
	}
	// Use `return someError` to stop iteration and report errors 
	return nil
}

• When called, generator-functions return a Generator, but don't execute any code yet.
• When Next() is called on the generator, the generator-function is executed until it reaches gengen.Yield or return
• When encountering a gengen.Yield(value), Next() will return true and Value() will return the value passed to gengen.Yield
• When encountering return someError, Next() will return false, stopping the iteration and Error() will return someError. If no error occurred - return nil to stop iteration.

Generating Generators (Tutorial)

Since Generators are not a part of Go, but rather some pretend-Go syntax, we can't use them directly. Instead, gengen uses code generation to implement them for you in real-Go.

You can follow along and write the code, or clone the demo from gengen-demo.

Setup

To use the gengen command, Go's tooling needs to know about it and fetch it. To do that, we add the following file to our module:

File: tools.go

//go:build tools
package main
import (
	_ "github.com/tmr232/gengen/cmd/gengen"
)

Once you have this in place, run go mod tidy. Having the import will ensure go mod tidy fetches the gengen command, and the build tag will ensure this code is not built into our project.

Writing The Code

The Range sample, in a real project, will look as follows:

File: demo.go

//go:build gengen

package main

import (
	"fmt"
	"github.com/tmr232/gengen"
)

//go:generate go run github.com/tmr232/gengen/cmd/gengen

func Range(stop int) gengen.Generator[int] {
	for i := 0; i < stop; i++ {
		gengen.Yield(i)
	}
	return nil
}

func main() {
	numberRange := Range(10)
	for numberRange.Next() {
		fmt.Println(numberRange.Value())
    }
	if numberRange.Error() != nil {
		panic(numberRange.Error())
    }
}

It has 3 key parts, except the generator itself:

1. It uses the gengen build-tag to separate this pretend-Go from real-Go. This is a requirement for the tool to work.
2. It imports and uses github.com/tmr232/gengen for gengen.Generator and gengen.Yield. This, tool is required.
3. It uses go:generate to run github.com/tmr232/gengen/cmd/gengen and generate the actual real-Go generator implementation from the definition shown here.

Generating & Running

To generate the generator implementation, run the following command:

go generate -tags gengen

This will create the following file:

File: demo_gengen.go

//go:build !gengen

// AUTOGENERATED DO NOT MODIFY

package main

import (
	"fmt"
	"github.com/tmr232/gengen"
)

func Range(stop int) gengen.Generator[int] {
	var i int
	__next := 0
	return gengen.MakeGenerator[int](
		func(__withValue func(value int) bool, __withError func(err error) bool, __exhausted func() bool) bool {
			switch __next {
			case 0:
				goto __Next0
			case 1:
				goto __Next1
			}
		__Next0:
			i = 0
		__Head1:
			if i < stop {
				goto __Body1
			} else {
				goto __After1
			}
		__Body1:
			__next = 1
			return __withValue(i)
		__Next1:
			i++
			goto __Head1
		__After1:
			return __exhausted()
		},
	)
}

func main() {
	numberRange := Range(10)
	for numberRange.Next() {
		fmt.Println(numberRange.Value())
	}
	if numberRange.Error() != nil {
		panic(numberRange.Error())
	}
}

Key changes are:

1. We now have !gengen as a build-tag, to separate from the pretend-Go definitions, and avoid conflicts.
2. The go:generate directive is gone.
3. The Range function has been replaced with a real-Go implementation of the generator-definition we wrote.

You can now use go run . to execute the code and get:

0
1
2
3
4
5
6
7
8
9

Known Issues

Code-analysis & code-generation are both hard. If you try to break this code - you'll definitely succeed. If you don't try to break this code - you'll probably break it regardless...

There are probably a lot of issues I am unaware of. But there are also known ones.

Bug reports are very welcome.

Unsupported Syntax

• go
• switch
• select
• Nested functions
• Anonymous types
• Type assertions

I plan to add support for all of these in the future.

Defer

Go's defer will not be supported.

In a generator context, there is no obvious time to defer a call to:

• On generation exhaustion - this will mean that the deferred call will only happen if we exhaust the generator. If we don't exhaust it - we're likely to leak resources.
• On calls to Next() - this is possible, doesn't make much sense as that can happen many times.

For that reason, if you need to manage resources for your generator - please do so outside the generator itself. Pass the initialized resource to the generator as an argument, and close it outside the generator when it is no longer needed.

Documentation

While the user-facing API is reasonably documented, the implementation of the code-generation is, well, not. The code is also expected to change quite a bit as I improve the implementation.