batchutil

batchutil

batchutil is a package for Go that provides functions to describe a concurrent batch processing easily. If you just set configurations and a function, you can run a concurrent program. It's very useful when you write a big batch processing like data migrations.

Install

Install using go get github.com/tkm-kj/batchutil.

Usage

Define configuration

cfg := batchutil.NewConfig(
    3,  // [concurrentLimit] limit value of concurrency(It's assuming database max connection)
    1,    // [startNumber] loop start number
    1450, // [endNumber] loop end number
    100,  // [batchSize] the size of batch(StartNumber is incremented by BatchSize until it exceeds EndNumber)
)

When you set 1 to ConcurrentLimit, batch processing runs serially.

Run batch processing

util := batchutil.NewUtil(cfg)
f := func(min, max int64) error {
    log.Printf("min: %d, max: %d", min, max)
    // I assume that you'll find list by `min` and `max` and do something
    return nil
}
err := util.Run(f)

The result is below.

2020/10/22 18:33:36 min: 1401, max: 1501
2020/10/22 18:33:36 min: 301, max: 401
2020/10/22 18:33:36 min: 1, max: 101
2020/10/22 18:33:36 min: 101, max: 201
2020/10/22 18:33:36 min: 501, max: 601
2020/10/22 18:33:36 min: 1301, max: 1401
2020/10/22 18:33:36 min: 201, max: 301
2020/10/22 18:33:36 min: 901, max: 1001
2020/10/22 18:33:36 min: 701, max: 801
2020/10/22 18:33:36 min: 801, max: 901
2020/10/22 18:33:36 min: 601, max: 701
2020/10/22 18:33:36 min: 1101, max: 1201
2020/10/22 18:33:36 min: 1001, max: 1101
2020/10/22 18:33:36 min: 1201, max: 1301
2020/10/22 18:33:36 min: 401, max: 501

Batch processing runs concurrently!

Run batch processing with context

Also, you can use batchutil with context.Context too. In other words, you can just skip batch process when an error occurs.

util := batchutil.NewUtil(cfg)
f := func(ctx context.Context, min, max int64) error {
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
    }

    if min == 101 {
        return errors.New("error")
    }
    log.Printf("min: %d, max: %d", min, max)
    return nil
}
err := util.RunWithContext(context.Background(), f)

The result is below.

2020/12/29 00:07:33 min: 201, max: 301
2020/12/29 00:07:33 min: 1, max: 101