Synchronization

This chapter is from the book

Go Fundamentals: Gopher Guides

Learn More Buy

This chapter is from the book

This chapter is from the book 

Go Fundamentals: Gopher Guides

Learn More Buy

Error Management with Error Groups

One of the downsides of the sync.WaitGroup is that it has no error management built in to capture errors that occur in the goroutines. It also has an API that requires exact implementation; otherwise, it panics.

To help address some of these issues, the golang.org/x/sync/errgroup,¹⁰ Listing 13.15, package was introduced, providing a simpler API as well as built-in error management.

Listing 13.15 The golang.org/x/sync/errgroup Package

$ go doc golang.org/x/sync/errgroup

package errgroup // import "golang.org/x/sync/errgroup"

Package errgroup provides synchronization, error propagation, and Context
cancelation for groups of goroutines working on subtasks of a common task.

type Group struct{ ... }
    func WithContext(ctx context.Context) (*Group, context.Context)

Go Version: go1.19

The Problem

Consider Listing 13.16. In the example, a number of goroutines are launched to call the generateThumbnail functions. A sync.WaitGroup is used to wait for all the goroutines to finish.

Listing 13.16 Managing Goroutines with sync.WaitGroup

func Test_ThumbnailGenerator(t *testing.T) {
    t.Parallel()

    // image that we need thumbnails for
    const image = "foo.png"

    var wg sync.WaitGroup

    // start 5 goroutines to generate thumbnails
    for i := 0; i < 5; i++ {
        wg.Add(1)

        // start a new goroutine for each thumbnail
        go generateThumbnail(&wg, image, i+1)

    }

    fmt.Println("Waiting for thumbnails to be generated")

    // wait for all goroutines to finish
    wg.Wait()

    fmt.Println("Finished generate all thumbnails")
}

Inside the generateThumbnail function, Listing 13.17, we see that there is an error that occurs if the size argument is divisible by 5, which causes a panic, and the application crashes.

Listing 13.17 The generateThumbnail Function

func generateThumbnail(wg *sync.WaitGroup, image string, size int) {
    defer wg.Done()

    // error if the size is divisible by 5
    if size%5 == 0 {
        // how do we return this error back to the main
        // goroutine without panicking?
        err := fmt.Errorf("%d is divisible by 5", size)
        panic(err)
    }

    // thumbnail to be generated
    thumb := fmt.Sprintf("%s@%dx.png", image, size)

    fmt.Println("Generating thumbnail:", thumb)

    // wait for the thumbnail to be ready
    time.Sleep(time.Millisecond * time.Duration(size))

    fmt.Println("Finished generating thumbnail:", thumb)
}

$ go test -v

=== RUN   Test_ThumbnailGenerator
=== PAUSE Test_ThumbnailGenerator
=== CONT  Test_ThumbnailGenerator
Waiting for thumbnails to be generated
panic: 5 is divisible by 5
goroutine 9 [running]:
demo.generateThumbnail(0x0?, {0x10268ac77?, 0x0?}, 0x0?)
        ./demo_test.go:47 +0x21c
created by demo.Test_ThumbnailGenerator
        ./demo_test.go:24 +0x4c
exit status 2
FAIL    demo    0.619s

Go Version: go1.19

The Error Group

The errgroup.Group¹¹ type provides a minimal API when compared to the sync.WaitGroup type. There are only two methods: errgroup.Group.Go¹² and errgroup.Group.Wait.¹³

The errgroup.Group type, Listing 13.18, manages the counter for you, so there is no need for counter management as there is with the sync.WaitGroup type.

Listing 13.18 The errgroup.Group Type

$ go doc golang.org/x/sync/errgroup.Group

package errgroup // import "golang.org/x/sync/errgroup"

type Group struct {
        // Has unexported fields.
}
    A Group is a collection of goroutines working on subtasks that are part
    of the same overall task.

    A zero Group is valid and does not cancel on error.

func WithContext(ctx context.Context) (*Group, context.Context)
func (g *Group) Go(f func() error)
func (g *Group) Wait() error

Go Version: go1.19

The Go Method

You use the errgroup.Group.Go method, Listing 13.19, to launch a goroutine for the provided func() error function provided. The func() error function is executed in a goroutine. If the function returns an error, the errgroup.Group type captures the error, cancels the other goroutines, and returns the error to the caller from the errgroup.Group.Wait method.

Listing 13.19 The errgroup.Group.Go Method

$ go doc golang.org/x/sync/errgroup.Group.Go

package errgroup // import "golang.org/x/sync/errgroup"

func (g *Group) Go(f func() error)
    Go calls the given function in a new goroutine.

    The first call to return a non-nil error cancels the group; its error
    will be returned by Wait.

Go Version: go1.19

The Wait Method

You use errgroup.Group.Wait method, Listing 13.20, to wait for all the goroutines to finish. If any of the goroutines return an error, the errgroup.Group type returns the error to the caller.

Listing 13.20 The errgroup.Group.Wait Method

$ go doc golang.org/x/sync/errgroup.Group.Wait

package errgroup // import "golang.org/x/sync/errgroup"

func (g *Group) Wait() error
    Wait blocks until all function calls from the Go method have returned,
    then returns the first non-nil error (if any) from them.

Go Version: go1.19

It is important to understand the errgroup.Group.Wait method returns only the first error that occurs. Any other errors that occur are ignored. In Listing 13.21, we are calling the Go method on an errorgroup.Group 10 times. Each time we pass a function that sleeps for a random time, prints a messages, and then returns an error. From the output of the test, you can see that the error returned by the Wait method was returned from function number 4. The other nine errors are discarded.

Listing 13.21 The errgroup.Group.Wait Method Returns Only the First Error

func Test_ErrorGroup_Multiple_Errors(t *testing.T) {
    t.Parallel()

    var wg errgroup.Group

    for i := 0; i < 10; i++ {
        i := i + 1

        wg.Go(func() error {

            time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))

            fmt.Printf("about to error from %d\n", i)

            return fmt.Errorf("error %d", i)
        })
    }

    err := wg.Wait()
    if err != nil {
        t.Fatal(err)
    }

}

$ go test -v

=== RUN   Test_ErrorGroup_Multiple_Errors
=== PAUSE Test_ErrorGroup_Multiple_Errors
=== CONT  Test_ErrorGroup_Multiple_Errors
about to error from 4
about to error from 6
about to error from 3
about to error from 1
about to error from 9
about to error from 10
about to error from 8
about to error from 5
about to error from 2
about to error from 7
    demo_test.go:38: error 4
--- FAIL: Test_ErrorGroup_Multiple_Errors (0.01s)
FAIL
exit status 1
FAIL    demo    0.263s

Go Version: go1.19

Listening for Error Group Cancellation

When launching a number of goroutines, it can often be useful to let others know the tasks have all completed. The errgroup.WithContext¹⁴ function, Listing 13.22, returns a new errgroup.Group, as well as a context.Context that can be listened to for cancellation.

Listing 13.22 The errgroup.WithContext Function

func Test_ErrorGroup_Context(t *testing.T) {
    t.Parallel()

    // create a new error group
    // and a context that will be canceled
    // when the group is done
    wg, ctx := errgroup.WithContext(context.Background())

    // create a quit channel for the goroutine
    // waiting for the context to be canceled
    // can close to signal the goroutine has finished
    quit := make(chan struct{})

    // launch a goroutine that will
    // wait for the errgroup context to finish
    go func() {
        fmt.Println("waiting for context to cancel")

        // wait for the context to be canceled
        <-ctx.Done()

        fmt.Println("context canceled")

        // close the quit channel so the test
        // will finish
        close(quit)
    }()

    // add a task to the errgroup
    wg.Go(func() error {
        time.Sleep(time.Millisecond * 5)
        return nil
    })

    // wait for the errgroup to finish
    err := wg.Wait()
    if err != nil {
        t.Fatal(err)
    }

    // wait for the context goroutine to finish
    <-quit
}

$ go test -v

=== RUN   Test_ErrorGroup_Context
=== PAUSE Test_ErrorGroup_Context
=== CONT  Test_ErrorGroup_Context
waiting for context to cancel
context canceled
--- PASS: Test_ErrorGroup_Context (0.01s)
PASS
ok      demo    0.725s

Go Version: go1.19

Wrapping Up Error Groups

The errgroup.Group type provides a simpler API than the sync.WaitGroup type. It also provides built-in error management. This makes managing goroutines with error handling much easier. The downside is that it is not as flexible as the sync.WaitGroup type, where you are in charge of managing the counter.

Which type you choose to use will vary from situation to situation, so it is important to understand the tradeoffs and benefits of each before deciding which type to use.

Using a errgroup.Group allows you to clean up your code significantly to make it easier to understand and to manage errors.

As you can see from Listing 13.23, the generateThumbnail function no longer needs to take a sync.WaitGroup as an argument.

Listing 13.23 Update the generateThumbnail Function to use errgroup.Group

func generateThumbnail(image string, size int) error {

    // error if the size is divisible by 5
    if size%5 == 0 {
        return fmt.Errorf("%d is divisible by 5", size)
    }

    // thumbnail to be generated
    thumb := fmt.Sprintf("%s@%dx.png", image, size)

    fmt.Println("Generating thumbnail:", thumb)

    // wait for the thumbnail to be ready
    time.Sleep(time.Millisecond * time.Duration(size))

    fmt.Println("Finished generating thumbnail:", thumb)

    return nil
}

Being able to return an error from the function means the function no longer needs to panic.

As you can see from the output, Listing 13.24, the generateThumbnail function no longer panics, and the test is now able to exit properly.

Listing 13.24 Using the errgroup.Group Type

func Test_ThumbnailGenerator(t *testing.T) {
    t.Parallel()

    // image that we need thumbnails for
    const image = "foo.png"

    // create a new error group
    var wg errgroup.Group

    // start 5 goroutines to generate thumbnails
    for i := 0; i < 5; i++ {

        // capture the i to the current scope
        i := i

        // start a new goroutine for each thumbnail
        wg.Go(func() error {

            // return the result of generateThumbnail
            return generateThumbnail(image, i)
        })

    }

    fmt.Println("Waiting for thumbnails to be generated")

    // wait for all goroutines to finish
    err := wg.Wait()

    // check for any errors
    if err != nil {
        t.Fatal(err)
    }

    fmt.Println("Finished generate all thumbnails")
}

$ go test -v

=== RUN   Test_ThumbnailGenerator
=== PAUSE Test_ThumbnailGenerator
=== CONT  Test_ThumbnailGenerator
Waiting for thumbnails to be generated
Generating thumbnail: foo.png@4x.png
Generating thumbnail: foo.png@3x.png
Generating thumbnail: foo.png@1x.png
Generating thumbnail: foo.png@2x.png
Finished generating thumbnail: foo.png@1x.png
Finished generating thumbnail: foo.png@2x.png
Finished generating thumbnail: foo.png@3x.png
Finished generating thumbnail: foo.png@4x.png
    demo_test.go:43: 0 is divisible by 5
--- FAIL: Test_ThumbnailGenerator (0.00s)
FAIL
exit status 1
FAIL    demo    0.570s

Go Version: go1.19