- Waiting for Goroutines with a WaitGroup
- Error Management with Error Groups
- Data Races
- Synchronizing Access with a Mutex
- Performing Tasks Only Once
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Error Management with Error Groups
One of the downsides of the sync.WaitGroup is that it has no error management built in to capture errors that occur in the goroutines. It also has an API that requires exact implementation; otherwise, it panics.
To help address some of these issues, the golang.org/x/sync/errgroup,10 Listing 13.15, package was introduced, providing a simpler API as well as built-in error management.
Listing 13.15 The golang.org/x/sync/errgroup Package
$ go doc golang.org/x/sync/errgroup package errgroup // import "golang.org/x/sync/errgroup" Package errgroup provides synchronization, error propagation, and Contextcancelation for groups of goroutines working on subtasks of a common task. type Group struct{ ... } func WithContext(ctx context.Context) (*Group, context.Context)
Go Version: go1.19
The Problem
Consider Listing 13.16. In the example, a number of goroutines are launched to call the generateThumbnail functions. A sync.WaitGroup is used to wait for all the goroutines to finish.
Listing 13.16 Managing Goroutines with sync.WaitGroup
func Test_ThumbnailGenerator(t *testing.T) {
t.Parallel()
// image that we need thumbnails for
const image = "foo.png"
var wg sync.WaitGroup
// start 5 goroutines to generate thumbnails
for i := 0; i < 5; i++ {
wg.Add(1)
// start a new goroutine for each thumbnail
go generateThumbnail(&wg, image, i+1)
}
fmt.Println("Waiting for thumbnails to be generated")
// wait for all goroutines to finish
wg.Wait()
fmt.Println("Finished generate all thumbnails")
}
Inside the generateThumbnail function, Listing 13.17, we see that there is an error that occurs if the size argument is divisible by 5, which causes a panic, and the application crashes.
Listing 13.17 The generateThumbnail Function
func generateThumbnail(wg *sync.WaitGroup, image string, size int) {
defer wg.Done()
// error if the size is divisible by 5
if size%5 == 0 {
// how do we return this error back to the main
// goroutine without panicking?
err := fmt.Errorf("%d is divisible by 5", size)
panic(err)
}
// thumbnail to be generated
thumb := fmt.Sprintf("%s@%dx.png", image, size)
fmt.Println("Generating thumbnail:", thumb)
// wait for the thumbnail to be ready
time.Sleep(time.Millisecond * time.Duration(size))
fmt.Println("Finished generating thumbnail:", thumb)
}
$ go test -v
=== RUN Test_ThumbnailGenerator
=== PAUSE Test_ThumbnailGenerator
=== CONT Test_ThumbnailGenerator
Waiting for thumbnails to be generated
panic: 5 is divisible by 5
goroutine 9 [running]:
demo.generateThumbnail(0x0?, {0x10268ac77?, 0x0?}, 0x0?)
./demo_test.go:47 +0x21c
created by demo.Test_ThumbnailGenerator
./demo_test.go:24 +0x4c
exit status 2
FAIL demo 0.619s
Go Version: go1.19
The Error Group
The errgroup.Group11 type provides a minimal API when compared to the sync.WaitGroup type. There are only two methods: errgroup.Group.Go12 and errgroup.Group.Wait.13
The errgroup.Group type, Listing 13.18, manages the counter for you, so there is no need for counter management as there is with the sync.WaitGroup type.
Listing 13.18 The errgroup.Group Type
$ go doc golang.org/x/sync/errgroup.Group
package errgroup // import "golang.org/x/sync/errgroup"
type Group struct {
// Has unexported fields.
}
A Group is a collection of goroutines working on subtasks that are part
of the same overall task.
A zero Group is valid and does not cancel on error.
func WithContext(ctx context.Context) (*Group, context.Context)
func (g *Group) Go(f func() error)
func (g *Group) Wait() error
Go Version: go1.19
The Go Method
You use the errgroup.Group.Go method, Listing 13.19, to launch a goroutine for the provided func() error function provided. The func() error function is executed in a goroutine. If the function returns an error, the errgroup.Group type captures the error, cancels the other goroutines, and returns the error to the caller from the errgroup.Group.Wait method.
Listing 13.19 The errgroup.Group.Go Method
$ go doc golang.org/x/sync/errgroup.Group.Go
package errgroup // import "golang.org/x/sync/errgroup"
func (g *Group) Go(f func() error)
Go calls the given function in a new goroutine.
The first call to return a non-nil error cancels the group; its error
will be returned by Wait.
Go Version: go1.19
The Wait Method
You use errgroup.Group.Wait method, Listing 13.20, to wait for all the goroutines to finish. If any of the goroutines return an error, the errgroup.Group type returns the error to the caller.
Listing 13.20 The errgroup.Group.Wait Method
$ go doc golang.org/x/sync/errgroup.Group.Wait
package errgroup // import "golang.org/x/sync/errgroup"
func (g *Group) Wait() error
Wait blocks until all function calls from the Go method have returned,
then returns the first non-nil error (if any) from them.
Go Version: go1.19
It is important to understand the errgroup.Group.Wait method returns only the first error that occurs. Any other errors that occur are ignored. In Listing 13.21, we are calling the Go method on an errorgroup.Group 10 times. Each time we pass a function that sleeps for a random time, prints a messages, and then returns an error. From the output of the test, you can see that the error returned by the Wait method was returned from function number 4. The other nine errors are discarded.
Listing 13.21 The errgroup.Group.Wait Method Returns Only the First Error
func Test_ErrorGroup_Multiple_Errors(t *testing.T) {
t.Parallel()
var wg errgroup.Group
for i := 0; i < 10; i++ {
i := i + 1
wg.Go(func() error {
time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))
fmt.Printf("about to error from %d\n", i)
return fmt.Errorf("error %d", i)
})
}
err := wg.Wait()
if err != nil {
t.Fatal(err)
}
}
$ go test -v
=== RUN Test_ErrorGroup_Multiple_Errors
=== PAUSE Test_ErrorGroup_Multiple_Errors
=== CONT Test_ErrorGroup_Multiple_Errors
about to error from 4
about to error from 6
about to error from 3
about to error from 1
about to error from 9
about to error from 10
about to error from 8
about to error from 5
about to error from 2
about to error from 7
demo_test.go:38: error 4
--- FAIL: Test_ErrorGroup_Multiple_Errors (0.01s)
FAIL
exit status 1
FAIL demo 0.263s
Go Version: go1.19
Listening for Error Group Cancellation
When launching a number of goroutines, it can often be useful to let others know the tasks have all completed. The errgroup.WithContext14 function, Listing 13.22, returns a new errgroup.Group, as well as a context.Context that can be listened to for cancellation.
Listing 13.22 The errgroup.WithContext Function
func Test_ErrorGroup_Context(t *testing.T) {
t.Parallel()
// create a new error group
// and a context that will be canceled
// when the group is done
wg, ctx := errgroup.WithContext(context.Background())
// create a quit channel for the goroutine
// waiting for the context to be canceled
// can close to signal the goroutine has finished
quit := make(chan struct{})
// launch a goroutine that will
// wait for the errgroup context to finish
go func() {
fmt.Println("waiting for context to cancel")
// wait for the context to be canceled
<-ctx.Done()
fmt.Println("context canceled")
// close the quit channel so the test
// will finish
close(quit)
}()
// add a task to the errgroup
wg.Go(func() error {
time.Sleep(time.Millisecond * 5)
return nil
})
// wait for the errgroup to finish
err := wg.Wait()
if err != nil {
t.Fatal(err)
}
// wait for the context goroutine to finish
<-quit
}
$ go test -v === RUN Test_ErrorGroup_Context === PAUSE Test_ErrorGroup_Context === CONT Test_ErrorGroup_Context waiting for context to cancel context canceled --- PASS: Test_ErrorGroup_Context (0.01s) PASS ok demo 0.725s
Go Version: go1.19
Wrapping Up Error Groups
The errgroup.Group type provides a simpler API than the sync.WaitGroup type. It also provides built-in error management. This makes managing goroutines with error handling much easier. The downside is that it is not as flexible as the sync.WaitGroup type, where you are in charge of managing the counter.
Which type you choose to use will vary from situation to situation, so it is important to understand the tradeoffs and benefits of each before deciding which type to use.
Using a errgroup.Group allows you to clean up your code significantly to make it easier to understand and to manage errors.
As you can see from Listing 13.23, the generateThumbnail function no longer needs to take a sync.WaitGroup as an argument.
Listing 13.23 Update the generateThumbnail Function to use errgroup.Group
func generateThumbnail(image string, size int) error {
// error if the size is divisible by 5
if size%5 == 0 {
return fmt.Errorf("%d is divisible by 5", size)
}
// thumbnail to be generated
thumb := fmt.Sprintf("%s@%dx.png", image, size)
fmt.Println("Generating thumbnail:", thumb)
// wait for the thumbnail to be ready
time.Sleep(time.Millisecond * time.Duration(size))
fmt.Println("Finished generating thumbnail:", thumb)
return nil
}
Being able to return an error from the function means the function no longer needs to panic.
As you can see from the output, Listing 13.24, the generateThumbnail function no longer panics, and the test is now able to exit properly.
Listing 13.24 Using the errgroup.Group Type
func Test_ThumbnailGenerator(t *testing.T) {
t.Parallel()
// image that we need thumbnails for
const image = "foo.png"
// create a new error group
var wg errgroup.Group
// start 5 goroutines to generate thumbnails
for i := 0; i < 5; i++ {
// capture the i to the current scope
i := i
// start a new goroutine for each thumbnail
wg.Go(func() error {
// return the result of generateThumbnail
return generateThumbnail(image, i)
})
}
fmt.Println("Waiting for thumbnails to be generated")
// wait for all goroutines to finish
err := wg.Wait()
// check for any errors
if err != nil {
t.Fatal(err)
}
fmt.Println("Finished generate all thumbnails")
}
$ go test -v
=== RUN Test_ThumbnailGenerator
=== PAUSE Test_ThumbnailGenerator
=== CONT Test_ThumbnailGenerator
Waiting for thumbnails to be generated
Generating thumbnail: foo.png@4x.png
Generating thumbnail: foo.png@3x.png
Generating thumbnail: foo.png@1x.png
Generating thumbnail: foo.png@2x.png
Finished generating thumbnail: foo.png@1x.png
Finished generating thumbnail: foo.png@2x.png
Finished generating thumbnail: foo.png@3x.png
Finished generating thumbnail: foo.png@4x.png
demo_test.go:43: 0 is divisible by 5
--- FAIL: Test_ThumbnailGenerator (0.00s)
FAIL
exit status 1
FAIL demo 0.570s
Go Version: go1.19