Go Channels buffering, iteration, and synchronization

Welcome to part 23 of the Go programming tutorial, where we'll be getting a little deeper into Go channels. In the previous tutorial, we covered how to use channels to send and receive values with goroutines. That said, it was just a basic example. In reality, we're likely to have questions of synchronization and iterating through known, or unknown, numbers of channel returns. Code up to this point:

package main

import "fmt"

func foo(c chan int, someValue int) {
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int)
    go foo(fooVal, 5)
    go foo(fooVal, 3)
    v1 := <-fooVal
    v2 := <-fooVal
    fmt.Println(v1, v2)
}

In the above example, immediately, you should be wondering something along the lines of "How come we didn't need to have a wait group for these routines to finish?"

Channels by default are blocking on sending and receiving values, so they will be waited on. Let's consider though that we might not actually know how many channels we have, or maybe we just have a lot and we know we shouldn't be hard-coding the returns. Instead, we might want to iterate over the channel data, however long it might be. Let's say instead, we want to do:

package main

import "fmt"

func foo(c chan int, someValue int) {
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int)

    for i := 0; i < 10; i++ {
        go foo(fooVal, i)
    }
}

Once again, one of the very few magical operators in go, range works wonders with channels too! We can iterate over a channel, where our channel is called fooVal, by doing:

    for item := range fooVal {
        fmt.Println(item)
    }

All together:

package main

import "fmt"

func foo(c chan int, someValue int) {
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int)

    for i := 0; i < 10; i++ {
        go foo(fooVal, i)
    }

    for item := range fooVal {
        fmt.Println(item)
    }
}

This will run, but will error out. Output is something like:

45
20
0
5
10
15
30
25
40
35
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
main.main()
        C:/Users/H/Desktop/golangstuff/gotut22-channels.go:16 +0xb2
exit status 2

We've seen this issue before. This happens when we're still waiting around for something, but there's nothing to be waited on anymore. What are we waiting for here? Well, our channel has no buffer, and the range statement doesn't have any sort of buffer or anything, it's just waiting and waiting. I am fairly impressed it even iterates at all, giving me the impression that range is actually even more magical than we were thinking. Anyway, the channel remains open. This is a problem. We can close the channel with close(), doing close(fooVal)

package main

import "fmt"

func foo(c chan int, someValue int) {
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int)

    for i := 0; i < 10; i++ {
        go foo(fooVal, i)
    }

    close(fooVal)

    for item := range fooVal {
        fmt.Println(item)
    }
}

Oh, great, we have no more errors... but now we see nothing. Something is still wrong. You might be thinking "oh, I've seen this before, we just need to synchronize!" You are right that we need to synchronize, but we need to do it for more reasons than just because the goroutines aren't finishing before the program. We can illustrate that with adding a sleep:

package main

import (
    "fmt"
    "time"
)

func foo(c chan int, someValue int) {
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int)
    for i := 0; i < 10; i++ {
        go foo(fooVal, i)
    }
    close(fooVal)
    for item := range fooVal {
        fmt.Println(item)
    }
    time.Sleep(time.Second*2)
}

This gives us a panic: panic: send on closed channel

This is because we're getting to the close(), not the end of the program, before the goroutines can finish and send the data. Still, however, we can use the sync package to solve our problem here. Let's import sync, create our wait group: var wg sync.WaitGroup, use wg.Add(1) before each goroutine, and do a defer wg.Done() inside the goroutine. Finally, do a wg.Wait() before we close the channel.

Finally, we want our channels to be non-blocking. We don't need them to be blocking. The blocking aspect of channels handles a form of synchronization, but we're controlling flow now, and letting the channels block each other will be problematic. Buffers on channels goes by items. In our case, we have 10 items, so we can add a buffer of 10 when we create our channel by doing: fooVal := make(chan int, 10). In this case, we need to be certain the buffer is higher than the amount of channels we're attempting to synchronize together here. We can do 10, or we could do 50, or 500. We couldn't do 5, for example.

With the following code:

package main

import (
    "fmt"
    "sync"
)

var wg sync.WaitGroup

func foo(c chan int, someValue int) {
    defer wg.Done()
    c <- someValue * 5
}

func main() {
    fooVal := make(chan int, 10)
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go foo(fooVal, i)
    }
    wg.Wait()
    close(fooVal)
    for item := range fooVal {
        fmt.Println(item)
    }
}

We get:

45
20
25
30
35
40
15
5
0
10

Okay, so at this point, we're basically go channel pros. We're ready to apply goroutines and channels to our webapp, and hope for the best!

The next tutorial: Adding Concurrency to speed up our Golang Web Application