How to Go
Worker Pools
In this example we'll look at how to implement a worker pool using goroutines and channels. A worker pool is a common concurrency pattern for limiting the number of concurrently running tasks.
Key Points
- Worker pools are a common concurrency pattern for limiting the number of concurrently running tasks.
- A pool of worker goroutines is started, and they all read from a jobs channel.
- The main goroutine sends jobs to the jobs channel and then closes it when all jobs have been sent.
- Results are collected on a separate results channel.
- This pattern is useful for processing a large number of tasks without overwhelming the system.
Worker Pool Example
This example shows a worker pool with 3 workers processing 5 jobs.
package main
import (
"fmt"
"time"
)
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
results <- j * 2
}
}
func main() {
const numJobs = 5
jobs := make(chan int, numJobs)
results := make(chan int, numJobs)
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
for j := 1; j <= numJobs; j++ {
jobs <- j
}
close(jobs)
for a := 1; a <= numJobs; a++ {
<-results
}
fmt.Println("All jobs done")
}worker 1 started job 1
worker 2 started job 2
worker 3 started job 3
worker 1 finished job 1
worker 1 started job 4
worker 2 finished job 2
worker 2 started job 5
worker 3 finished job 3
worker 1 finished job 4
worker 2 finished job 5
All jobs doneWorker Pool Components
| Component | Role | Example |
|---|---|---|
| Jobs Channel | Channel to send work to the workers. | jobs := make(chan int, 100) |
| Results Channel | Channel to receive results from the workers. | results := make(chan int, 100) |
| Worker Function | The function that each worker goroutine runs. | func worker(id int, jobs <-chan int, results chan<- int) |
| Starting Workers | Loop to start the desired number of workers. | for w := 1; w <= 3; w++ { go worker(...) } |