...
▽

Package throttler

import "github.com/nozzle/throttler"
Overview
Index
Examples

Overview ▾

Package throttler fills the gap between sync.WaitGroup and manually monitoring your goroutines with channels. The API is almost identical to Wait Groups, but it allows you to set a max number of workers that can be running simultaneously. It uses channels internally to block until a job completes by calling Done(err) or until all jobs have been completed.

After exiting the loop where you are using Throttler, you can call the `Err` or `Errs` method to check for errors. `Err` will return a single error representative of all the errors Throttler caught. The `Errs` method will return all the errors as a slice of errors (`[]error`).

Compare the Throttler example to the sync.WaitGroup example http://golang.org/pkg/sync/#example_WaitGroup

See a fully functional example on the playground at http://bit.ly/throttler-v3

type Throttler

Throttler stores all the information about the number of workers, the active workers and error information 

type Throttler struct {
    // contains filtered or unexported fields
}

▾ Example

This example fetches several URLs concurrently, using a Throttler to block until all the fetches are complete. Compare to http://golang.org/pkg/sync/#example_WaitGroup

Code:

var urls = []string{
    "http://www.golang.org/",
    "http://www.google.com/",
    "http://www.somestupidname.com/",
}
// Create a new Throttler that will get 2 urls at a time
t := New(2, len(urls))
for _, url := range urls {
    // Launch a goroutine to fetch the URL.
    go func(url string) {
        // Fetch the URL.
        err := http.Get(url)
        // Let Throttler know when the goroutine completes
        // so it can dispatch another worker
        t.Done(err)
    }(url)
    // Pauses until a worker is available or all jobs have been completed
    // Returning the total number of goroutines that have errored
    // lets you choose to break out of the loop without starting any more
    errorCount := t.Throttle()
    if errorCount > 0 {
        break
    }
}

▾ Example (Errors)

This example fetches several URLs concurrently, using a Throttler to block until all the fetches are complete and checks the errors returned. Compare to http://golang.org/pkg/sync/#example_WaitGroup

Code:

var urls = []string{
    "http://www.golang.org/",
    "http://www.google.com/",
    "http://www.somestupidname.com/",
}
// Create a new Throttler that will get 2 urls at a time
t := New(2, len(urls))
for _, url := range urls {
    // Launch a goroutine to fetch the URL.
    go func(url string) {
        // Let Throttler know when the goroutine completes
        // so it can dispatch another worker
        defer t.Done(nil)
        // Fetch the URL.
        http.Get(url)
    }(url)
    // Pauses until a worker is available or all jobs have been completed
    t.Throttle()
}

if t.Err() != nil {
    // Loop through the errors to see the details
    for i, err := range t.Errs() {
        fmt.Printf("error #%d: %s", i, err)
    }
    return t.Err()
}

return nil

func New 

func New(maxWorkers, totalJobs int) *Throttler

New returns a Throttler that will govern the max number of workers and will work with the total number of jobs. It panics if maxWorkers < 1.

func NewBatchedThrottler 

func NewBatchedThrottler(maxWorkers, batchingTotal, batchSize int) *Throttler

NewBatchedThrottler returns a Throttler (just like New), but also enables batching.

func (*Throttler) BatchEndIndex 

func (t *Throttler) BatchEndIndex() int

BatchEndIndex returns the ending index for the next batch. It either returns the full batch size or the remaining amount of jobs. The job count isn't modified until th.Throttle() is called, so if you don't call Throttle before executing this again, it will return the same index as before.

func (*Throttler) BatchStartIndex 

func (t *Throttler) BatchStartIndex() int

BatchStartIndex returns the starting index for the next batch. The job count isn't modified until th.Throttle() is called, so if you don't call Throttle before executing this again, it will return the same index as before

func (*Throttler) Done 

func (t *Throttler) Done(err error)

Done lets Throttler know that a job has been completed so that another worker can be activated. If Done is called less times than totalJobs, Throttle will block forever

func (*Throttler) Err 

func (t *Throttler) Err() error

Err returns an error representative of all errors caught by throttler

func (*Throttler) Errs 

func (t *Throttler) Errs() []error

Errs returns a slice of any errors that were received from calling Done()

func (*Throttler) SetMaxWorkers 

func (t *Throttler) SetMaxWorkers(maxWorkers int)

SetMaxWorkers lets you change the total number of workers that can run concurrently. NOTE: If all workers are currently running, this setting is not guaranteed to take effect until one of them completes and Throttle() is called again

func (*Throttler) Throttle 

func (t *Throttler) Throttle() int

Throttle works similarly to sync.WaitGroup, except inside your goroutine dispatch loop rather than after. It will not block until the number of active workers matches the max number of workers designated in the call to NewThrottler or all of the jobs have been dispatched. It stops blocking when Done has been called as many times as totalJobs.

func (*Throttler) TotalJobs 

func (t *Throttler) TotalJobs() int

TotalJobs returns the total number of jobs throttler is performing