const CACHE_EXT = ".gmeasure-cache"
DefaultPrecisionBundle captures the default precisions for Vale and Duration measurements.
var DefaultPrecisionBundle = PrecisionBundle{ Duration: 100 * time.Microsecond, ValueFormat: "%.3f", }
The Annotation decorator allows you to attach an annotation to a given recorded data-point:
For example:
e := gmeasure.NewExperiment("My Experiment")
e.RecordValue("length", 3.141, gmeasure.Annotation("bob"))
e.RecordValue("length", 2.71, gmeasure.Annotation("jane"))
...will result in a Measurement named "length" that records two values )[3.141, 2.71]) annotation with (["bob", "jane"])
type Annotation string
CachedExperimentHeader captures the name of the Cached Experiment and its Version
type CachedExperimentHeader struct {
Name string
Version int
}
Experiment is gmeasure's core data type. You use experiments to record Measurements and generate reports. Experiments are thread-safe and all methods can be called from multiple goroutines.
type Experiment struct {
Name string
// Measurements includes all Measurements recorded by this experiment. You should access them by name via Get() and GetStats()
Measurements Measurements
// contains filtered or unexported fields
}
func NewExperiment(name string) *Experiment
NexExperiment creates a new experiment with the passed-in name.
When using Ginkgo we recommend immediately registering the experiment as a ReportEntry:
experiment = NewExperiment("My Experiment")
AddReportEntry(experiment.Name, experiment)
this will ensure an experiment report is emitted as part of the test output and exported with any test reports.
func (e *Experiment) ColorableString() string
ColorableString returns a Ginkgo formatted summary of the experiment and all its Measurements. It is called automatically by Ginkgo's reporting infrastructure when the Experiment is registered as a ReportEntry via AddReportEntry.
func (e *Experiment) Get(name string) Measurement
Get returns the Measurement with the associated name. If no Measurement is found a zero Measurement{} is returned.
func (e *Experiment) GetStats(name string) Stats
GetStats returns the Stats for the Measurement with the associated name. If no Measurement is found a zero Stats{} is returned.
experiment.GetStats(name) is equivalent to experiment.Get(name).Stats()
func (e *Experiment) MeasureDuration(name string, callback func(), args ...interface{}) time.Duration
MeasureDuration runs the passed-in callback and times how long it takes to complete. The resulting duration is recorded on a Duration Measurement with the passed-in name. If the Measurement does not exist it is created.
MeasureDuration supports the Style(), Precision(), and Annotation() decorations.
func (e *Experiment) MeasureValue(name string, callback func() float64, args ...interface{}) float64
MeasureValue runs the passed-in callback and records the return value on a Value Measurement with the passed-in name. If the Measurement does not exist it is created.
MeasureValue supports the Style(), Units(), Precision(), and Annotation() decorations.
func (e *Experiment) NewStopwatch() *Stopwatch
NewStopwatch() returns a stopwatch configured to record duration measurements with this experiment.
func (e *Experiment) RecordDuration(name string, duration time.Duration, args ...interface{})
RecordDuration records the passed-in duration on a Duration Measurement with the passed-in name. If the Measurement does not exist it is created.
RecordDuration supports the Style(), Precision(), and Annotation() decorations.
func (e *Experiment) RecordNote(note string, args ...interface{})
RecordNote records a Measurement of type MeasurementTypeNote - this is simply a textual note to annotate the experiment. It will be emitted in any experiment reports.
RecordNote supports the Style() decoration.
func (e *Experiment) RecordValue(name string, value float64, args ...interface{})
RecordValue records the passed-in value on a Value Measurement with the passed-in name. If the Measurement does not exist it is created.
RecordValue supports the Style(), Units(), Precision(), and Annotation() decorations.
func (e *Experiment) Sample(callback func(idx int), samplingConfig SamplingConfig)
Sample samples the passed-in callback repeatedly. The sampling is governed by the passed in SamplingConfig.
The SamplingConfig can limit the total number of samples and/or the total time spent sampling the callback. The SamplingConfig can also instruct Sample to run with multiple concurrent workers.
The callback is called with a zero-based index that incerements by one between samples.
func (e *Experiment) SampleAnnotatedDuration(name string, callback func(idx int) Annotation, samplingConfig SamplingConfig, args ...interface{})
SampleDuration samples the passed-in callback and times how long it takes to complete each sample. The resulting durations are recorded on a Duration Measurement with the passed-in name. If the Measurement does not exist it is created.
The callback is given a zero-based index that increments by one between samples. The callback must return an Annotation - this annotation is attached to the measured duration.
SampleAnnotatedDuration supports the Style() and Precision() decorations.
func (e *Experiment) SampleAnnotatedValue(name string, callback func(idx int) (float64, Annotation), samplingConfig SamplingConfig, args ...interface{})
SampleAnnotatedValue samples the passed-in callback and records the return value on a Value Measurement with the passed-in name. If the Measurement does not exist it is created.
The callback is given a zero-based index that increments by one between samples. The callback must return a float64 and an Annotation - the annotation is attached to the recorded value.
SampleValue supports the Style(), Units(), and Precision() decorations.
func (e *Experiment) SampleDuration(name string, callback func(idx int), samplingConfig SamplingConfig, args ...interface{})
SampleDuration samples the passed-in callback and times how long it takes to complete each sample. The resulting durations are recorded on a Duration Measurement with the passed-in name. If the Measurement does not exist it is created.
The callback is given a zero-based index that increments by one between samples. The Sampling is configured via the passed-in SamplingConfig
SampleDuration supports the Style(), Precision(), and Annotation() decorations. When passed an Annotation() the same annotation is applied to all sample measurements.
func (e *Experiment) SampleValue(name string, callback func(idx int) float64, samplingConfig SamplingConfig, args ...interface{})
SampleValue samples the passed-in callback and records the return value on a Value Measurement with the passed-in name. If the Measurement does not exist it is created.
The callback is given a zero-based index that increments by one between samples. The callback must return a float64. The Sampling is configured via the passed-in SamplingConfig
SampleValue supports the Style(), Units(), Precision(), and Annotation() decorations. When passed an Annotation() the same annotation is applied to all sample measurements.
func (e *Experiment) String() string
ColorableString returns an unformatted summary of the experiment and all its Measurements.
ExperimentCache provides a director-and-file based cache of experiments
type ExperimentCache struct {
Path string
}
func NewExperimentCache(path string) (ExperimentCache, error)
NewExperimentCache creates and initializes a new cache. Path must point to a directory (if path does not exist, NewExperimentCache will create a directory at path).
Cached Experiments are stored as separate files in the cache directory - the filename is a hash of the Experiment name. Each file contains two JSON-encoded objects - a CachedExperimentHeader that includes the experiment's name and cache version number, and then the Experiment itself.
func (cache ExperimentCache) Clear() error
Clear empties out the cache - this will delete any and all detected cache files in the cache directory. Use with caution!
func (cache ExperimentCache) Delete(name string) error
Delete removes the experiment with the passed-in name from the cache
func (cache ExperimentCache) List() ([]CachedExperimentHeader, error)
List returns a list of all Cached Experiments found in the cache.
func (cache ExperimentCache) Load(name string, version int) *Experiment
Load fetches an experiment from the cache. Lookup occurs by name. Load requires that the version numer in the cache is equal to or greater than the passed-in version.
If an experiment with corresponding name and version >= the passed-in version is found, it is unmarshaled and returned.
If no experiment is found, or the cached version is smaller than the passed-in version, Load will return nil.
When paired with Ginkgo you can cache experiments and prevent potentially expensive recomputation with this pattern:
const EXPERIMENT_VERSION = 1 //bump this to bust the cache and recompute _all_ experiments
Describe("some experiments", func() {
var cache gmeasure.ExperimentCache
var experiment *gmeasure.Experiment
BeforeEach(func() {
cache = gmeasure.NewExperimentCache("./gmeasure-cache")
name := CurrentSpecReport().LeafNodeText
experiment = cache.Load(name, EXPERIMENT_VERSION)
if experiment != nil {
AddReportEntry(experiment)
Skip("cached")
}
experiment = gmeasure.NewExperiment(name)
AddReportEntry(experiment)
})
It("foo runtime", func() {
experiment.SampleDuration("runtime", func() {
//do stuff
}, gmeasure.SamplingConfig{N:100})
})
It("bar runtime", func() {
experiment.SampleDuration("runtime", func() {
//do stuff
}, gmeasure.SamplingConfig{N:100})
})
AfterEach(func() {
if !CurrentSpecReport().State.Is(types.SpecStateSkipped) {
cache.Save(experiment.Name, EXPERIMENT_VERSION, experiment)
}
})
})
func (cache ExperimentCache) Save(name string, version int, experiment *Experiment) error
Save stores the passed-in experiment to the cache with the passed-in name and version.
Measurement records all captured data for a given measurement. You generally don't make Measurements directly - but you can fetch them from Experiments using Get().
When using Ginkgo, you can register Measurements as Report Entries via AddReportEntry. This will emit all the captured data points when Ginkgo generates the report.
type Measurement struct {
// Type is the MeasurementType - one of MeasurementTypeNote, MeasurementTypeDuration, or MeasurementTypeValue
Type MeasurementType
// ExperimentName is the name of the experiment that this Measurement is associated with
ExperimentName string
// If Type is MeasurementTypeNote, Note is populated with the note text.
Note string
// If Type is MeasurementTypeDuration or MeasurementTypeValue, Name is the name of the recorded measurement
Name string
// Style captures the styling information (if any) for this Measurement
Style string
// Units capture the units (if any) for this Measurement. Units is set to "duration" if the Type is MeasurementTypeDuration
Units string
// PrecisionBundle captures the precision to use when rendering data for this Measurement.
// If Type is MeasurementTypeDuration then PrecisionBundle.Duration is used to round any durations before presentation.
// If Type is MeasurementTypeValue then PrecisionBundle.ValueFormat is used to format any values before presentation
PrecisionBundle PrecisionBundle
// If Type is MeasurementTypeDuration, Durations will contain all durations recorded for this measurement
Durations []time.Duration
// If Type is MeasurementTypeValue, Values will contain all float64s recorded for this measurement
Values []float64
// If Type is MeasurementTypeDuration or MeasurementTypeValue then Annotations will include string annotations for all recorded Durations or Values.
// If the user does not pass-in an Annotation() decoration for a particular value or duration, the corresponding entry in the Annotations slice will be the empty string ""
Annotations []string
}
func (m Measurement) ColorableString() string
ColorableString generates a styled report that includes all the data points for this Measurement. It is called automatically by Ginkgo's reporting infrastructure when the Measurement is registered as a ReportEntry via AddReportEntry.
func (m Measurement) Stats() Stats
Stats returns a Stats struct summarizing the statistic of this measurement
func (m Measurement) String() string
String generates an unstyled report that includes all the data points for this Measurement.
type MeasurementType uint
const (
MeasurementTypeInvalid MeasurementType = iota
MeasurementTypeNote
MeasurementTypeDuration
MeasurementTypeValue
)
func (s MeasurementType) MarshalJSON() ([]byte, error)
func (s MeasurementType) String() string
func (s *MeasurementType) UnmarshalJSON(b []byte) error
type Measurements []Measurement
func (m Measurements) IdxWithName(name string) int
The PrecisionBundle decorator controls the rounding of value and duration measurements. See Precision().
type PrecisionBundle struct {
Duration time.Duration
ValueFormat string
}
func Precision(p interface{}) PrecisionBundle
Precision() allows you to specify the precision of a value or duration measurement - this precision is used when rendering the measurement to screen.
To control the precision of Value measurements, pass Precision an integer. This will denote the number of decimal places to render (equivalen to the format string "%.Nf") To control the precision of Duration measurements, pass Precision a time.Duration. Duration measurements will be rounded oo the nearest time.Duration when rendered.
For example:
e := gmeasure.NewExperiment("My Experiment")
e.RecordValue("length", 3.141, gmeasure.Precision(2))
e.RecordValue("length", 2.71)
e.RecordDuration("cooking time", 3214 * time.Millisecond, gmeasure.Precision(100*time.Millisecond))
e.RecordDuration("cooking time", 2623 * time.Millisecond)
Ranking ranks a set of Stats by a specified RankingCritera. Use RankStats to create a Ranking.
When using Ginkgo, you can register Rankings as Report Entries via AddReportEntry. This will emit a formatted table representing the Stats in rank-order when Ginkgo generates the report.
type Ranking struct {
Criteria RankingCriteria
Stats []Stats
}
func RankStats(criteria RankingCriteria, stats ...Stats) Ranking
RankStats creates a new ranking of the passed-in stats according to the passed-in criteria.
func (c Ranking) ColorableString() string
ColorableString generates a styled report that includes a table of the rank-ordered Stats It is called automatically by Ginkgo's reporting infrastructure when the Ranking is registered as a ReportEntry via AddReportEntry.
func (c Ranking) String() string
String generates an unstyled report that includes a table of the rank-ordered Stats
func (c Ranking) Winner() Stats
Winner returns the Stats with the most optimal rank based on the specified ranking criteria. For example, if the RankingCriteria is LowerMaxIsBetter then the Stats with the lowest value or duration for StatMax will be returned as the "winner"
RankingCriteria is an enum representing the criteria by which Stats should be ranked. The enum names should be self explanatory. e.g. LowerMeanIsBetter means that Stats with lower mean values are considered more beneficial, with the lowest mean being declared the "winner" .
type RankingCriteria uint
const (
LowerMeanIsBetter RankingCriteria = iota
HigherMeanIsBetter
LowerMedianIsBetter
HigherMedianIsBetter
LowerMinIsBetter
HigherMinIsBetter
LowerMaxIsBetter
HigherMaxIsBetter
)
func (s RankingCriteria) MarshalJSON() ([]byte, error)
func (s RankingCriteria) String() string
func (s *RankingCriteria) UnmarshalJSON(b []byte) error
SamplingConfig configures the Sample family of experiment methods. These methods invoke passed-in functions repeatedly to sample and record a given measurement. SamplingConfig is used to control the maximum number of samples or time spent sampling (or both). When both are specified sampling ends as soon as one of the conditions is met. SamplingConfig can also ensure a minimum interval between samples and can enable concurrent sampling.
type SamplingConfig struct {
// N - the maximum number of samples to record
N int
// Duration - the maximum amount of time to spend recording samples
Duration time.Duration
// MinSamplingInterval - the minimum time that must elapse between samplings. It is an error to specify both MinSamplingInterval and NumParallel.
MinSamplingInterval time.Duration
// NumParallel - the number of parallel workers to spin up to record samples. It is an error to specify both MinSamplingInterval and NumParallel.
NumParallel int
}
Stat is an enum representing the statistics you can request of a Stats struct
type Stat uint
const (
StatInvalid Stat = iota
StatMin
StatMax
StatMean
StatMedian
StatStdDev
)
func (s Stat) MarshalJSON() ([]byte, error)
func (s Stat) String() string
func (s *Stat) UnmarshalJSON(b []byte) error
Stats records the key statistics for a given measurement. You generally don't make Stats directly - but you can fetch them from Experiments using GetStats() and from Measurements using Stats().
When using Ginkgo, you can register Measurements as Report Entries via AddReportEntry. This will emit all the captured data points when Ginkgo generates the report.
type Stats struct {
// Type is the StatType - one of StatTypeDuration or StatTypeValue
Type StatsType
// ExperimentName is the name of the Experiment that recorded the Measurement from which this Stat is derived
ExperimentName string
// MeasurementName is the name of the Measurement from which this Stat is derived
MeasurementName string
// Units captures the Units of the Measurement from which this Stat is derived
Units string
// Style captures the Style of the Measurement from which this Stat is derived
Style string
// PrecisionBundle captures the precision to use when rendering data for this Measurement.
// If Type is StatTypeDuration then PrecisionBundle.Duration is used to round any durations before presentation.
// If Type is StatTypeValue then PrecisionBundle.ValueFormat is used to format any values before presentation
PrecisionBundle PrecisionBundle
// N represents the total number of data points in the Meassurement from which this Stat is derived
N int
// If Type is StatTypeValue, ValueBundle will be populated with float64s representing this Stat's statistics
ValueBundle map[Stat]float64
// If Type is StatTypeDuration, DurationBundle will be populated with float64s representing this Stat's statistics
DurationBundle map[Stat]time.Duration
// AnnotationBundle is populated with Annotations corresponding to the data points that can be associated with a Stat.
// For example AnnotationBundle[StatMin] will return the Annotation for the data point that has the minimum value/duration.
AnnotationBundle map[Stat]string
}
func (s Stats) DurationFor(stat Stat) time.Duration
DurationFor returns the time.Duration for a particular Stat. You should only use this if the Stats has Type StatsTypeDuration For example:
mean := experiment.GetStats("runtime").ValueFor(gmeasure.StatMean)
will return the mean duration for the "runtime" Measurement.
func (s Stats) FloatFor(stat Stat) float64
FloatFor returns a float64 representation of the passed-in Stat. When Type is StatsTypeValue this is equivalent to s.ValueFor(stat). When Type is StatsTypeDuration this is equivalent to float64(s.DurationFor(stat))
func (s Stats) String() string
String returns a minimal summary of the stats of the form "MIN < [MEDIAN] | <MEAN> ±STDDEV < MAX"
func (s Stats) StringFor(stat Stat) string
StringFor returns a formatted string representation of the passed-in Stat. The formatting honors the precision directives provided in stats.PrecisionBundle
func (s Stats) ValueFor(stat Stat) float64
ValueFor returns the float64 value for a particular Stat. You should only use this if the Stats has Type StatsTypeValue For example:
median := experiment.GetStats("length").ValueFor(gmeasure.StatMedian)
will return the median data point for the "length" Measurement.
type StatsType uint
const (
StatsTypeInvalid StatsType = iota
StatsTypeValue
StatsTypeDuration
)
func (s StatsType) MarshalJSON() ([]byte, error)
func (s StatsType) String() string
func (s *StatsType) UnmarshalJSON(b []byte) error
Stopwatch provides a convenient abstraction for recording durations. There are two ways to make a Stopwatch:
You can make a Stopwatch from an Experiment via experiment.NewStopwatch(). This is how you first get a hold of a Stopwatch.
You can subsequently call stopwatch.NewStopwatch() to get a fresh Stopwatch. This is only necessary if you need to record durations on a different goroutine as a single Stopwatch is not considered thread-safe.
The Stopwatch starts as soon as it is created. You can Pause() the stopwatch and Reset() it as needed.
Stopwatches refer back to their parent Experiment. They use this reference to record any measured durations back with the Experiment.
type Stopwatch struct {
Experiment *Experiment
// contains filtered or unexported fields
}
func (s *Stopwatch) NewStopwatch() *Stopwatch
NewStopwatch returns a new Stopwatch pointing to the same Experiment as this Stopwatch
func (s *Stopwatch) Pause() *Stopwatch
Pause pauses the Stopwatch. While pasued the Stopwatch does not accumulate elapsed time. This is useful for ignoring expensive operations that are incidental to the behavior you are attempting to characterize. Note: You must call Resume() before you can Record() subsequent measurements.
For example:
stopwatch := experiment.NewStopwatch()
// first expensive operation
stopwatch.Record("first operation").Reset()
// second expensive operation - part 1
stopwatch.Pause()
// something expensive that we don't care about
stopwatch.Resume()
// second expensive operation - part 2
stopwatch.Record("second operation").Reset() // the recorded duration captures the time elapsed during parts 1 and 2 of the second expensive operation, but not the bit in between
The Stopwatch must be running when Pause is called.
func (s *Stopwatch) Record(name string, args ...interface{}) *Stopwatch
Record captures the amount of time that has passed since the Stopwatch was created or most recently Reset(). It records the duration on it's associated Experiment in a Measurement with the passed-in name.
Record takes all the decorators that experiment.RecordDuration takes (e.g. Annotation("...") can be used to annotate this duration)
Note that Record does not Reset the Stopwatch. It does, however, return the Stopwatch so the following pattern is common:
stopwatch := experiment.NewStopwatch()
// first expensive operation
stopwatch.Record("first operation").Reset() //records the duration of the first operation and resets the stopwatch.
// second expensive operation
stopwatch.Record("second operation").Reset() //records the duration of the second operation and resets the stopwatch.
omitting the Reset() after the first operation would cause the duration recorded for the second operation to include the time elapsed by both the first _and_ second operations.
The Stopwatch must be running (i.e. not paused) when Record is called.
func (s *Stopwatch) Reset() *Stopwatch
Reset resets the Stopwatch. Subsequent recorded durations will measure the time elapsed from the moment Reset was called. If the Stopwatch was Paused it is unpaused after calling Reset.
func (s *Stopwatch) Resume() *Stopwatch
Resume resumes a paused Stopwatch. Any time that elapses after Resume is called will be accumulated as elapsed time when a subsequent duration is Recorded.
The Stopwatch must be Paused when Resume is called
The Style decorator allows you to associate a style with a measurement. This is used to generate colorful console reports using Ginkgo V2's console formatter. Styles are strings in curly brackets that correspond to a color or style.
For example:
e := gmeasure.NewExperiment("My Experiment")
e.RecordValue("length", 3.141, gmeasure.Style("{{blue}}{{bold}}"))
e.RecordValue("length", 2.71)
e.RecordDuration("cooking time", 3 * time.Second, gmeasure.Style("{{red}}{{underline}}"))
e.RecordDuration("cooking time", 2 * time.Second)
will emit a report with blue bold entries for the length measurement and red underlined entries for the cooking time measurement.
Units are only set the first time a value or duration of a given name is recorded. In the example above any subsequent calls to e.RecordValue("length", X) will maintain the "{{blue}}{{bold}}" style even if a new Style is passed in later.
type Style string
The Units decorator allows you to specify units (an arbitrary string) when recording values. It is ignored when recording durations.
e := gmeasure.NewExperiment("My Experiment")
e.RecordValue("length", 3.141, gmeasure.Units("inches"))
Units are only set the first time a value of a given name is recorded. In the example above any subsequent calls to e.RecordValue("length", X) will maintain the "inches" units even if a new set of Units("UNIT") are passed in later.
type Units string