README
¶
Go Benchmark Overlay Patches
This directory contains CodSpeed instrumentation overlays for Go's standard testing package benchmarks.
Files
benchmark1.24.0.go,benchmark1.25.0.go- Modified versions of Go'stesting/benchmark.gowith CodSpeed instrumentationbenchmark1.24.0.patch,benchmark1.25.0.patch- Patch files showing differences from upstreamcodspeed.go- CodSpeed-specific benchmark extensionsinstrument-hooks.go- Bindings to the instrument-hooks library
Supporting a new Go Version
To generate the patch files, run this (or use the existing patches):
# Download the unpatched file:
export VERSION=1.25.0
wget -O benchmark$VERSION.go https://github.com/golang/go/raw/refs/tags/go$VERSION/src/testing/benchmark.go
# Then compare against the patched file (same version!):
diff -a -u -N benchmark$VERSION.go overlay/benchmark$VERSION.go > overlay/benchmark$VERSION.patch
You can then download the latest benchmark file and apply the patch:
wget -O overlay/benchmark$VERSION.go https://github.com/golang/go/raw/refs/tags/go$VERSION/src/testing/benchmark.go
patch overlay/benchmark$VERSION.go < overlay/benchmark$VERSION.patch
Then manually fix any conflicts or issues that arise.
Documentation
¶
Index ¶
- Constants
- func CurrentTimestamp() uint64
- func RunBenchmarks(matchString func(pat, str string) (bool, error), ...)
- type B
- func (b *B) AddBenchmarkMarkers(endTimestamp uint64)
- func (b *B) Elapsed() time.Duration
- func (b *B) Loop() bool
- func (b *B) ReportAllocs()
- func (b *B) ReportMetric(n float64, unit string)
- func (b *B) ResetTimer()
- func (b *B) Run(name string, f func(b *B)) bool
- func (b *B) RunParallel(body func(*PB))
- func (b *B) SaveMeasurement()
- func (b *B) SetBytes(n int64)
- func (b *B) SetParallelism(p int)
- func (b *B) StartTimer()
- func (b *B) StartTimerWithoutMarker()
- func (b *B) StopTimer()
- func (b *B) StopTimerWithoutMarker()
- type BenchmarkResult
- type InstrumentHooks
- func (i *InstrumentHooks) AddBenchmarkTimestamps(startTimestamp, endTimestamp uint64)
- func (i *InstrumentHooks) Close()
- func (i *InstrumentHooks) IsInstrumented() bool
- func (i *InstrumentHooks) SetEnvironment(sectionName, key, value string)
- func (i *InstrumentHooks) SetExecutedBenchmark(pid int32, name string)
- func (i *InstrumentHooks) SetIntegration(name, version string)
- func (i *InstrumentHooks) StartBenchmark()
- func (i *InstrumentHooks) StopBenchmark()
- func (i *InstrumentHooks) WriteEnvironment(pid int32)
- type InternalBenchmark
- type PB
Constants ¶
const BenchMaxTimeMult = 3
This is used to determine the maximum time a benchmark is allowed to run, with the overhead of our modifications in the testing package included. When using a benchtime=3s and multiplier of 3, the benchmark can run at most 9s.
Variables ¶
This section is empty.
Functions ¶
func CurrentTimestamp ¶
func CurrentTimestamp() uint64
func RunBenchmarks ¶
RunBenchmarks is an internal function but exported because it is cross-package; it is part of the implementation of the "go test" command.
Types ¶
type B ¶
type B struct {
N int
// contains filtered or unexported fields
}
B is a type passed to Benchmark functions to manage benchmark timing and control the number of iterations.
A benchmark ends when its Benchmark function returns or calls any of the methods [B.FailNow], [B.Fatal], [B.Fatalf], [B.SkipNow], [B.Skip], or [B.Skipf]. Those methods must be called only from the goroutine running the Benchmark function. The other reporting methods, such as the variations of [B.Log] and [B.Error], may be called simultaneously from multiple goroutines.
Like in tests, benchmark logs are accumulated during execution and dumped to standard output when done. Unlike in tests, benchmark logs are always printed, so as not to hide output whose existence may be affecting benchmark results.
func (*B) AddBenchmarkMarkers ¶
func (b *B) AddBenchmarkMarkers(endTimestamp uint64)
func (*B) Elapsed ¶
Elapsed returns the measured elapsed time of the benchmark. The duration reported by Elapsed matches the one measured by B.StartTimer, B.StopTimer, and B.ResetTimer.
func (*B) Loop ¶
func (b *B) Loop() bool
Loop returns true as long as the benchmark should continue running.
A typical benchmark is structured like:
func Benchmark(b *testing.B) {
... setup ...
for b.Loop() {
... code to measure ...
}
... cleanup ...
}
Loop resets the benchmark timer the first time it is called in a benchmark, so any setup performed prior to starting the benchmark loop does not count toward the benchmark measurement. Likewise, when it returns false, it stops the timer so cleanup code is not measured.
The compiler never optimizes away calls to functions within the body of a "for b.Loop() { ... }" loop. This prevents surprises that can otherwise occur if the compiler determines that the result of a benchmarked function is unused. The loop must be written in exactly this form, and this only applies to calls syntactically between the curly braces of the loop. Optimizations are performed as usual in any functions called by the loop.
After Loop returns false, b.N contains the total number of iterations that ran, so the benchmark may use b.N to compute other average metrics.
Prior to the introduction of Loop, benchmarks were expected to contain an explicit loop from 0 to b.N. Benchmarks should either use Loop or contain a loop to b.N, but not both. Loop offers more automatic management of the benchmark timer, and runs each benchmark function only once per measurement, whereas b.N-based benchmarks must run the benchmark function (and any associated setup and cleanup) several times.
func (*B) ReportAllocs ¶
func (b *B) ReportAllocs()
ReportAllocs enables malloc statistics for this benchmark. It is equivalent to setting -test.benchmem, but it only affects the benchmark function that calls ReportAllocs.
func (*B) ReportMetric ¶
ReportMetric adds "n unit" to the reported benchmark results. If the metric is per-iteration, the caller should divide by b.N, and by convention units should end in "/op". ReportMetric overrides any previously reported value for the same unit. ReportMetric panics if unit is the empty string or if unit contains any whitespace. If unit is a unit normally reported by the benchmark framework itself (such as "allocs/op"), ReportMetric will override that metric. Setting "ns/op" to 0 will suppress that built-in metric.
func (*B) ResetTimer ¶
func (b *B) ResetTimer()
ResetTimer zeroes the elapsed benchmark time and memory allocation counters and deletes user-reported metrics. It does not affect whether the timer is running.
func (*B) Run ¶
Run benchmarks f as a subbenchmark with the given name. It reports whether there were any failures.
A subbenchmark is like any other benchmark. A benchmark that calls Run at least once will not be measured itself and will be called once with N=1.
func (*B) RunParallel ¶
func (b *B) RunParallel(body func(*PB))
RunParallel runs a benchmark in parallel. It creates multiple goroutines and distributes b.N iterations among them. The number of goroutines defaults to GOMAXPROCS. To increase parallelism for non-CPU-bound benchmarks, call B.SetParallelism before RunParallel. RunParallel is usually used with the go test -cpu flag.
The body function will be run in each goroutine. It should set up any goroutine-local state and then iterate until pb.Next returns false. It should not use the B.StartTimer, B.StopTimer, or B.ResetTimer functions, because they have global effect. It should also not call B.Run.
RunParallel reports ns/op values as wall time for the benchmark as a whole, not the sum of wall time or CPU time over each parallel goroutine.
func (*B) SaveMeasurement ¶
func (b *B) SaveMeasurement()
func (*B) SetBytes ¶
func (b *B) SetBytes(n int64)
SetBytes records the number of bytes processed in a single operation. If this is called, the benchmark will report ns/op and MB/s.
func (*B) SetParallelism ¶
func (b *B) SetParallelism(p int)
SetParallelism sets the number of goroutines used by B.RunParallel to p*GOMAXPROCS. There is usually no need to call SetParallelism for CPU-bound benchmarks. If p is less than 1, this call will have no effect.
func (*B) StartTimer ¶
func (b *B) StartTimer()
func (*B) StartTimerWithoutMarker ¶
func (b *B) StartTimerWithoutMarker()
StartTimer starts timing a test. This function is called automatically before a benchmark starts, but it can also be used to resume timing after a call to B.StopTimer.
func (*B) StopTimerWithoutMarker ¶
func (b *B) StopTimerWithoutMarker()
StopTimer stops timing a test. This can be used to pause the timer while performing steps that you don't want to measure.
type BenchmarkResult ¶
type BenchmarkResult struct {
N int // The number of iterations.
T time.Duration // The total time taken.
Bytes int64 // Bytes processed in one iteration.
MemAllocs uint64 // The total number of memory allocations.
MemBytes uint64 // The total number of bytes allocated.
CodspeedTimePerRoundNs []time.Duration
CodspeedItersPerRound []int64
// Extra records additional metrics reported by ReportMetric.
Extra map[string]float64
}
BenchmarkResult contains the results of a benchmark run.
func Benchmark ¶
func Benchmark(f func(b *B)) BenchmarkResult
Benchmark benchmarks a single function. It is useful for creating custom benchmarks that do not use the "go test" command.
If f depends on testing flags, then [Init] must be used to register those flags before calling Benchmark and before calling flag.Parse.
If f calls Run, the result will be an estimate of running all its subbenchmarks that don't call Run in sequence in a single benchmark.
func (BenchmarkResult) AllocedBytesPerOp ¶
func (r BenchmarkResult) AllocedBytesPerOp() int64
AllocedBytesPerOp returns the "B/op" metric, which is calculated as r.MemBytes / r.N.
func (BenchmarkResult) AllocsPerOp ¶
func (r BenchmarkResult) AllocsPerOp() int64
AllocsPerOp returns the "allocs/op" metric, which is calculated as r.MemAllocs / r.N.
func (BenchmarkResult) MemString ¶
func (r BenchmarkResult) MemString() string
MemString returns r.AllocedBytesPerOp and r.AllocsPerOp in the same format as 'go test'.
func (BenchmarkResult) NsPerOp ¶
func (r BenchmarkResult) NsPerOp() int64
NsPerOp returns the "ns/op" metric.
func (BenchmarkResult) String ¶
func (r BenchmarkResult) String() string
String returns a summary of the benchmark results. It follows the benchmark result line format from https://golang.org/design/14313-benchmark-format, not including the benchmark name. Extra metrics override built-in metrics of the same name. String does not include allocs/op or B/op, since those are reported by BenchmarkResult.MemString.
type InstrumentHooks ¶
type InstrumentHooks struct {
// contains filtered or unexported fields
}
func NewInstrumentHooks ¶
func NewInstrumentHooks() *InstrumentHooks
func (*InstrumentHooks) AddBenchmarkTimestamps ¶
func (i *InstrumentHooks) AddBenchmarkTimestamps(startTimestamp, endTimestamp uint64)
func (*InstrumentHooks) Close ¶
func (i *InstrumentHooks) Close()
func (*InstrumentHooks) IsInstrumented ¶
func (i *InstrumentHooks) IsInstrumented() bool
func (*InstrumentHooks) SetEnvironment ¶ added in v1.1.0
func (i *InstrumentHooks) SetEnvironment(sectionName, key, value string)
func (*InstrumentHooks) SetExecutedBenchmark ¶
func (i *InstrumentHooks) SetExecutedBenchmark(pid int32, name string)
func (*InstrumentHooks) SetIntegration ¶
func (i *InstrumentHooks) SetIntegration(name, version string)
func (*InstrumentHooks) StartBenchmark ¶
func (i *InstrumentHooks) StartBenchmark()
func (*InstrumentHooks) StopBenchmark ¶
func (i *InstrumentHooks) StopBenchmark()
func (*InstrumentHooks) WriteEnvironment ¶ added in v1.1.0
func (i *InstrumentHooks) WriteEnvironment(pid int32)
type InternalBenchmark ¶
InternalBenchmark is an internal type but exported because it is cross-package; it is part of the implementation of the "go test" command.
Source Files