hyperfine(1)

NAME

hyperfine - command-line benchmarking tool

SYNOPSIS

hyperfine [-hVNS] [--ignore-failure[=MODE]] [--warmup NUM] [--min-runs NUM] [--max-runs NUM] [--runs NUM] [--setup CMD] [--prepare CMD] [--conclude CMD] [--cleanup CMD] [--parameter-scan VAR MIN MAX] [--parameter-step-size DELTA] [--parameter-list VAR VALUES] [--shell SHELL] [--style TYPE] [--metrics METRIC[:UNIT],...|default|time|all] [--export-asciidoc FILE] [--export-csv FILE] [--export-json FILE] [--export-markdown FILE] [--export-orgmode FILE] [--output WHERE] [--input WHERE] [--command-name NAME] [COMMAND...]

DESCRIPTION

A command-line benchmarking tool which includes:

OPTIONS

-w, --warmup NUM

Perform NUM warmup runs before the actual benchmark. This can be used to fill (disk) caches for I/O-heavy programs.

-m, --min-runs NUM

Perform at least NUM runs for each command. Default: 10.

-M, --max-runs NUM

Perform at most NUM runs for each command. By default, there is no limit.

-r, --runs NUM

Perform exactly NUM runs for each command. If this option is not specified, hyperfine automatically determines the number of runs.

-s, --setup CMD...

Execute CMD once before each set of benchmark runs. This is useful for compiling your software with the provided parameters, or to do any other work that should happen once before a series of benchmark runs, not every time as would happen with the --prepare option.

-p, --prepare CMD...

Execute CMD before each benchmark run. This is useful for clearing disk caches, for example. The --prepare option can be specified once for all commands or multiple times, once for each command. In the latter case, each preparation command will be run prior to the corresponding benchmark command.

-C, --conclude CMD...

Execute CMD after each benchmark run. This is useful for clearing disk caches, for example. The --conclude option can be specified once for all commands or multiple times, once for each command. In the latter case, each conclusion command will be run after the corresponding benchmark command.

-c, --cleanup CMD...

Execute CMD after the completion of all benchmarking runs for each individual command to be benchmarked. This is useful if the commands to be benchmarked produce artifacts that need to be cleaned up. It only runs once per series of benchmark runs, as opposed to the --conclude option which runs after every run.

-P, --parameter-scan VAR MIN MAX

Perform benchmark runs for each value in the range MIN..MAX. Replaces the string '{VAR}' in each command by the current parameter value.

Example:

hyperfine --prepare 'make clean' -P threads 1 8 'make -j {threads}'

This performs benchmarks for 'make -j 1', 'make -j 2', ..., 'make -j 8'.

To have the value increase following different patterns, use shell arithmetic.

Example:

hyperfine --shell=bash -P size 0 3 'sleep $((2**{size}))'

This performs benchmarks with power of 2 increases: 'sleep 1', 'sleep 2', 'sleep 4', ...

The exact syntax may vary depending on your shell and OS.

-D, --parameter-step-size DELTA

This argument requires --parameter-scan to be specified as well. Traverse the range MIN..MAX in steps of DELTA, which must be positive.

Example:

hyperfine -P delay 0.3 0.7 -D 0.2 'sleep {delay}'

This performs benchmarks for 'sleep 0.3', 'sleep 0.5' and 'sleep 0.7'.

-L, --parameter-list VAR VALUES

Perform benchmark runs for each value in the comma-separated list of VALUES. Replaces the string '{VAR}' in each command by the current parameter value.

Example:

hyperfine -L compiler g++,clang++ '{compiler} -O2 main.cpp'

This performs benchmarks for 'g++ -O2 main.cpp' and 'clang++ -O2 main.cpp'.

The option can be specified multiple times to run benchmarks for all possible parameter combinations.

--env ASSIGNMENTS

Set environment variables for benchmark commands and their respective setup, prepare, conclude, and cleanup commands. Assignments are space-separated and support shell-style quoting, but no shell expansion, e.g. --env 'OMP_NUM_THREADS=8 CFLAGS="-O3 -march=native"'. Specify once for all commands or once per benchmark command, like --prepare. Use an empty string for no overrides. Values can contain parameter placeholders.

For example, hyperfine -P threads 1 8 --env 'OMP_NUM_THREADS={threads}' 'my_command' benchmarks 'my_command' with OMP_NUM_THREADS ranging from 1 to 8.

--shell SHELL

Set the shell to use for executing benchmarked commands, including warmup runs (default: none). This can be the name or the path to the shell executable, or a full command line like "bash --norc". It can also be set to "default" to explicitly select the platform shell (sh on Unix, cmd.exe on Windows). Finally, this can also be set to "none" to disable the shell. In this case, benchmarked commands will be executed directly. They can still have arguments, but more complex things like "sleep 0.1; sleep 0.2" are not possible without a shell. Setup, prepare, conclude, and cleanup commands always use a shell: the selected shell, or the platform shell when this option is omitted or set to "none".

-S

An alias for '--shell=default' (sh on Unix, cmd.exe on Windows).

-N

An alias for '--shell=none'.

-i, --ignore-failure[=MODE]

Ignore failures of the benchmarked programs. Without a value or with all-non-zero, all non-zero exit codes are ignored. Alternatively, provide a comma-separated list of exit codes to ignore, for example --ignore-failure=1,2. An explicit value must be attached to the option with an equals sign.

--style TYPE

Set output style TYPE (default: auto). Set this to 'basic' to disable output coloring and interactive elements. Set it to 'full' to enable all effects even if no interactive terminal was detected. Set this to 'nocolor' to keep the interactive output without any colors. Set this to 'color' to keep the colors without any interactive output. Set this to 'none' to disable all the output of the tool.

--metrics METRIC[:UNIT],...|default|time|all

Select the metrics to measure, in order, with an optional unit for each. Default: time_wall_clock,memory_peak_resident, or time_wall_clock on Windows, where peak resident memory is not yet supported.

Use default for the platform defaults, time for time_wall_clock,time_cpu,time_user,time_system, or all for all available metrics in the order listed below.

Time metrics time_wall_clock, time_cpu, time_user, and time_system accept units ns, us, ms, s, min, and h. Peak memory, memory_peak_resident, accepts B, kB, MB, GB, TB, KiB, MiB, GiB, and TiB.

Counter metrics cpu_cycles, instructions, cache_references, cache_misses, and branch_misses accept count (unscaled), k (thousand), M (million), and B (billion).

If a unit is omitted, terminal and markup output automatically select a suitable unit.

Example:

hyperfine --metrics memory_peak_resident:MiB,time_wall_clock:ms,instructions:B './baseline' './candidate'

--export-asciidoc FILE

Export the primary metric's summary statistics as an AsciiDoc table to the given FILE. Use its selected unit, or choose a unit automatically when omitted.

--export-csv FILE

Export summary statistics for all selected metrics as CSV to the given FILE, one row per command. Each metric contributes columns for unit, mean, stddev, median, min, and max, prefixed with the metric name, in selection order. The command column comes first and parameter columns come last. Use the explicitly selected units, or base units (seconds, bytes, or unscaled counts) when omitted.

--export-json FILE

Export summary statistics and individual samples for all collected metrics as JSON to the given FILE. Include the primary metric, and always use base units (seconds, bytes, or unscaled counts), regardless of display settings.

--export-markdown FILE

Export the primary metric's summary statistics as a Markdown table to the given FILE. Use its selected unit, or choose a unit automatically when omitted.

--export-orgmode FILE

Export the primary metric's summary statistics as an Emacs org-mode table to the given FILE. Use its selected unit, or choose a unit automatically when omitted.

--show-output

Print the stdout and stderr of the benchmark instead of suppressing it. This will increase the time it takes for benchmarks to run, so it should only be used for debugging purposes or when trying to benchmark output speed.

--output WHERE

Control where the output of the benchmark is redirected. Note that some programs like 'grep' detect when standard output is /dev/null and apply certain optimizations. To avoid that, consider using --output=pipe.

WHERE can be:

null

Redirect output to /dev/null (the default).

pipe

Feed the output through a pipe before discarding it.

inherit

Don't redirect the output at all (same as '--show-output').

<FILE>

Write the output to the given file.

This option can be specified once for all commands or multiple times, once for each command. Note: If you want to log the output of each and every iteration, you can use a shell redirection and the $HYPERFINE_ITERATION environment variable (requires -S / --shell=default):

--input WHERE

Control where the input of the benchmark comes from.

WHERE can be:

null

Read from /dev/null (the default).

<FILE>

Read the input from the given file.

-n, --command-name NAME

Give a meaningful NAME to a command. This can be specified multiple times if several commands are benchmarked.

-h, --help

Print help

-V, --version

Print version

EXAMPLES

Basic benchmark of 'find . -name todo.txt':

hyperfine 'find . -name todo.txt'

Perform benchmarks for 'sleep 0.2' and 'sleep 3.2' with a minimum 5 runs each:

hyperfine --min-runs 5 'sleep 0.2' 'sleep 3.2'

Perform a benchmark of 'grep' with a warm disk cache by executing 3 runs up front that are not part of the measurement:

hyperfine -S --warmup 3 'grep -R TODO *'

Export the results of a parameter scan benchmark to a markdown table:

hyperfine --export-markdown output.md --parameter-scan time 1 5 'sleep {time}'

Demonstrate when each of --setup, --prepare, --conclude, cmd and --cleanup will run:

hyperfine -L n 1,2 -r 2 --show-output \
	--setup 'echo setup n={n}' \
	--prepare 'echo prepare={n}' \
	--conclude 'echo conclude={n}' \
	--cleanup 'echo cleanup n={n}' \
	'echo command n={n}'

AUTHOR

David Peter <[email protected]>

Source, bug tracker, and additional information can be found on GitHub: https://github.com/sharkdp/hyperfine