pythonperformancebenchmarksuite(1)

Installation

Command to install pyperformance:

python3 -m pip install pyperformance

The command installs a new pyperformance program.

If needed, pyperf and six dependencies are installed automatically.

pyperformance works on Python 3.6 and newer, but it may work on Python 3.4 and 3.5.

At runtime, Python development files (header files) may be needed to install some dependencies like dulwich_log or psutil, to build their C extension. Commands on Fedora to install dependencies:

Windows notes

On Windows, to allow pyperformance to build dependencies from source like greenlet, dulwich or psutil, if you want to use a python.exe built from source, you should not use the python.exe directly. Instead, you must run the little-known command PC\layout to create a filesystem layout that resembles an installed Python:

.\python.bat -m PC.layout --preset-default --copy installed -v

(Use the --help flag for more info about PC\layout.)

Now you can use the "installed" Python executable:

installed\python.exe -m pip install pyperformance
installed\python.exe -m pyperformance run ...

Using an actually installed Python executable (e.g. via py) works fine too.

Run benchmarks

Commands to compare Python 3.6 and Python 3.7 performance:

pyperformance run --python=python3.6 -o py36.json
pyperformance run --python=python3.7 -o py37.json
pyperformance compare py36.json py37.json

Note: python3 -m pyperformance ... syntax works as well (ex: python3 -m pyperformance run -o py37.json), but requires to install pyperformance on each tested Python version.

JSON files are produced by the pyperf module and so can be analyzed using pyperf commands:

python3 -m pyperf show py36.json
python3 -m pyperf check py36.json
python3 -m pyperf metadata py36.json
python3 -m pyperf stats py36.json
python3 -m pyperf hist py36.json
python3 -m pyperf dump py36.json
(...)

It's also possible to use pyperf to compare results of two JSON files:

python3 -m pyperf compare_to py36.json py37.json --table

Basic commands

pyperformance actions:

run                 Run benchmarks on the running python
show                Display a benchmark file
compare             Compare two benchmark files
list                List benchmarks of the running Python
list_groups         List benchmark groups of the running Python
venv                Actions on the virtual environment

Common options

Options available to all commands:

-h, --help            show this help message and exit

run

Run benchmarks on the running python.

Usage:

pyperformance run [-h] [-r] [-f] [--debug-single-value] [-v] [-m]
                     [--affinity CPU_LIST] [-o FILENAME]
                     [--append FILENAME] [--manifest MANIFEST]
                     [--timeout TIMEOUT] [-b BM_LIST]
                     [--inherit-environ VAR_LIST] [-p PYTHON]
                     [--hook HOOK]

options:

-h, --help            show this help message and exit
-r, --rigorous        Spend longer running tests to get more
                      accurate results
-f, --fast            Get rough answers quickly
--debug-single-value  Debug: fastest mode, only compute a single
                      value
-v, --verbose         Print more output
-m, --track-memory    Track memory usage. This only works on Linux.
--affinity CPU_LIST   Specify CPU affinity for benchmark runs. This
                      way, benchmarks can be forced to run on a
                      given CPU to minimize run to run variation.
-o FILENAME, --output FILENAME
                      Run the benchmarks on only one interpreter and
                      write benchmark into FILENAME. Provide only
                      baseline_python, not changed_python.
--append FILENAME     Add runs to an existing file, or create it if
                      it doesn't exist
--timeout TIMEOUT     Specify a timeout in seconds for a single
                      benchmark run (default: disabled)
--manifest MANIFEST   benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST
                      Comma-separated list of benchmarks to run. Can
                      contain both positive and negative arguments:
                      --benchmarks=run_this,also_this,-not_this. If
                      there are no positive arguments, we'll run all
                      benchmarks except the negative arguments.
                      Otherwise we run only the positive arguments.
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)
--same-loops SAME_LOOPS
                      Use the same number of loops as a previous run
                      (i.e., don't recalibrate). Should be a path to a
                      .json file from a previous run.
--hook HOOK
                      Apply the given pyperf hook when running the
                      benchmarks.

show

Display a benchmark file.

Usage:

show FILENAME

positional arguments:

FILENAME

compare

Compare two benchmark files.

Usage:

pyperformance compare [-h] [-v] [-O STYLE] [--csv CSV_FILE]
                      [--inherit-environ VAR_LIST] [-p PYTHON]
                      baseline_file.json changed_file.json

positional arguments:

baseline_file.json
changed_file.json

options:

-v, --verbose         Print more output
-O STYLE, --output_style STYLE
                      What style the benchmark output should take.
                      Valid options are 'normal' and 'table'.
                      Default is normal.
--csv CSV_FILE        Name of a file the results will be written to,
                      as a three-column CSV file containing minimum
                      runtimes for each benchmark.
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

list

List benchmarks of the running Python.

Usage:

pyperformance list [-h] [--manifest MANIFEST] [-b BM_LIST]
                   [--inherit-environ VAR_LIST] [-p PYTHON]

options:

--manifest MANIFEST   benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST
                      Comma-separated list of benchmarks to run. Can
                      contain both positive and negative arguments:
                      --benchmarks=run_this,also_this,-not_this. If
                      there are no positive arguments, we'll run all
                      benchmarks except the negative arguments.
                      Otherwise we run only the positive arguments.
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

Use python3 -m pyperformance list -b all to list all benchmarks.

list_groups

List benchmark groups of the running Python.

Usage:

pyperformance list_groups [-h] [--manifest MANIFEST]
                          [--inherit-environ VAR_LIST]
                          [-p PYTHON]

options:

--manifest MANIFEST   benchmark manifest file to use
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

venv

Actions on the virtual environment.

Actions:

show      Display the path to the virtual environment and its status
          (created or not)
create    Create the virtual environment
recreate  Force the recreation of the the virtual environment
remove    Remove the virtual environment

Common options:

--venv VENV           Path to the virtual environment
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

venv show

Display the path to the virtual environment and its status (created or not).

Usage:

pyperformance venv show [-h] [--venv VENV]
                        [--inherit-environ VAR_LIST] [-p PYTHON]

venv create

Create the virtual environment.

Usage:

pyperformance venv create [-h] [--venv VENV]
                          [--manifest MANIFEST] [-b BM_LIST]
                          [--inherit-environ VAR_LIST]
                          [-p PYTHON]

options:

--manifest MANIFEST   benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST
                      Comma-separated list of benchmarks to run. Can
                      contain both positive and negative arguments:
                      --benchmarks=run_this,also_this,-not_this. If
                      there are no positive arguments, we'll run all
                      benchmarks except the negative arguments.
                      Otherwise we run only the positive arguments.

venv recreate

Force the recreation of the the virtual environment.

Usage:

pyperformance venv recreate [-h] [--venv VENV]
                            [--manifest MANIFEST] [-b BM_LIST]
                            [--inherit-environ VAR_LIST]
                            [-p PYTHON]

options:

--manifest MANIFEST   benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST
                      Comma-separated list of benchmarks to run. Can
                      contain both positive and negative arguments:
                      --benchmarks=run_this,also_this,-not_this. If
                      there are no positive arguments, we'll run all
                      benchmarks except the negative arguments.
                      Otherwise we run only the positive arguments.

venv remove

Remove the virtual environment.

Usage:

pyperformance venv remove [-h] [--venv VENV]
                          [--inherit-environ VAR_LIST]
                          [-p PYTHON]

Compile Python to run benchmarks

pyperformance actions:

compile             Compile and install CPython and run benchmarks on
                    installed Python
compile_all         Compile and install CPython and run benchmarks on
                    installed Python on all branches and revisions of
                    CONFIG_FILE
upload              Upload JSON results to a Codespeed website

All these commands require a configuration file.

Simple configuration usable for compile (but not for compile_all nor upload), doc/benchmark.conf:

[config]
json_dir = ~/prog/python/bench_json
[scm]
repo_dir = ~/prog/python/master
update = True
[compile]
bench_dir = ~/prog/python/bench_tmpdir
[run_benchmark]
system_tune = True
affinity = 2,3

Configuration file sample with comments, doc/benchmark.conf.sample:

[config]
# Directory where JSON files are written.
# - uploaded files are moved to json_dir/uploaded/
# - results of patched Python are written into json_dir/patch/
json_dir = ~/json
# If True, compile CPython in debug mode (LTO and PGO disabled),
# run benchmarks with --debug-single-sample, and disable upload.
#
# Use this option to quickly test a configuration.
debug = False
[scm]
# Directory of CPython source code (Git repository)
repo_dir = ~/cpython
# Update the Git repository (git fetch)?
update = True
# Name of the Git remote, used to create revision of
# the Git branch. For example, use revision 'remotes/origin/3.6'
# for the branch '3.6'.
git_remote = remotes/origin
[compile]
# Create files into bench_dir:
# - bench_dir/bench-xxx.log
# - bench_dir/prefix/: where Python is installed
# - bench_dir/venv/: Virtual environment used by pyperformance
bench_dir = ~/bench_tmpdir
# Link Time Optimization (LTO)?
lto = True
# Profiled Guided Optimization (PGO)?
pgo = True
# Build the experimental just-in-time (JIT) compiler?
# Possible values are:
# - no: (default) do not build the JIT or the micro-op interpreter.
#   The new PYTHON_JIT environment variable has no effect.
# - yes: build the JIT and enable it by default. PYTHON_JIT=0 can be used to
#   disable it at runtime.
# - yes-off: build the JIT, but do not enable it by default. PYTHON_JIT=1 can
#   be used to enable it at runtime.
# - interpreter: do not build the JIT, but do build and enable the micro-op
#   interpreter. This is useful for those of us who find ourselves developing
#   or debugging micro-ops (but don’t want to deal with the JIT).
#   PYTHON_JIT=0 can be used to disable the micro-op interpreter at runtime.
jit = no
# The space-separated list of libraries that are package-only,
# i.e., locally installed but not on header and library paths.
# For each such library, determine the install path and add an
# appropriate subpath to CFLAGS and LDFLAGS declarations passed
# to configure. As an exception, the prefix for openssl, if that
# library is present here, is passed via the --with-openssl
# option. Currently, this only works with Homebrew on macOS.
# If running on macOS with Homebrew, you probably want to use:
#     pkg_only = openssl readline sqlite3 xz zlib
# The version of zlib shipping with macOS probably works as well,
# as long as Apple's SDK headers are installed.
pkg_only =
# Install Python? If false, run Python from the build directory
#
# WARNING: Running Python from the build directory introduces subtle changes
# compared to running an installed Python. Moreover, creating a virtual
# environment using a Python run from the build directory fails in many cases,
# especially on Python older than 3.4. Only disable installation if you
# really understand what you are doing!
install = True
# Specify '-j' parameter in 'make' command
jobs = 8
[run_benchmark]
# Run "sudo python3 -m pyperf system tune" before running benchmarks?
system_tune = True
# --manifest option for 'pyperformance run'
manifest =
# --benchmarks option for 'pyperformance run'
benchmarks =
# --affinity option for 'pyperf system tune' and 'pyperformance run'
affinity =
# Upload generated JSON file?
#
# Upload is disabled on patched Python, in debug mode or if install is
# disabled.
upload = False
# --rigorous option for 'pyperformance run'
rigorous = False
# Configuration to upload results to a Codespeed website
[upload]
url =
environment =
executable =
project =
[compile_all]
# List of CPython Git branches
branches = default 3.6 3.5 2.7
# List of revisions to benchmark by compile_all
[compile_all_revisions]
# list of 'sha1=' (default branch: 'master') or 'sha1=branch'
# used by the "pyperformance compile_all" command
# e.g.:
11159d2c9d6616497ef4cc62953a5c3cc8454afb =

compile

Compile Python, install Python and run benchmarks on the installed Python.

Usage:

pyperformance compile [-h] [--patch PATCH] [-U] [-T]
                      [--inherit-environ VAR_LIST] [-p PYTHON]
                      config_file revision [branch]

positional arguments:

config_file           Configuration filename
revision              Python benchmarked revision
branch                Git branch

options:

--patch PATCH         Patch file
-U, --no-update       Don't update the Git repository
-T, --no-tune         Don't run 'pyperf system tune' to tune the
                      system for benchmarks
--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

Notes:

compile_all

Compile all branches and revisions of CONFIG_FILE.

Usage:

pyperformance compile_all [-h] [--inherit-environ VAR_LIST] [-p PYTHON]
                          config_file

positional arguments:

config_file           Configuration filename

options:

--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

upload

Upload results from a JSON file to a Codespeed website.

Usage:

pyperformance upload [-h] [--inherit-environ VAR_LIST] [-p PYTHON]
                     config_file json_file

positional arguments:

config_file           Configuration filename
json_file             JSON filename

options:

--inherit-environ VAR_LIST
                      Comma-separated list of environment variable
                      names that are inherited from the parent
                      environment when running benchmarking
                      subprocesses.
-p PYTHON, --python PYTHON
                      Python executable (default: use running
                      Python)

How to get stable benchmarks

pyperformance virtual environment

To run benchmarks, pyperformance first creates a virtual environment. It installs requirements with fixed versions to get a reproductible environment. The system Python has unknown module installed with unknown versions, and can have .pth files run at Python startup which can modify Python behaviour or at least slow down Python startup.

What is the goal of pyperformance

A benchmark is always written for a specific purpose. Depending how the benchmark is written and how the benchmark is run, the result can be different and so have a different meaning.

The pyperformance benchmark suite has multiple goals:

Don't disable GC nor ASLR

The pyperf module and pyperformance benchmarks are designed to produce reproductible results, but not at the price of running benchmarks in a special mode which would not be used to run applications in production. For these reasons, the Python garbage collector, Python randomized hash function and system ASLR (Address Space Layout Randomization) are not disabled. Benchmarks don't call gc.collect() neither since CPython implements it with stop-the-world <https://en.wikipedia.org/wiki/Tracing_garbage_collection#Stop-the-world_vs._incremental_vs._concurrent> and so applications don't call it to not kill performances.

Include outliers and spikes

Moreover, while the pyperf documentation explains how to reduce the random noise of the system and other applications, some benchmarks use the system and so can get different timing depending on the system workload, depending on I/O performances, etc. Outliers and temporary spikes in results are not automatically removed: values are summarized by computing the average (arithmetic mean) and standard deviation which "contains" these spikes, instead of using median and the median absolute deviation for example which to ignore outliers. It is deliberate choice since applications running in production are impacted by such temporary slowdown caused by various things like a garbage collection or a JIT compilation.

Warmups and steady state

A borderline issue are the benchmarks "warmups". The first values of each worker process are always slower: 10% slower in the best case, it can be 1000% slower or more on PyPy. Right now (2017-04-14), pyperformance ignore first values considered as warmup until a benchmark reachs its "steady state". The "steady state" can include temporary spikes every 5 values (ex: caused by the garbage collector), and it can still imply further JIT compiler optimizations but with a "low" impact on the average pyperformance.

To be clear "warmup" and "steady state" are a work-in-progress and a very complex topic, especially on PyPy and its JIT compiler.

Notes

Tool for comparing the performance of two Python implementations.

pyperformance will run Student's two-tailed T test on the benchmark results at the 95% confidence level to indicate whether the observed difference is statistically significant.

Omitting the -b option will result in the default group of benchmarks being run Omitting -b is the same as specifying -b default.

To run every benchmark pyperformance knows about, use -b all. To see a full list of all available benchmarks, use --help.

Negative benchmarks specifications are also supported: -b -2to3 will run every benchmark in the default group except for 2to3 (this is the same as -b default,-2to3). -b all,-django will run all benchmarks except the Django templates benchmark. Negative groups (e.g., -b -default) are not supported. Positive benchmarks are parsed before the negative benchmarks are subtracted.

If --track_memory is passed, pyperformance will continuously sample the benchmark's memory usage. This currently only works on Linux 2.6.16 and higher or Windows with PyWin32. Because --track_memory introduces performance jitter while collecting memory measurements, only memory usage is reported in the final report.

Available Groups

Like individual benchmarks (see "Available benchmarks" below), benchmarks group are allowed after the -b option. Use python3 -m pyperformance list_groups to list groups and their benchmarks.

Available benchmark groups:

Use the python3 -m pyperformance list_groups command to list groups and their benchmarks.

Available Benchmarks

In pyperformance 0.5.5, the following microbenchmarks have been removed because they are too short, not representative of real applications and are too unstable.

2to3

Run the 2to3 tool on the pyperformance/benchmarks/data/2to3/ directory: copy of the django/core/*.py files of Django 1.1.4, 9 files.

Run the python -m lib2to3 -f all <files> command where python is sys.executable. So the test does not only mesure the performance of Python itself, but also the performance of the lib2to3 module which can change depending on the Python version.

Note:

async_tree

Async workload benchmark, which calls asyncio.gather() on a tree (6 levels deep, 6 branches per level) with the leaf nodes simulating some [potentially] async work (depending on the benchmark variant). Available variants:

These benchmarks also have an "eager" flavor that uses asyncio eager task factory, if available.

base64

Benchmark the base64 module's encoding and decoding functions. Each algorithm has _small and _large variants that test both encode and decode in a single benchmark:

Available benchmarks:

See the base64 module <https://docs.python.org/dev/library/base64.html>.

chameleon

Render a template using the chameleon module to create an HTML table of 500 lignes and 10 columns.

See the chameleon.PageTemplate class.

chaos

Create chaosgame-like fractals. Command lines options:

--thickness THICKNESS
                      Thickness (default: 0.25)
--width WIDTH         Image width (default: 256)
--height HEIGHT       Image height (default: 256)
--iterations ITERATIONS
                      Number of iterations (default: 5000)
--filename FILENAME.PPM
                      Output filename of the PPM picture
--rng-seed RNG_SEED   Random number generator seed (default: 1234)

When --filename option is used, the timing includes the time to create the PPM file.

Copyright (C) 2005 Carl Friedrich Bolz [image: Chaos game, bm_chaos benchmark] [image]

Image generated by bm_chaos (took 3 sec on CPython 3.5) with the command:

python3 pyperformance/benchmarks/bm_chaos.py --worker -l1 -w0 -n1 --filename chaos.ppm --width=512 --height=512 --iterations 50000

crypto_pyaes

benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the pyaes module.

The benchmark is slower on CPython 3 compared to CPython 2.7, because CPython 3 has no more "small int" type (int). The CPython 3 int type now always has an arbitrary size, as CPython 2.7 long type.

See pyaes <https://github.com/ricmoo/pyaes>: A pure-Python implementation of the AES block cipher algorithm and the common modes of operation (CBC, CFB, CTR, ECB and OFB).

deepcopy

Benchmark the Python copy.deepcopy method. The deepcopy method is performed on a nested dictionary and a dataclass.

deltablue

DeltaBlue benchmark

Ported for the PyPy project. Contributed by Daniel Lindsley

This implementation of the DeltaBlue benchmark was directly ported from the V8's source code <https://github.com/v8/v8/blob/master/benchmarks/deltablue.js>, which was in turn derived from the Smalltalk implementation by John Maloney and Mario Wolczko. The original Javascript implementation was licensed under the GPL.

It's been updated in places to be more idiomatic to Python (for loops over collections, a couple magic methods, OrderedCollection being a list & things altering those collections changed to the builtin methods) but largely retains the layout & logic from the original. (Ugh.)

django_template

Use the Django template system to build a 150x150-cell HTML table.

Use Context and Template classes of the django.template module.

dulwich_log

Iterate on commits of the asyncio Git repository using the Dulwich module. Use pyperformance/benchmarks/data/asyncio.git/ repository.

Pseudo-code of the benchmark:

repo = dulwich.repo.Repo(repo_path)
head = repo.head()
for entry in repo.get_walker(head):
    pass

See the Dulwich project <https://www.dulwich.io/>.

docutils

Use Docutils <https://docutils.sourceforge.io/> to convert Docutils' documentation to HTML. Representative of building a medium-sized documentation set.

fannkuch

The Computer Language Benchmarks Game: <http://benchmarksgame.alioth.debian.org/>

Contributed by Sokolov Yura, modified by Tupteq.

float

Artificial, floating point-heavy benchmark originally used by Factor.

Create 100,000 point objects which compute math.cos(), math.sin() and math.sqrt()

Changed in version 0.5.5: Use __slots__ on the Point class to focus the benchmark on float rather than testing performance of class attributes.

genshi

Render a template using Genshi (genshi.template module):

See the Genshi project <http://pythonhosted.org/Genshi/>.

go

Artificial intelligence playing the Go board game. Use Zobrist hashing <https://en.wikipedia.org/wiki/Zobrist_hashing>.

hexiom

Solver of Hexiom board game (level 25 by default). Command line option:

--level {2,10,20,25,30,36}   Hexiom board level (default: 25)

hg_startup

Get Mercurial's help screen.

Measure the performance of the python path/to/hg help command using pyperf.Runner.bench_command(), where python is sys.executable and path/to/hg is the Mercurial program installed in a virtual environmnent.

The bench_command() redirects stdout and stderr into /dev/null.

See the Mercurial project <https://www.mercurial-scm.org/>.

html5lib

Parse the pyperformance/benchmarks/data/w3_tr_html5.html HTML file (132 KB) using html5lib. The file is the HTML 5 specification, but truncated to parse the file in less than 1 second (around 250 ms).

On CPython, after 3 warmups, the benchmarks enters a cycle of 5 values: every 5th value is 10% slower. Plot of 1 run of 50 values (the warmup is not rendered): [image: html5lib values] [image]

See the html5lib project <https://html5lib.readthedocs.io/>.

json_dumps, json_loads

Benchmark dumps() and loads() functions of the json module.

bm_json_dumps.py command line option:

--cases CASES         Comma separated list of cases. Available cases: EMPTY,
                      SIMPLE, NESTED, HUGE. By default, run all cases.

logging

Benchmarks on the logging module:

Script command line option:

format
silent
simple

See the logging module <https://docs.python.org/dev/library/logging.html>.

mako

Use the Mako template system to build a 150x150-cell HTML table. Includes:

See the Mako project <http://docs.makotemplates.org/>.

mdp

Battle with damages and topological sorting of nodes in a graph.

See Topological sorting <https://en.wikipedia.org/wiki/Topological_sorting>.

meteor_contest

Solver for Meteor Puzzle board.

Meteor Puzzle board: <http://benchmarksgame.alioth.debian.org/u32/meteor-description.html#meteor>

The Computer Language Benchmarks Game: <http://benchmarksgame.alioth.debian.org/>

Contributed by Daniel Nanz, 2008-08-21.

nbody

N-body benchmark from the Computer Language Benchmarks Game. Microbenchmark on floating point operations.

This is intended to support Unladen Swallow's perf.py. Accordingly, it has been modified from the Shootout version:

Pulled from: <http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=python3&id=1>

Contributed by Kevin Carson. Modified by Tupteq, Fredrik Johansson, and Daniel Nanz.

python_startup, python_startup_nosite

Run the benchmark with pyperf.Runner.bench_command().

nqueens

Simple, brute-force N-Queens solver.

See Eight queens puzzle <https://en.wikipedia.org/wiki/Eight_queens_puzzle>.

pathlib

Test the performance of operations of the pathlib module of the standard library.

This benchmark stresses the creation of small objects, globbing, and system calls.

See the documentation of the pathlib module <https://docs.python.org/dev/library/pathlib.html>.

pickle

pickle benchmarks (serialize):

unpickle benchmarks (deserialize):

pidigits

Calculating 2,000 digits of π. This benchmark stresses big integer arithmetic.

Command line option:

--digits DIGITS     Number of computed pi digits (default: 2000)

Adapted from code on: <http://benchmarksgame.alioth.debian.org/>

pyflate

Benchmark of a pure-Python bzip2 decompressor: decompress the pyperformance/benchmarks/data/interpreter.tar.bz2 file in memory.

Copyright 2006--2007-01-21 Paul Sladen: <http://www.paul.sladen.org/projects/compression/>

You may use and distribute this code under any DFSG-compatible license (eg. BSD, GNU GPLv2).

Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. This is probably most useful for research purposes/index building; there is certainly some room for improvement in the Huffman bit-matcher.

With the as-written implementation, there was a known bug in BWT decoding to do with repeated strings. This has been worked around; see 'bwt_reverse()'. Correct output is produced in all test cases but ideally the problem would be found...

raytrace

Simple raytracer.

Command line options:

--width WIDTH             Image width (default: 100)
--height HEIGHT           Image height (default: 100)
--filename FILENAME.PPM   Output filename of the PPM picture

This file contains definitions for a simple raytracer. Copyright Callum and Tony Garnock-Jones, 2008.

This file may be freely redistributed under the MIT license, <http://www.opensource.org/licenses/mit-license.php>

From <https://leastfixedpoint.com/tonyg/kcbbs/lshift_archive/toy-raytracer-in-python-20081029.html> [image: Pure Python raytracer] [image]

Image generated by the command (took 68.4 sec on CPython 3.5):

python3 pyperformance/benchmarks/bm_raytrace.py --worker --filename=raytrace.ppm  -l1 -w0 -n1 -v --width=800 --height=600

regex_compile

Stress the performance of Python's regex compiler, rather than the regex execution speed.

Benchmark how quickly Python's regex implementation can compile regexes.

We bring in all the regexes used by the other regex benchmarks, capture them by stubbing out the re module, then compile those regexes repeatedly. We muck with the re module's caching to force it to recompile every regex we give it.

regex_dna

regex DNA benchmark using "fasta" to generate the test case.

The Computer Language Benchmarks Game <http://benchmarksgame.alioth.debian.org/>

regex-dna Python 3 #5 program: contributed by Dominique Wahli 2to3 modified by Justin Peel

fasta Python 3 #3 program: modified by Ian Osgood modified again by Heinrich Acker modified by Justin Peel Modified by Christopher Sean Forgeron

regex_effbot

Some of the original benchmarks used to tune mainline Python's current regex engine.

regex_v8

Python port of V8's regex benchmark.

Automatically generated on 2009-01-30.

This benchmark is generated by loading 50 of the most popular pages on the web and logging all regexp operations performed. Each operation is given a weight that is calculated from an estimate of the popularity of the pages where it occurs and the number of times it is executed while loading each page. Finally the literal letters in the data are encoded using ROT13 in a way that does not affect how the regexps match their input.

Ported to Python for Unladen Swallow. The original JS version can be found at <https://github.com/v8/v8/blob/master/benchmarks/regexp.js>, r1243.

richards

The classic Python Richards benchmark.

Based on a Java version.

Based on original version written in BCPL by Dr Martin Richards in 1981 at Cambridge University Computer Laboratory, England and a C++ version derived from a Smalltalk version written by L Peter Deutsch.

Java version: Copyright (C) 1995 Sun Microsystems, Inc. Translation from C++, Mario Wolczko Outer loop added by Alex Jacoby

scimark

spectral_norm

MathWorld: "Hundred-Dollar, Hundred-Digit Challenge Problems", Challenge #3. <http://mathworld.wolfram.com/Hundred-DollarHundred-DigitChallengeProblems.html>

The Computer Language Benchmarks Game <http://benchmarksgame.alioth.debian.org/u64q/spectralnorm-description.html#spectralnorm>

Contributed by Sebastien Loisel. Fixed by Isaac Gouy. Sped up by Josh Goldfoot. Dirtily sped up by Simon Descarpentries. Concurrency by Jason Stitt.

sqlalchemy_declarative, sqlalchemy_imperative

See the SQLAlchemy project <https://www.sqlalchemy.org/>.

sqlite_synth

Benchmark Python aggregate for SQLite.

The goal of the benchmark (written for PyPy) is to test CFFI performance and going back and forth between SQLite and Python a lot. Therefore the queries themselves are really simple.

See the SQLite project <https://www.sqlite.org/> and the Python sqlite3 module (stdlib) <https://docs.python.org/dev/library/sqlite3.html>.

sympy

Benchmark on the sympy module:

On CPython, some sympy_sum values are 5%-10% slower:

$ python3 -m pyperf dump sympy_sum.json
Run 1: 1 warmup, 50 values, 1 loop
- warmup 1: 404 ms (+63%)
- value 1: 244 ms
- value 2: 245 ms
- value 3: 258 ms <----
- value 4: 245 ms
- value 5: 245 ms
- value 6: 279 ms (+12%) <----
- value 7: 246 ms
- value 8: 244 ms
- value 9: 245 ms
- value 10: 255 ms <----
- value 11: 245 ms
- value 12: 245 ms
- value 13: 256 ms <----
- value 14: 248 ms
- value 15: 245 ms
- value 16: 245 ms
...

Plot of 1 run of 50 values (the warmup is not rendered): [image: sympy_sum values] [image]

See the sympy project <http://www.sympy.org/>.

telco

Telco Benchmark for measuring the performance of decimal calculations:

The Python benchmark is implemented with the decimal module.

See the Python decimal module (stdlib) <https://docs.python.org/dev/library/decimal.html>.

tornado_http

Benchmark HTTP server of the tornado module

See the Tornado project <http://www.tornadoweb.org/>.

unpack_sequence

Microbenchmark for unpacking lists and tuples.

Pseudo-code:

a, b, c, d, e, f, g, h, i, j = to_unpack

where to_unpack is tuple(range(10)) or list(range(10)).

xml_etree

Benchmark the ElementTree API of the xml.etree module:

See the Python xml.etree.ElementTree module (stdlib) <https://docs.python.org/dev/library/xml.etree.elementtree.html>.

Using Custom Benchmarks

To use custom benchmarks, you will need to use the --manifest CLI option and provide the path to the manifest file describing those benchmarks.

The pyperformance File Formats

pyperformance uses two file formats to identify benchmarks:

For each benchmark, there are two required files and several optional ones. Those files are expected to be in a specific directory structure (unless customized in the metadata).

The structure (see below) is such that it's easy to maintain a benchmark (or set of benchmarks) on GitHub and distribute it on PyPI. It also simplifies publishing a Python project's benchmarks. The alternative is pointing people at a repo.

Benchmarks can inherit metadata from other metadata files. This is useful for keeping common metadata for a set of benchmarks (e.g. "version") in one file. Likewise, benchmarks for a Python project can inherit metadata from the project's pyproject.toml.

Sometimes a benchmark will have one or more variants that run using the same script. Variants like this are supported by pyperformance without requiring much extra effort.

Benchmark Directory Structure

Normally a benchmark is structured like this:

bm_NAME/
   data/  # if needed
   requirements.txt  # lock file, if any
   pyproject.toml
   run_benchmark.py

(Note the "bm_" prefix on the directory name.)

"pyproject.toml" holds the metadata. "run_benchmark.py" holds the actual benchmark code. Both are necessary.

pyperformance treats the metadata file as the fundamental source of information about a benchmark. A manifest for a set of benchmarks is effectively a mapping of names to metadata files. So a metadata file is essential. It can be located anywhere on disk. However, if it isn't located in the structure described above then the metadata must identify where to find the other files.

Other than that, only a benchmark script (e.g. "run_benchmark.py" above) is required. All other files are optional.

When a benchmark has variants, each has its own metadata file next to the normal "pyproject.toml", named "bm_NAME.toml". (Note the "bm_" prefix.) The format of variant metadata files is exactly the same. pyperformance treats them the same, except that the sibling "pyproject.toml" is inherited by default.

Manifest Files

A manifest file identifies a set of benchmarks, as well as (optionally) how they should be grouped. pyperformance uses the manifest to determine which benchmarks are available to run (and thus which to run by default).

A manifest normally looks like this:

[benchmarks]
name        metafile
bench1      somedir/bm_bench1/pyproject.toml
bench2      somedir/pyproject.toml
bench3      ../anotherdir

The "benchmarks" section is a table with rows of tab-separated-values. The "name" value is how pyperformance will identify the benchmark. The "metafile" value is where pyperformance will look for the benchmark's metadata. If a metafile is a directory then it looks for "pyproject.toml" in that directory.

Benchmark Groups

The other sections in the manifest file relate to grouping:

[benchmarks]
name        metafile
bench1      somedir/bm_bench1
bench2      somedir/bm_bench2
bench3      anotherdir/mybench.toml
[groups]
tag1
tag2
[group default]
bench2
bench3
[group tricky]
bench2

The "groups" section specifies available groups that may be identified by benchmark tags (see about tags in the metadata section below). Any other group sections in the manifest are automatically added to the list of available groups.

If no "default" group is specified then one is automatically added with all benchmarks from the "benchmarks" section in it. If there is no "groups" section and no individual group sections (other than "default") then the set of all tags of the known benchmarks is treated as "groups". A group named "all" as also automatically added which has all known benchmarks in it.

Benchmarks can be excluded from a group by using a - (minus) prefix. Any benchmark alraedy in the list (at that point) that matches will be dropped from the list. If the first entry in the section is an exclusion then all known benchmarks are first added to the list before the exclusion is applied.

For example:

[benchmarks]
name        metafile
bench1      somedir/bm_bench1
bench2      somedir/bm_bench2
bench3      anotherdir/mybench.toml
[group default]
-bench1

This means by default only "bench2" and "bench3" are run.

Merging Manifests

To combine manifests, use the [includes] section in the manifest:

[includes]
project1/benchmarks/MANIFEST
project2/benchmarks/MANIFEST
<default>

Note that <default> is the same as including the manifest file for the default pyperformance benchmarks.

A Local Benchmark Suite

Often a project will have more than one benchmark that it will treat as a suite. pyperformance handles this without any extra work.

In the dirctory holding the manifest file put all the benchmarks. Then put <local> in the "metafile" column, like this:

[benchmarks]
name        metafile
bench1      <local>
bench2      <local>
bench3      <local>
bench4      <local>
bench5      <local>

It will look for DIR/bm_NAME/pyproject.toml.

If there are also variants, identify the main benchmark in the "metafile" value, like this:

[benchmarks]
name        metafile
bench1      <local>
bench2      <local>
bench3      <local>
variant1    <local:bench3>
variant2    <local:bench3>

pyperformance will look for DIR/bm_BASE/bm_NAME.toml, where "BASE" is the part after "local:".

A Project's Benchmark Suite

A Python project can identify its benchmark suite by putting the path to the manifest file in the project's top-level pyproject.toml. Additional manifests can be identified as well:

[tool.pyperformance]
manifest = "..."
manifests = ["...", "..."]

(Reminder: that is the pyproject.toml, not the manifest file.)

Benchmark Metadata Files

A benchmark's metadata file (usually pyproject.toml) follows the format specified in PEP 621 <https://www.python.org/dev/peps/pep-0621> and PEP 518 <https://www.python.org/dev/peps/pep-0518>. So there are two supported sections in the file: "project" and "tool.pyperformance".

A typical metadata file will look something like this:

[project]
version = "0.9.1"
dependencies = ["pyperf"]
dynamic = ["name"]
[tool.pyperformance]
name = "my_benchmark"

A highly detailed one might look like this:

[project]
name = "pyperformance_bm_json_dumps"
version = "0.9.1"
description = "A benchmark for json.dumps()"
requires-python = ">=3.8"
dependencies = ["pyperf"]
urls = {repository = "https://github.com/python/pyperformance"}
dynamic = ["version"]
[tool.pyperformance]
name = "json_dumps"
tags = "serialize"
runscript = "bench.py"
datadir = ".data-files/extras"
extra_opts = ["--special"]

Inheritance

For one benchmark to inherit from another (or from common metadata), the "inherits" field is available:

[project]
dependencies = ["pyperf"]
dynamic = ["name", "version"]
[tool.pyperformance]
name = "my_benchmark"
inherits = "../common.toml"

All values in either section of the inherited metadata are treated as defaults, on top of which the current metadata is applied. In the above example, for instance, a value for "version" in common.toml would be used here.

If the "inherits" value is a directory (even for "..") then "base.toml" in that directory will be inherited.

For variants, the base pyproject.toml is the default value for "inherits".

Inferred Values

In some situations, omitted values will be inferred from other available data (even for required fields).

When the name is inferred from the filename for a regularly structured benchmark, the "bm_" prefix is removed from the benchmark's directory. If it is a variant that prefix is removed from the metadata filename, as well as the .toml suffix.

The [project] Section

"R": required "T": inferred from the tool section "B": inferred from the inherited metadata "D": for default benchmarks, inferred from pyperformance

"dynamic" is required by PEP 621 for when a field will be filled in dynamically by the tool. This is especially important for required fields.

All other PEP 621 fields are optional (e.g. requires-python = ">=3.8", {repository = "https://github.com/..."}).

The [tool.pyperformance] Section

"R": required "B": inferred from the inherited metadata "F": inferred from filename

Optimizations

2016-12-14: speedup method calls

Optimization: Speedup method calls 1.2x <https://bugs.python.org/issue26110>, commit f2392133 <https://github.com/python/cpython/commit/f2392133eba777f05947a8996c507690b95379c3>.

2016-04-22: pymalloc allocator

Optimization: PyMem_Malloc() now uses the fast pymalloc allocator <http://bugs.python.org/issue26249>, commit f5c4b990 <https://github.com/python/cpython/commit/f5c4b99034fae12ac2b9498dd12b5b3f352b90c8>.

Changes of at least 5%:

2015-12-07: Optimize ElementTree.iterparse(), xml_etree_iterparse

Optimization: Issue #25638: Optimized ElementTree.iterparse(); it is now 2x faster <http://bugs.python.org/issue25638>, commit 9ec5e25f2 <https://github.com/python/cpython/commit/9ec5e25f26a490510bb5da5c26a276cd30a263a0>.

2015-09-19: PGO uses test suite, pidigits

Optimization: Issue #24915: Add Clang support to PGO builds and use the test suite for profile data <http://bugs.python.org/issue24915>, commit 7188a3ef <https://github.com/python/cpython/commit/7188a3efe07b9effdb760f3a96783f250214f0be>.

Changes of at least 5%:

Changes of at least 5%, sadly two benchmarks also became slower:

2015-05-30: C implementation of collections.OrderedDict, html5lib

Optimization: Issue #16991: Add a C implementation of collections.OrderedDict <http://bugs.python.org/issue16991>, commit 96c6af9b <https://github.com/python/cpython/commit/96c6af9b207c188c52ac53ce87bb7f2dea3f328b>.

2015-05-23: C implementation of functools.lru_cache(), sympy

Optimization: Issue #14373: Added C implementation of functools.lru_cache() <http://bugs.python.org/issue14373>, commit 1c858c35 <https://github.com/python/cpython/commit/1c858c352b8c11419f79f586334c49378726dba8>.

Changes of at least 5%:

pickle_dict is seen as 1.06x slower, but since pickle doesn't use functools.lru_cache(), the change is ignored in the table.

Slowdown

2016-09-11: regex_compile

Slowdown: convert re flags to (much friendlier) IntFlag constants (issue #28082) <http://bugs.python.org/issue28082>, commit f93395bc <https://github.com/python/cpython/commit/f93395bc5125c99539597bf134ca8bcf9707655b>.

Timeline

April, 2016 -> May, 2016

2016-04-01 .. 2016-05-01:

call_method

Timeline 2016-04-01 .. 2017-01-01:

crypto_pyaes

2016-03-01 .. 2016-06-01:

json_loads

Progress on 21 months, 2015-01-01 .. 2016-10-01:

logging_silent

pickle

pickle, 2016-08-02 .. 2016-09-08:

pickle dict/list:

unpickle:

python_startup

2015-04-01 .. 2015-10-01:

2016-04-01 .. 2017-01-01:

regex_compile

telco

scimark

2016-10-01 .. 2017-03-31:

sqlalchemy_declarative

sympy

2016-04-01 .. 2016-10-01:

xml_etree_generate

Version 1.14.0

Version 1.13.0 (2025-10-27)

Version 1.12.0 (2025-10-24)

Version 1.11.0 (2024-03-09)

Version 1.10.0 (2023-10-22)

Version 1.0.9 (2023-06-14)

Version 1.0.8 (2023-06-02)

Version 1.0.7 (2023-04-22)

Version 1.0.6 (2022-11-20)

Version 1.0.4 (2022-01-25)

Version 1.0.3 (2021-12-20)

Version 1.0.2 (2021-05-11)

Version 1.0.1 (2020-03-26)

Version 1.0.0 (2019-12-17)

Version 0.9.1 (2019-07-29)

Version 0.9.0 (2019-05-29)

Version 0.8.0 (2019-05-10)

Version 0.7.0 (2018-10-16)

Version 0.6.1 (2018-01-11)

Version 0.6.0 (2017-07-06)

Version 0.5.5 (2017-05-29)

Version 0.5.4 (2017-04-10)

Version 0.5.3 (2017-03-27)

Version 0.5.2 (2017-03-17)

Version 0.5.1 (2017-01-16)

Version 0.5.0 (2016-11-16)

Version 0.4.0 (2016-11-07)

Version 0.3.2 (2016-10-19)

Version 0.3.1 (2016-10-19)

Version 0.3.0 (2016-10-11)

New benchmarks:

Enhancements:

Bugfixes:

Rename benchmarks:

Minor changes:

Version 0.2.2 (2016-09-19)

Version 0.2.1 (2016-09-10)

Version 0.2 (2016-09-01)

Version 0.1.2 (2016-08-27)

Version 0.1.1 (2016-08-24)

Version 0.1 (2016-08-24)

History

Projected moved to <https://github.com/python/performance> in August 2016. Files reorganized, benchmarks patched to use the perf module to run benchmark in multiple processes.

Project started in December 2008 by Collin Winter and Jeffrey Yasskin for the Unladen Swallow project. The project was hosted at <https://hg.python.org/benchmarks> until Feb 2016

Other Python Benchmarks:

See also the Python speed mailing list <https://mail.python.org/mailman/listinfo/speed> and the Python pyperf module <http://pyperf.readthedocs.io/> (used by pyperformance).

pyperformance is not tuned for PyPy yet: use the PyPy benchmarks project <https://foss.heptapod.net/pypy/benchmarks> instead to measure PyPy performances.

Image generated by bm_raytrace (pure Python raytrace): [image: Pure Python raytracer] [image]