--- name: "pythonperformancebenchmarksuite" section: "1" platform: "linux" package: "python-pyperformance" version: "1.14.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/pythonperformancebenchmarksuite" --- # pythonperformancebenchmarksuite(1) Python Performance Benchmark Suite Documentation The pyperformance project is intended to be an authoritative source of benchmarks for all Python implementations. The focus is on real-world benchmarks, rather than synthetic benchmarks, using whole applications when possible. pyperformance is distributed under the MIT license. Documentation: ## NAME pythonperformancebenchmarksuite - Python Performance Benchmark Suite Documentation The **pyperformance** project is intended to be an authoritative source of benchmarks for all Python implementations. The focus is on real-world benchmarks, rather than synthetic benchmarks, using whole applications when possible. - • pyperformance documentation \ - • pyperformance GitHub project \ (source code, issues) - • Download pyperformance on PyPI \ pyperformance is distributed under the MIT license. Documentation: ## USAGE ### Installation Command to install pyperformance: ``` python3 -m pip install pyperformance ``` The command installs a new **pyperformance** program. If needed, **pyperf** and **six** dependencies are installed automatically. pyperformance works on Python 3.6 and newer, but it may work on Python 3.4 and 3.5. At runtime, Python development files (header files) may be needed to install some dependencies like **dulwich\_log** or **psutil**, to build their C extension. Commands on Fedora to install dependencies: - • Python 3: **sudo dnf install python3-devel** - • PyPy: **sudo dnf install pypy-devel** ### Windows notes On Windows, to allow pyperformance to build dependencies from source like **greenlet**, **dulwich** or **psutil**, if you want to use a **python.exe** built from source, you should not use the **python.exe** directly. Instead, you must run the little-known command **PC\\layout** to create a filesystem layout that resembles an installed Python: ``` .\python.bat -m PC.layout --preset-default --copy installed -v ``` (Use the **--help** flag for more info about **PC\\layout**.) Now you can use the "installed" Python executable: ``` installed\python.exe -m pip install pyperformance installed\python.exe -m pyperformance run ... ``` Using an *actually* installed Python executable (e.g. via **py**) works fine too. ### Run benchmarks Commands to compare Python 3.6 and Python 3.7 performance: ``` pyperformance run --python=python3.6 -o py36.json pyperformance run --python=python3.7 -o py37.json pyperformance compare py36.json py37.json ``` Note: **python3 -m pyperformance ...** syntax works as well (ex: **python3 -m** **pyperformance run -o py37.json**), but requires to install pyperformance on each tested Python version. JSON files are produced by the pyperf module and so can be analyzed using pyperf commands: ``` python3 -m pyperf show py36.json python3 -m pyperf check py36.json python3 -m pyperf metadata py36.json python3 -m pyperf stats py36.json python3 -m pyperf hist py36.json python3 -m pyperf dump py36.json (...) ``` It's also possible to use pyperf to compare results of two JSON files: ``` python3 -m pyperf compare_to py36.json py37.json --table ``` ### Basic commands pyperformance actions: ``` run Run benchmarks on the running python show Display a benchmark file compare Compare two benchmark files list List benchmarks of the running Python list_groups List benchmark groups of the running Python venv Actions on the virtual environment ``` ### Common options Options available to all commands: ``` -h, --help show this help message and exit ``` ### run Run benchmarks on the running python. Usage: ``` pyperformance run [-h] [-r] [-f] [--debug-single-value] [-v] [-m] [--affinity CPU_LIST] [-o FILENAME] [--append FILENAME] [--manifest MANIFEST] [--timeout TIMEOUT] [-b BM_LIST] [--inherit-environ VAR_LIST] [-p PYTHON] [--hook HOOK] ``` options: ``` -h, --help show this help message and exit -r, --rigorous Spend longer running tests to get more accurate results -f, --fast Get rough answers quickly --debug-single-value Debug: fastest mode, only compute a single value -v, --verbose Print more output -m, --track-memory Track memory usage. This only works on Linux. --affinity CPU_LIST Specify CPU affinity for benchmark runs. This way, benchmarks can be forced to run on a given CPU to minimize run to run variation. -o FILENAME, --output FILENAME Run the benchmarks on only one interpreter and write benchmark into FILENAME. Provide only baseline_python, not changed_python. --append FILENAME Add runs to an existing file, or create it if it doesn't exist --timeout TIMEOUT Specify a timeout in seconds for a single benchmark run (default: disabled) --manifest MANIFEST benchmark manifest file to use -b BM_LIST, --benchmarks BM_LIST Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments. --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) --same-loops SAME_LOOPS Use the same number of loops as a previous run (i.e., don't recalibrate). Should be a path to a .json file from a previous run. --hook HOOK Apply the given pyperf hook when running the benchmarks. ``` ### show Display a benchmark file. Usage: ``` show FILENAME ``` positional arguments: ``` FILENAME ``` ### compare Compare two benchmark files. Usage: ``` pyperformance compare [-h] [-v] [-O STYLE] [--csv CSV_FILE] [--inherit-environ VAR_LIST] [-p PYTHON] baseline_file.json changed_file.json ``` positional arguments: ``` baseline_file.json changed_file.json ``` options: ``` -v, --verbose Print more output -O STYLE, --output_style STYLE What style the benchmark output should take. Valid options are 'normal' and 'table'. Default is normal. --csv CSV_FILE Name of a file the results will be written to, as a three-column CSV file containing minimum runtimes for each benchmark. --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` ### list List benchmarks of the running Python. Usage: ``` pyperformance list [-h] [--manifest MANIFEST] [-b BM_LIST] [--inherit-environ VAR_LIST] [-p PYTHON] ``` options: ``` --manifest MANIFEST benchmark manifest file to use -b BM_LIST, --benchmarks BM_LIST Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments. --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` Use **python3 -m pyperformance list -b all** to list all benchmarks. ### list\_groups List benchmark groups of the running Python. Usage: ``` pyperformance list_groups [-h] [--manifest MANIFEST] [--inherit-environ VAR_LIST] [-p PYTHON] ``` options: ``` --manifest MANIFEST benchmark manifest file to use --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` ### venv Actions on the virtual environment. Actions: ``` show Display the path to the virtual environment and its status (created or not) create Create the virtual environment recreate Force the recreation of the the virtual environment remove Remove the virtual environment ``` Common options: ``` --venv VENV Path to the virtual environment --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` ### venv show Display the path to the virtual environment and its status (created or not). Usage: ``` pyperformance venv show [-h] [--venv VENV] [--inherit-environ VAR_LIST] [-p PYTHON] ``` ### venv create Create the virtual environment. Usage: ``` pyperformance venv create [-h] [--venv VENV] [--manifest MANIFEST] [-b BM_LIST] [--inherit-environ VAR_LIST] [-p PYTHON] ``` options: ``` --manifest MANIFEST benchmark manifest file to use -b BM_LIST, --benchmarks BM_LIST Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments. ``` ### venv recreate Force the recreation of the the virtual environment. Usage: ``` pyperformance venv recreate [-h] [--venv VENV] [--manifest MANIFEST] [-b BM_LIST] [--inherit-environ VAR_LIST] [-p PYTHON] ``` options: ``` --manifest MANIFEST benchmark manifest file to use -b BM_LIST, --benchmarks BM_LIST Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments. ``` ### venv remove Remove the virtual environment. Usage: ``` pyperformance venv remove [-h] [--venv VENV] [--inherit-environ VAR_LIST] [-p PYTHON] ``` ### Compile Python to run benchmarks pyperformance actions: ``` compile Compile and install CPython and run benchmarks on installed Python compile_all Compile and install CPython and run benchmarks on installed Python on all branches and revisions of CONFIG_FILE upload Upload JSON results to a Codespeed website ``` All these commands require a configuration file. Simple configuration usable for **compile** (but not for **compile\_all** nor **upload**), **doc/benchmark.conf**: ``` [config] json_dir = ~/prog/python/bench_json [scm] repo_dir = ~/prog/python/master update = True [compile] bench_dir = ~/prog/python/bench_tmpdir [run_benchmark] system_tune = True affinity = 2,3 ``` Configuration file sample with comments, **doc/benchmark.conf.sample**: ``` [config] # Directory where JSON files are written. # - uploaded files are moved to json_dir/uploaded/ # - results of patched Python are written into json_dir/patch/ json_dir = ~/json # If True, compile CPython in debug mode (LTO and PGO disabled), # run benchmarks with --debug-single-sample, and disable upload. # # Use this option to quickly test a configuration. debug = False [scm] # Directory of CPython source code (Git repository) repo_dir = ~/cpython # Update the Git repository (git fetch)? update = True # Name of the Git remote, used to create revision of # the Git branch. For example, use revision 'remotes/origin/3.6' # for the branch '3.6'. git_remote = remotes/origin [compile] # Create files into bench_dir: # - bench_dir/bench-xxx.log # - bench_dir/prefix/: where Python is installed # - bench_dir/venv/: Virtual environment used by pyperformance bench_dir = ~/bench_tmpdir # Link Time Optimization (LTO)? lto = True # Profiled Guided Optimization (PGO)? pgo = True # Build the experimental just-in-time (JIT) compiler? # Possible values are: # - no: (default) do not build the JIT or the micro-op interpreter. # The new PYTHON_JIT environment variable has no effect. # - yes: build the JIT and enable it by default. PYTHON_JIT=0 can be used to # disable it at runtime. # - yes-off: build the JIT, but do not enable it by default. PYTHON_JIT=1 can # be used to enable it at runtime. # - interpreter: do not build the JIT, but do build and enable the micro-op # interpreter. This is useful for those of us who find ourselves developing # or debugging micro-ops (but don’t want to deal with the JIT). # PYTHON_JIT=0 can be used to disable the micro-op interpreter at runtime. jit = no # The space-separated list of libraries that are package-only, # i.e., locally installed but not on header and library paths. # For each such library, determine the install path and add an # appropriate subpath to CFLAGS and LDFLAGS declarations passed # to configure. As an exception, the prefix for openssl, if that # library is present here, is passed via the --with-openssl # option. Currently, this only works with Homebrew on macOS. # If running on macOS with Homebrew, you probably want to use: # pkg_only = openssl readline sqlite3 xz zlib # The version of zlib shipping with macOS probably works as well, # as long as Apple's SDK headers are installed. pkg_only = # Install Python? If false, run Python from the build directory # # WARNING: Running Python from the build directory introduces subtle changes # compared to running an installed Python. Moreover, creating a virtual # environment using a Python run from the build directory fails in many cases, # especially on Python older than 3.4. Only disable installation if you # really understand what you are doing! install = True # Specify '-j' parameter in 'make' command jobs = 8 [run_benchmark] # Run "sudo python3 -m pyperf system tune" before running benchmarks? system_tune = True # --manifest option for 'pyperformance run' manifest = # --benchmarks option for 'pyperformance run' benchmarks = # --affinity option for 'pyperf system tune' and 'pyperformance run' affinity = # Upload generated JSON file? # # Upload is disabled on patched Python, in debug mode or if install is # disabled. upload = False # --rigorous option for 'pyperformance run' rigorous = False # Configuration to upload results to a Codespeed website [upload] url = environment = executable = project = [compile_all] # List of CPython Git branches branches = default 3.6 3.5 2.7 # List of revisions to benchmark by compile_all [compile_all_revisions] # list of 'sha1=' (default branch: 'master') or 'sha1=branch' # used by the "pyperformance compile_all" command # e.g.: 11159d2c9d6616497ef4cc62953a5c3cc8454afb = ``` ### compile Compile Python, install Python and run benchmarks on the installed Python. Usage: ``` pyperformance compile [-h] [--patch PATCH] [-U] [-T] [--inherit-environ VAR_LIST] [-p PYTHON] config_file revision [branch] ``` positional arguments: ``` config_file Configuration filename revision Python benchmarked revision branch Git branch ``` options: ``` --patch PATCH Patch file -U, --no-update Don't update the Git repository -T, --no-tune Don't run 'pyperf system tune' to tune the system for benchmarks --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` Notes: - • PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: **Modules/socketmodule.c:7743:1: internal compiler error: in edge\_badness,** **at ipa-inline.c:895** ### compile\_all Compile all branches and revisions of CONFIG\_FILE. Usage: ``` pyperformance compile_all [-h] [--inherit-environ VAR_LIST] [-p PYTHON] config_file ``` positional arguments: ``` config_file Configuration filename ``` options: ``` --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` ### upload Upload results from a JSON file to a Codespeed website. Usage: ``` pyperformance upload [-h] [--inherit-environ VAR_LIST] [-p PYTHON] config_file json_file ``` positional arguments: ``` config_file Configuration filename json_file JSON filename ``` options: ``` --inherit-environ VAR_LIST Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses. -p PYTHON, --python PYTHON Python executable (default: use running Python) ``` ### How to get stable benchmarks - • Run **python3 -m pyperf system tune** command - • Compile Python using LTO (Link Time Optimization) and PGO (profile guided optimizations): use the pyperformance compile command with uses LTO and PGO by default - • See advices of the pyperf documentation: How to get reproductible benchmark results \. ### pyperformance virtual environment To run benchmarks, pyperformance first creates a virtual environment. It installs requirements with fixed versions to get a reproductible environment. The system Python has unknown module installed with unknown versions, and can have **.pth** files run at Python startup which can modify Python behaviour or at least slow down Python startup. ### What is the goal of pyperformance A benchmark is always written for a specific purpose. Depending how the benchmark is written and how the benchmark is run, the result can be different and so have a different meaning. The pyperformance benchmark suite has multiple goals: - • Help to detect performance regression in a Python implementation - • Validate that an optimization change makes Python faster and don't performance regressions, or only minor regressions - • Compare two implementations of Python, for example CPython and PyPy - • Showcase of Python performance which ideally would be representative of performances of applications running on production ### Don't disable GC nor ASLR The pyperf module and pyperformance benchmarks are designed to produce reproductible results, but not at the price of running benchmarks in a special mode which would not be used to run applications in production. For these reasons, the Python garbage collector, Python randomized hash function and system ASLR (Address Space Layout Randomization) are **not disabled**. Benchmarks don't call **gc.collect()** neither since CPython implements it with stop-the-world \ and so applications don't call it to not kill performances. ### Include outliers and spikes Moreover, while the pyperf documentation explains how to reduce the random noise of the system and other applications, some benchmarks use the system and so can get different timing depending on the system workload, depending on I/O performances, etc. Outliers and temporary spikes in results are **not** **automatically removed**: values are summarized by computing the average (arithmetic mean) and standard deviation which "contains" these spikes, instead of using median and the median absolute deviation for example which to ignore outliers. It is deliberate choice since applications running in production are impacted by such temporary slowdown caused by various things like a garbage collection or a JIT compilation. ### Warmups and steady state A borderline issue are the benchmarks "warmups". The first values of each worker process are always slower: 10% slower in the best case, it can be 1000% slower or more on PyPy. Right now (2017-04-14), pyperformance ignore first values considered as warmup until a benchmark reachs its "steady state". The "steady state" can include temporary spikes every 5 values (ex: caused by the garbage collector), and it can still imply further JIT compiler optimizations but with a "low" impact on the average pyperformance. To be clear "warmup" and "steady state" are a work-in-progress and a very complex topic, especially on PyPy and its JIT compiler. ### Notes Tool for comparing the performance of two Python implementations. pyperformance will run Student's two-tailed T test on the benchmark results at the 95% confidence level to indicate whether the observed difference is statistically significant. Omitting the **-b** option will result in the default group of benchmarks being run Omitting **-b** is the same as specifying *-b default*. To run every benchmark pyperformance knows about, use **-b all**. To see a full list of all available benchmarks, use *--help*. Negative benchmarks specifications are also supported: *-b -2to3* will run every benchmark in the default group except for 2to3 (this is the same as *-b default,-2to3*). *-b all,-django* will run all benchmarks except the Django templates benchmark. Negative groups (e.g., *-b -default*) are not supported. Positive benchmarks are parsed before the negative benchmarks are subtracted. If **--track\_memory** is passed, pyperformance will continuously sample the benchmark's memory usage. This currently only works on Linux 2.6.16 and higher or Windows with PyWin32. Because **--track\_memory** introduces performance jitter while collecting memory measurements, only memory usage is reported in the final report. ## BENCHMARKS Also see Custom Benchmarks \<\> regarding how to create your own benchmarks or use someone else's. ### Available Groups Like individual benchmarks (see "Available benchmarks" below), benchmarks group are allowed after the *-b* option. Use **python3 -m pyperformance list\_groups** to list groups and their benchmarks. Available benchmark groups: - • **all**: Group including all benchmarks - • **apps**: "High-level" applicative benchmarks (2to3, Chameleon, Tornado HTTP) - • **default**: Group of benchmarks run by default by the **run** command - • **math**: Float and integers - • **regex**: Collection of regular expression benchmarks - • **serialize**: Benchmarks on **pickle** and **json** modules - • **startup**: Collection of microbenchmarks focused on Python interpreter start-up time. - • **template**: Templating libraries Use the **python3 -m pyperformance list\_groups** command to list groups and their benchmarks. ### Available Benchmarks In pyperformance 0.5.5, the following microbenchmarks have been removed because they are too short, not representative of real applications and are too unstable. - • **call\_method\_slots** - • **call\_method\_unknown** - • **call\_method** - • **call\_simple** - • **pybench** ### 2to3 Run the 2to3 tool on the **pyperformance/benchmarks/data/2to3/** directory: copy of the **django/core/\*.py** files of Django 1.1.4, 9 files. Run the **python -m lib2to3 -f all \** command where **python** is **sys.executable**. So the test does not only mesure the performance of Python itself, but also the performance of the **lib2to3** module which can change depending on the Python version. **Note:** Files are called **.py.txt** instead of **.py** to not run PEP 8 checks on them, and more generally to not modify them. ### async\_tree Async workload benchmark, which calls **asyncio.gather()** on a tree (6 levels deep, 6 branches per level) with the leaf nodes simulating some \[potentially\] async work (depending on the benchmark variant). Available variants: - • **async\_tree**: no actual async work at any leaf node. - • **async\_tree\_io**: all leaf nodes simulate async IO workload (async sleep 50ms). - • **async\_tree\_memoization**: all leaf nodes simulate async IO workload with 90% of the data memoized. - • **async\_tree\_cpu\_io\_mixed**: half of the leaf nodes simulate CPU-bound workload (**math.factorial(500)**) and the other half simulate the same workload as the **async\_tree\_memoization** variant. These benchmarks also have an "eager" flavor that uses asyncio eager task factory, if available. ### base64 Benchmark the **base64** module's encoding and decoding functions. Each algorithm has **\_small** and **\_large** variants that test both encode and decode in a single benchmark: - • **\_small**: Balanced iterations across 20B, 127B, 3KiB, and 9KB data sizes more likely to show the impact of overhead. - • **\_large**: Large data focus with 100KiB and ~1MiB data sizes likely to demonstrate implementation efficiency. Available benchmarks: - • **base64\_small**, **base64\_large**: Standard Base64 encoding and decoding - • **urlsafe\_base64\_small**: URL-safe Base64 (small only, as URLs shouldn't contain huge data) - • **base32\_small**, **base32\_large**: Base32 encoding and decoding - • **base16\_small**, **base16\_large**: Base16/hex encoding and decoding - • **ascii85\_small**, **ascii85\_large**: Ascii85 encoding and decoding (includes **wrapcol=76** code path) - • **base85\_small**, **base85\_large**: Base85 encoding and decoding See the base64 module \. ### chameleon Render a template using the **chameleon** module to create an HTML table of 500 lignes and 10 columns. See the **chameleon.PageTemplate** class. ### chaos Create chaosgame-like fractals. Command lines options: ``` --thickness THICKNESS Thickness (default: 0.25) --width WIDTH Image width (default: 256) --height HEIGHT Image height (default: 256) --iterations ITERATIONS Number of iterations (default: 5000) --filename FILENAME.PPM Output filename of the PPM picture --rng-seed RNG_SEED Random number generator seed (default: 1234) ``` When **--filename** option is used, the timing includes the time to create the PPM file. Copyright (C) 2005 Carl Friedrich Bolz \[image: Chaos game, bm\_chaos benchmark\] \[image\] Image generated by bm\_chaos (took 3 sec on CPython 3.5) with the command: ``` python3 pyperformance/benchmarks/bm_chaos.py --worker -l1 -w0 -n1 --filename chaos.ppm --width=512 --height=512 --iterations 50000 ``` ### crypto\_pyaes benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the **pyaes** module. The benchmark is slower on CPython 3 compared to CPython 2.7, because CPython 3 has no more "small int" type (**int**). The CPython 3 **int** type now always has an arbitrary size, as CPython 2.7 **long** type. See pyaes \: A pure-Python implementation of the AES block cipher algorithm and the common modes of operation (CBC, CFB, CTR, ECB and OFB). ### deepcopy Benchmark the Python *copy.deepcopy* method. The *deepcopy* method is performed on a nested dictionary and a dataclass. ### deltablue DeltaBlue benchmark Ported for the PyPy project. Contributed by Daniel Lindsley This implementation of the DeltaBlue benchmark was directly ported from the V8's source code \, which was in turn derived from the Smalltalk implementation by John Maloney and Mario Wolczko. The original Javascript implementation was licensed under the GPL. It's been updated in places to be more idiomatic to Python (for loops over collections, a couple magic methods, **OrderedCollection** being a list & things altering those collections changed to the builtin methods) but largely retains the layout & logic from the original. (Ugh.) ### django\_template Use the Django template system to build a 150x150-cell HTML table. Use **Context** and **Template** classes of the **django.template** module. ### dulwich\_log Iterate on commits of the asyncio Git repository using the Dulwich module. Use **pyperformance/benchmarks/data/asyncio.git/** repository. Pseudo-code of the benchmark: ``` repo = dulwich.repo.Repo(repo_path) head = repo.head() for entry in repo.get_walker(head): pass ``` See the Dulwich project \. ### docutils Use Docutils \ to convert Docutils' documentation to HTML. Representative of building a medium-sized documentation set. ### fannkuch The Computer Language Benchmarks Game: \ Contributed by Sokolov Yura, modified by Tupteq. ### float Artificial, floating point-heavy benchmark originally used by Factor. Create 100,000 point objects which compute **math.cos()**, **math.sin()** and **math.sqrt()** Changed in version 0.5.5: Use **\_\_slots\_\_** on the Point class to focus the benchmark on float rather than testing performance of class attributes. ### genshi Render a template using Genshi (**genshi.template** module): - • **genshi\_text**: Render a HTML template using the **NewTextTemplate** class - • **genshi\_xml**: Render an XML template using the **MarkupTemplate** class See the Genshi project \. ### go Artificial intelligence playing the Go board game. Use Zobrist hashing \. ### hexiom Solver of Hexiom board game (level 25 by default). Command line option: ``` --level {2,10,20,25,30,36} Hexiom board level (default: 25) ``` ### hg\_startup Get Mercurial's help screen. Measure the performance of the **python path/to/hg help** command using **pyperf.Runner.bench\_command()**, where **python** is **sys.executable** and **path/to/hg** is the Mercurial program installed in a virtual environmnent. The **bench\_command()** redirects stdout and stderr into **/dev/null**. See the Mercurial project \. ### html5lib Parse the **pyperformance/benchmarks/data/w3\_tr\_html5.html** HTML file (132 KB) using **html5lib**. The file is the HTML 5 specification, but truncated to parse the file in less than 1 second (around 250 ms). On CPython, after 3 warmups, the benchmarks enters a cycle of 5 values: every 5th value is 10% slower. Plot of 1 run of 50 values (the warmup is not rendered): \[image: html5lib values\] \[image\] See the html5lib project \. ### json\_dumps, json\_loads Benchmark **dumps()** and **loads()** functions of the **json** module. **bm\_json\_dumps.py** command line option: ``` --cases CASES Comma separated list of cases. Available cases: EMPTY, SIMPLE, NESTED, HUGE. By default, run all cases. ``` ### logging Benchmarks on the **logging** module: - • **logging\_format**: Benchmark **logger.warn(fmt, str)** - • **logging\_simple**: Benchmark **logger.warn(msg)** - • **logging\_silent**: Benchmark **logger.debug(msg)** when the log is ignored Script command line option: ``` format silent simple ``` See the logging module \. ### mako Use the Mako template system to build a 150x150-cell HTML table. Includes: - • two template inherences - • HTML escaping, XML escaping, URL escaping, whitespace trimming - • function defitions and calls - • forloops See the Mako project \. ### mdp Battle with damages and topological sorting of nodes in a graph. See Topological sorting \. ### meteor\_contest Solver for Meteor Puzzle board. Meteor Puzzle board: \ The Computer Language Benchmarks Game: \ Contributed by Daniel Nanz, 2008-08-21. ### nbody N-body benchmark from the Computer Language Benchmarks Game. Microbenchmark on floating point operations. This is intended to support Unladen Swallow's perf.py. Accordingly, it has been modified from the Shootout version: - • Accept standard Unladen Swallow benchmark options. - • Run report\_energy()/advance() in a loop. - • Reimplement itertools.combinations() to work with older Python versions. Pulled from: \ Contributed by Kevin Carson. Modified by Tupteq, Fredrik Johansson, and Daniel Nanz. ### python\_startup, python\_startup\_nosite - • **python\_startup**: Measure the Python startup time, run **python -c pass** where **python** is **sys.executable** - • **python\_startup\_nosite**: Measure the Python startup time without importing the **site** module, run **python -S -c pass** where **python** is **sys.executable** Run the benchmark with **pyperf.Runner.bench\_command()**. ### nqueens Simple, brute-force N-Queens solver. See Eight queens puzzle \. ### pathlib Test the performance of operations of the **pathlib** module of the standard library. This benchmark stresses the creation of small objects, globbing, and system calls. See the documentation of the pathlib module \. ### pickle pickle benchmarks (serialize): - • **pickle**: use the cPickle module to pickle a variety of datasets. - • **pickle\_dict**: microbenchmark; use the cPickle module to pickle a lot of dicts. - • **pickle\_list**: microbenchmark; use the cPickle module to pickle a lot of lists. - • **pickle\_pure\_python**: use the pure-Python pickle module to pickle a variety of datasets. unpickle benchmarks (deserialize): - • **unpickle**: use the cPickle module to unnpickle a variety of datasets. - • **unpickle\_list** - • **unpickle\_pure\_python**: use the pure-Python pickle module to unpickle a variety of datasets. ### pidigits Calculating 2,000 digits of π. This benchmark stresses big integer arithmetic. Command line option: ``` --digits DIGITS Number of computed pi digits (default: 2000) ``` Adapted from code on: \ ### pyflate Benchmark of a pure-Python bzip2 decompressor: decompress the **pyperformance/benchmarks/data/interpreter.tar.bz2** file in memory. Copyright 2006--2007-01-21 Paul Sladen: \ You may use and distribute this code under any DFSG-compatible license (eg. BSD, GNU GPLv2). Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. This is probably most useful for research purposes/index building; there is certainly some room for improvement in the Huffman bit-matcher. With the as-written implementation, there was a known bug in BWT decoding to do with repeated strings. This has been worked around; see 'bwt\_reverse()'. Correct output is produced in all test cases but ideally the problem would be found... ### raytrace Simple raytracer. Command line options: ``` --width WIDTH Image width (default: 100) --height HEIGHT Image height (default: 100) --filename FILENAME.PPM Output filename of the PPM picture ``` This file contains definitions for a simple raytracer. Copyright Callum and Tony Garnock-Jones, 2008. This file may be freely redistributed under the MIT license, \ From \ \[image: Pure Python raytracer\] \[image\] Image generated by the command (took 68.4 sec on CPython 3.5): ``` python3 pyperformance/benchmarks/bm_raytrace.py --worker --filename=raytrace.ppm -l1 -w0 -n1 -v --width=800 --height=600 ``` ### regex\_compile Stress the performance of Python's regex compiler, rather than the regex execution speed. Benchmark how quickly Python's regex implementation can compile regexes. We bring in all the regexes used by the other regex benchmarks, capture them by stubbing out the re module, then compile those regexes repeatedly. We muck with the re module's caching to force it to recompile every regex we give it. ### regex\_dna regex DNA benchmark using "fasta" to generate the test case. The Computer Language Benchmarks Game \ regex-dna Python 3 #5 program: contributed by Dominique Wahli 2to3 modified by Justin Peel fasta Python 3 #3 program: modified by Ian Osgood modified again by Heinrich Acker modified by Justin Peel Modified by Christopher Sean Forgeron ### regex\_effbot Some of the original benchmarks used to tune mainline Python's current regex engine. ### regex\_v8 Python port of V8's regex benchmark. Automatically generated on 2009-01-30. This benchmark is generated by loading 50 of the most popular pages on the web and logging all regexp operations performed. Each operation is given a weight that is calculated from an estimate of the popularity of the pages where it occurs and the number of times it is executed while loading each page. Finally the literal letters in the data are encoded using ROT13 in a way that does not affect how the regexps match their input. Ported to Python for Unladen Swallow. The original JS version can be found at \, r1243. ### richards The classic Python Richards benchmark. Based on a Java version. Based on original version written in BCPL by Dr Martin Richards in 1981 at Cambridge University Computer Laboratory, England and a C++ version derived from a Smalltalk version written by L Peter Deutsch. Java version: Copyright (C) 1995 Sun Microsystems, Inc. Translation from C++, Mario Wolczko Outer loop added by Alex Jacoby ### scimark - • **scimark\_sor**: Successive over-relaxation (SOR) \ benchmark - • **scimark\_sparse\_mat\_mult**: sparse matrix \ multiplication \ benchmark - • **scimark\_monte\_carlo**: benchmark on the Monte Carlo algorithm \ to compute the area of a disc - • **scimark\_lu**: LU decomposition \ benchmark - • **scimark\_fft**: Fast Fourier transform (FFT) \ benchmark ### spectral\_norm MathWorld: "Hundred-Dollar, Hundred-Digit Challenge Problems", Challenge #3. \ The Computer Language Benchmarks Game \ Contributed by Sebastien Loisel. Fixed by Isaac Gouy. Sped up by Josh Goldfoot. Dirtily sped up by Simon Descarpentries. Concurrency by Jason Stitt. ### sqlalchemy\_declarative, sqlalchemy\_imperative - • **sqlalchemy\_declarative**: SQLAlchemy Declarative benchmark using SQLite - • **sqlalchemy\_imperative**: SQLAlchemy Imperative benchmark using SQLite See the SQLAlchemy project \. ### sqlite\_synth Benchmark Python aggregate for SQLite. The goal of the benchmark (written for PyPy) is to test CFFI performance and going back and forth between SQLite and Python a lot. Therefore the queries themselves are really simple. See the SQLite project \ and the Python sqlite3 module (stdlib) \. ### sympy Benchmark on the **sympy** module: - • **sympy\_expand**: Benchmark **sympy.expand()** - • **sympy\_integrate**: Benchmark **sympy.integrate()** - • **sympy\_str**: Benchmark **str(sympy.expand())** - • **sympy\_sum**: Benchmark **sympy.summation()** On CPython, some **sympy\_sum** values are 5%-10% slower: ``` $ python3 -m pyperf dump sympy_sum.json Run 1: 1 warmup, 50 values, 1 loop - warmup 1: 404 ms (+63%) - value 1: 244 ms - value 2: 245 ms - value 3: 258 ms <---- - value 4: 245 ms - value 5: 245 ms - value 6: 279 ms (+12%) <---- - value 7: 246 ms - value 8: 244 ms - value 9: 245 ms - value 10: 255 ms <---- - value 11: 245 ms - value 12: 245 ms - value 13: 256 ms <---- - value 14: 248 ms - value 15: 245 ms - value 16: 245 ms ... ``` Plot of 1 run of 50 values (the warmup is not rendered): \[image: sympy\_sum values\] \[image\] See the sympy project \. ### telco Telco Benchmark for measuring the performance of decimal calculations: - • \ - • \ - • A call type indicator, **c**, is set from the bottom (least significant) bit of the duration (hence **c** is 0 or 1). - • A rate, **r**, is determined from the call type. Those calls with **c=0** have a low **r**: **0.0013**; the remainder (‘distance calls’) have a ‘premium’ **r**: **0.00894**. (The rates are, very roughly, in Euros or dollarates per second.) - • A price, **p**, for the call is then calculated (**p=r\*n**). This is rounded to exactly 2 fractional digits using round-half-even (Banker’s round to nearest). - • A basic tax, **b**, is calculated: **b=p\*0.0675** (6.75%). This is truncated to exactly 2 fractional digits (round-down), and the total basic tax variable is then incremented (**sumB=sumB+b**). - • For distance calls: a distance tax, **d**, is calculated: **d=p\*0.0341** (3.41%). This is truncated to exactly 2 fractional digits (round-down), and then the total distance tax variable is incremented (**sumD=sumD+d**). - • The total price, **t**, is calculated (**t=p+b**, and, if a distance call, **t=t+d**). - • The total prices variable is incremented (**sumT=sumT+t**). - • The total price, **t**, is converted to a string, **s**. The Python benchmark is implemented with the **decimal** module. See the Python decimal module (stdlib) \. ### tornado\_http Benchmark HTTP server of the **tornado** module See the Tornado project \. ### unpack\_sequence Microbenchmark for unpacking lists and tuples. Pseudo-code: ``` a, b, c, d, e, f, g, h, i, j = to_unpack ``` where **to\_unpack** is **tuple(range(10))** or **list(range(10))**. ### xml\_etree Benchmark the **ElementTree** API of the **xml.etree** module: - • **xml\_etree\_generate**: Create an XML document - • **xml\_etree\_iterparse**: Benchmark **etree.iterparse()** - • **xml\_etree\_parse**: Benchmark **etree.parse()** - • **xml\_etree\_process**: Process an XML document See the Python xml.etree.ElementTree module (stdlib) \. ## CUSTOM BENCHMARKS pyperformance includes its own set of benchmarks (see Benchmarks \<\>). However, it also supports using custom benchmarks. ### Using Custom Benchmarks To use custom benchmarks, you will need to use the **--manifest** CLI option and provide the path to the manifest file describing those benchmarks. ### The pyperformance File Formats **pyperformance** uses two file formats to identify benchmarks: - • manifest - a set of benchmarks - • metadata - a single benchmark For each benchmark, there are two required files and several optional ones. Those files are expected to be in a specific directory structure (unless customized in the metadata). The structure (see below) is such that it's easy to maintain a benchmark (or set of benchmarks) on GitHub and distribute it on PyPI. It also simplifies publishing a Python project's benchmarks. The alternative is pointing people at a repo. Benchmarks can inherit metadata from other metadata files. This is useful for keeping common metadata for a set of benchmarks (e.g. "version") in one file. Likewise, benchmarks for a Python project can inherit metadata from the project's pyproject.toml. Sometimes a benchmark will have one or more variants that run using the same script. Variants like this are supported by **pyperformance** without requiring much extra effort. ### Benchmark Directory Structure Normally a benchmark is structured like this: ``` bm_NAME/ data/ # if needed requirements.txt # lock file, if any pyproject.toml run_benchmark.py ``` (Note the "bm\_" prefix on the directory name.) "pyproject.toml" holds the metadata. "run\_benchmark.py" holds the actual benchmark code. Both are necessary. **pyperformance** treats the metadata file as the fundamental source of information about a benchmark. A manifest for a set of benchmarks is effectively a mapping of names to metadata files. So a metadata file is essential. It can be located anywhere on disk. However, if it isn't located in the structure described above then the metadata must identify where to find the other files. Other than that, only a benchmark script (e.g. "run\_benchmark.py" above) is required. All other files are optional. When a benchmark has variants, each has its own metadata file next to the normal "pyproject.toml", named "bm\_NAME.toml". (Note the "bm\_" prefix.) The format of variant metadata files is exactly the same. **pyperformance** treats them the same, except that the sibling "pyproject.toml" is inherited by default. ### Manifest Files A manifest file identifies a set of benchmarks, as well as (optionally) how they should be grouped. **pyperformance** uses the manifest to determine which benchmarks are available to run (and thus which to run by default). A manifest normally looks like this: ``` [benchmarks] name metafile bench1 somedir/bm_bench1/pyproject.toml bench2 somedir/pyproject.toml bench3 ../anotherdir ``` The "benchmarks" section is a table with rows of tab-separated-values. The "name" value is how **pyperformance** will identify the benchmark. The "metafile" value is where **pyperformance** will look for the benchmark's metadata. If a metafile is a directory then it looks for "pyproject.toml" in that directory. ### Benchmark Groups The other sections in the manifest file relate to grouping: ``` [benchmarks] name metafile bench1 somedir/bm_bench1 bench2 somedir/bm_bench2 bench3 anotherdir/mybench.toml [groups] tag1 tag2 [group default] bench2 bench3 [group tricky] bench2 ``` The "groups" section specifies available groups that may be identified by benchmark tags (see about tags in the metadata section below). Any other group sections in the manifest are automatically added to the list of available groups. If no "default" group is specified then one is automatically added with all benchmarks from the "benchmarks" section in it. If there is no "groups" section and no individual group sections (other than "default") then the set of all tags of the known benchmarks is treated as "groups". A group named "all" as also automatically added which has all known benchmarks in it. Benchmarks can be excluded from a group by using a **-** (minus) prefix. Any benchmark alraedy in the list (at that point) that matches will be dropped from the list. If the first entry in the section is an exclusion then all known benchmarks are first added to the list before the exclusion is applied. For example: ``` [benchmarks] name metafile bench1 somedir/bm_bench1 bench2 somedir/bm_bench2 bench3 anotherdir/mybench.toml [group default] -bench1 ``` This means by default only "bench2" and "bench3" are run. ### Merging Manifests To combine manifests, use the **\[includes\]** section in the manifest: ``` [includes] project1/benchmarks/MANIFEST project2/benchmarks/MANIFEST ``` Note that **\** is the same as including the manifest file for the default pyperformance benchmarks. ### A Local Benchmark Suite Often a project will have more than one benchmark that it will treat as a suite. **pyperformance** handles this without any extra work. In the dirctory holding the manifest file put all the benchmarks. Then put **\** in the "metafile" column, like this: ``` [benchmarks] name metafile bench1 bench2 bench3 bench4 bench5 ``` It will look for **DIR/bm\_NAME/pyproject.toml**. If there are also variants, identify the main benchmark in the "metafile" value, like this: ``` [benchmarks] name metafile bench1 bench2 bench3 variant1 variant2 ``` **pyperformance** will look for **DIR/bm\_BASE/bm\_NAME.toml**, where "BASE" is the part after "local:". ### A Project's Benchmark Suite A Python project can identify its benchmark suite by putting the path to the manifest file in the project's top-level pyproject.toml. Additional manifests can be identified as well: ``` [tool.pyperformance] manifest = "..." manifests = ["...", "..."] ``` (Reminder: that is the pyproject.toml, not the manifest file.) ### Benchmark Metadata Files A benchmark's metadata file (usually pyproject.toml) follows the format specified in PEP 621 \ and PEP 518 \. So there are two supported sections in the file: "project" and "tool.pyperformance". A typical metadata file will look something like this: ``` [project] version = "0.9.1" dependencies = ["pyperf"] dynamic = ["name"] [tool.pyperformance] name = "my_benchmark" ``` A highly detailed one might look like this: ``` [project] name = "pyperformance_bm_json_dumps" version = "0.9.1" description = "A benchmark for json.dumps()" requires-python = ">=3.8" dependencies = ["pyperf"] urls = {repository = "https://github.com/python/pyperformance"} dynamic = ["version"] [tool.pyperformance] name = "json_dumps" tags = "serialize" runscript = "bench.py" datadir = ".data-files/extras" extra_opts = ["--special"] ``` ### Inheritance For one benchmark to inherit from another (or from common metadata), the "inherits" field is available: ``` [project] dependencies = ["pyperf"] dynamic = ["name", "version"] [tool.pyperformance] name = "my_benchmark" inherits = "../common.toml" ``` All values in either section of the inherited metadata are treated as defaults, on top of which the current metadata is applied. In the above example, for instance, a value for "version" in common.toml would be used here. If the "inherits" value is a directory (even for "..") then "base.toml" in that directory will be inherited. For variants, the base pyproject.toml is the default value for "inherits". ### Inferred Values In some situations, omitted values will be inferred from other available data (even for required fields). - • **project.name** \<= **tool.pyperformance.name** - • **project.\*** \<= inherited metadata (except for "name" and "dynamic") - • **tool.pyperformance.name** \<= metadata filename - • **tool.pyperformance.\*** \<= inherited metadata (except for "name" and "inherits") When the name is inferred from the filename for a regularly structured benchmark, the "bm\_" prefix is removed from the benchmark's directory. If it is a variant that prefix is removed from the metadata filename, as well as the .toml suffix. ### The \[project\] Section | | | | | | | | --- | --- | --- | --- | --- | --- | | field | type | R | T | B | D | | project.name | str | X | X | | | | project.version | ver | X | | X | X | | project.dependencies | \[str\] | | | X | | | project.dynamic | \[str\] | | | | | "R": required "T": inferred from the tool section "B": inferred from the inherited metadata "D": for default benchmarks, inferred from pyperformance "dynamic" is required by PEP 621 for when a field will be filled in dynamically by the tool. This is especially important for required fields. All other PEP 621 fields are optional (e.g. **requires-python = "\>=3.8"**, **{repository = "https://github.com/..."}**). ### The \[tool.pyperformance\] Section | | | | | | | --- | --- | --- | --- | --- | | field | type | R | B | F | | tool.name | str | X | | X | | tool.tags | \[str\] | | X | | | tool.extra\_opts | \[str\] | | X | | | tool.inherits | file | | | | | tool.runscript | file | | X | | | tool.datadir | file | | X | | "R": required "B": inferred from the inherited metadata "F": inferred from filename - • tags: optional list of names to group benchmarks - • extra\_opts: optional list of args to pass to **tool.runscript** - • runscript: the benchmark script to use instead of run\_benchmark.py. ## CPYTHON RESULTS, 2017 This page lists benchmarks which became faster in CPython. ### Optimizations ### 2016-12-14: speedup method calls Optimization: Speedup method calls 1.2x \, commit f2392133 \. | | | | | --- | --- | --- | | Benchmark | 2016-12-01 (27580c1fb5e8) | 2017-01-01 (67e1aa0b58be) | | call\_method | 14.1 ms | 11.2 ms: 1.26x faster (-21%) | | call\_method\_slots | 13.9 ms | 11.1 ms: 1.25x faster (-20%) | | call\_method\_unknown | 16.0 ms | 14.3 ms: 1.12x faster (-11%) | ### 2016-04-22: pymalloc allocator Optimization: PyMem\_Malloc() now uses the fast pymalloc allocator \, commit f5c4b990 \. Changes of at least 5%: | | | | | --- | --- | --- | | Benchmark | 2016-04-21 (5439fc4901db) | 2016-04-22 (f5c4b99034fa) | | unpickle\_list | 10.4 us | 7.64 us: 1.36x faster (-27%) | | json\_dumps | 28.0 ms | 25.2 ms: 1.11x faster (-10%) | | unpickle\_pure\_python | 741 us | 678 us: 1.09x faster (-9%) | | unpickle | 33.9 us | 31.3 us: 1.08x faster (-8%) | | meteor\_contest | 197 ms | 183 ms: 1.08x faster (-7%) | | mako | 36.9 ms | 34.3 ms: 1.07x faster (-7%) | | pathlib | 41.0 ms | 38.4 ms: 1.07x faster (-6%) | | call\_method\_slots | 14.8 ms | 13.9 ms: 1.07x faster (-6%) | | telco | 19.5 ms | 18.3 ms: 1.07x faster (-6%) | | scimark\_lu | 413 ms | 388 ms: 1.07x faster (-6%) | | nqueens | 221 ms | 207 ms: 1.07x faster (-6%) | | fannkuch | 937 ms | 882 ms: 1.06x faster (-6%) | | regex\_compile | 319 ms | 301 ms: 1.06x faster (-6%) | | raytrace | 1.16 sec | 1.09 sec: 1.06x faster (-5%) | | pickle\_pure\_python | 1.11 ms | 1.05 ms: 1.05x faster (-5%) | | genshi\_text | 70.1 ms | 66.6 ms: 1.05x faster (-5%) | ### 2015-12-07: Optimize ElementTree.iterparse(), xml\_etree\_iterparse Optimization: Issue #25638: Optimized ElementTree.iterparse(); it is now 2x faster \, commit 9ec5e25f2 \. | | | | | --- | --- | --- | | Benchmark | 2015-12-01 (df144092a340) | 2016-01-01 (71db90356390) | | xml\_etree\_iterparse | 423 ms | 206 ms: 2.05x faster (-51%) | ### 2015-09-19: PGO uses test suite, pidigits Optimization: Issue #24915: Add Clang support to PGO builds and use the test suite for profile data \, commit 7188a3ef \. Changes of at least 5%: | | | | | --- | --- | --- | | Benchmark | 2015-09-18 (4b363e270108) | 2015-09-18\_22-13 (7188a3efe07b) | | pickle | 33.7 us | 26.4 us: 1.28x faster (-22%) | | pidigits | 332 ms | 286 ms: 1.16x faster (-14%) | | pickle\_list | 9.90 us | 8.84 us: 1.12x faster (-11%) | | unpickle | 37.2 us | 33.3 us: 1.12x faster (-11%) | | unpickle\_list | 11.1 us | 9.95 us: 1.11x faster (-10%) | | regex\_dna | 330 ms | 297 ms: 1.11x faster (-10%) | | regex\_effbot | 6.43 ms | 5.80 ms: 1.11x faster (-10%) | | pickle\_dict | 69.3 us | 64.1 us: 1.08x faster (-8%) | | mako | 39.1 ms | 36.2 ms: 1.08x faster (-7%) | | call\_simple | 12.2 ms | 11.6 ms: 1.05x faster (-5%) | | genshi\_xml | 175 ms | 166 ms: 1.05x faster (-5%) | Changes of at least 5%, sadly two benchmarks also became slower: | | | | | --- | --- | --- | | Benchmark | 2015-09-18\_14-32-master-4b363e270108 | 2015-09-18\_22-13-master-7188a3efe07b | | unpickle\_pure\_python | 776 us | 821 us: 1.06x slower (+6%) | | regex\_v8 | 49.5 ms | 52.6 ms: 1.06x slower (+6%) | ### 2015-05-30: C implementation of collections.OrderedDict, html5lib Optimization: Issue #16991: Add a C implementation of collections.OrderedDict \, commit 96c6af9b \. | | | | | --- | --- | --- | | Benchmark | 2015-05-02 (3b4d30a27bd6) | 2015-06-01 (41874c570cf3) | | html5lib | 285 ms | 233 ms: 1.23x faster (-19%) | ### 2015-05-23: C implementation of functools.lru\_cache(), sympy Optimization: Issue #14373: Added C implementation of functools.lru\_cache() \, commit 1c858c35 \. Changes of at least 5%: | | | | | --- | --- | --- | | Benchmark | 2015-05-23\_19-15-master-c70908558d8e | 2015-05-23\_19-42-master-1c858c352b8c | | sympy\_expand | 1.45 sec | 1.14 sec: 1.27x faster (-21%) | | sympy\_sum | 308 ms | 247 ms: 1.25x faster (-20%) | | sympy\_str | 621 ms | 500 ms: 1.24x faster (-19%) | | sympy\_integrate | 54.2 ms | 45.7 ms: 1.19x faster (-16%) | | scimark\_lu | 497 ms | 471 ms: 1.06x faster (-5%) | **pickle\_dict** is seen as 1.06x slower, but since pickle doesn't use functools.lru\_cache(), the change is ignored in the table. ### Slowdown ### 2016-09-11: regex\_compile Slowdown: convert re flags to (much friendlier) IntFlag constants (issue #28082) \, commit f93395bc \. | | | | | | --- | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2016-07-01 (355048970b2a) | 2016-10-01 (78a111c7d867) | | regex\_compile | 339 ms | 309 ms: 1.10x faster (-9%) | 383 ms: 1.13x slower (+13%) | ### Timeline ### April, 2016 -\> May, 2016 2016-04-01 .. 2016-05-01: | | | | | --- | --- | --- | | Benchmark | 2016-04-01 (dcfebb32e277) | 2016-05-01 (f1e2671fdf88) | | nqueens | 255 ms | 207 ms: 1.23x faster (-19%) | | raytrace | 1.31 sec | 1.09 sec: 1.19x faster (-16%) | | float | 290 ms | 243 ms: 1.19x faster (-16%) | | chaos | 273 ms | 235 ms: 1.16x faster (-14%) | | hexiom | 21.0 ms | 18.6 ms: 1.13x faster (-11%) | | deltablue | 16.4 ms | 14.6 ms: 1.12x faster (-11%) | | go | 557 ms | 502 ms: 1.11x faster (-10%) | | nbody | 254 ms | 232 ms: 1.10x faster (-9%) | ### call\_method Timeline 2016-04-01 .. 2017-01-01: | | | | | | | --- | --- | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2016-07-01 (355048970b2a) | 2016-10-01 (78a111c7d867) | 2017-01-01 (67e1aa0b58be) | | call\_method | 15.8 ms | 14.9 ms: 1.06x faster (-6%) | 14.1 ms: 1.13x faster (-11%) | 11.2 ms: 1.42x faster (-29%) | | call\_method\_slots | 15.7 ms | 15.2 ms: 1.03x faster (-3%) | 14.0 ms: 1.13x faster (-11%) | 11.1 ms: 1.42x faster (-29%) | | call\_method\_unknown | 17.7 ms | 15.9 ms: 1.11x faster (-10%) | 15.6 ms: 1.13x faster (-11%) | 14.3 ms: 1.23x faster (-19%) | ### crypto\_pyaes | | | | | --- | --- | --- | | Benchmark | 2016-04-01 (master) | 2016-05-01 (master) | | crypto\_pyaes | 226 ms | 205 ms: 1.10x faster (-9%) | 2016-03-01 .. 2016-06-01: | | | | | --- | --- | --- | | Benchmark | 2016-03-01 (13d09afff127) | 2016-06-01 (d80ab7d94578) | | crypto\_pyaes | 231 ms | 199 ms: 1.16x faster (-14%) | ### json\_loads Progress on 21 months, 2015-01-01 .. 2016-10-01: | | | | | --- | --- | --- | | Benchmark | 2015-01-01 (52074ac866eb) | 2016-10-01 (78a111c7d867) | | json\_loads | 64.0 us | 56.6 us: 1.13x faster (-11%) | ### logging\_silent | | | | | --- | --- | --- | | Benchmark | 2016-01-01 (899b72cee21c) | 2016-07-01 (355048970b2a) | | logging\_silent | 718 ns | 606 ns: 1.18x faster (-16%) | ### pickle pickle, 2016-08-02 .. 2016-09-08: | | | | | --- | --- | --- | | Benchmark | 2016-08-02 (133138a284be) | 2016-09-08 (10427f44852b) | | pickle | 25.5 us | 21.4 us: 1.19x faster (-16%) | pickle dict/list: | | | | | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2016-10-01 (78a111c7d867) | | pickle\_dict | 64.5 us | 57.7 us: 1.12x faster (-11%) | | pickle\_list | 9.06 us | 7.79 us: 1.16x faster (-14%) | unpickle: | | | | | --- | --- | --- | | Benchmark | 2015-07-01 (d7982beca93c) | 2015-10-01 (30b7138fe12b) | | unpickle | 36.9 us | 32.8 us: 1.13x faster (-11%) | ### python\_startup 2015-04-01 .. 2015-10-01: | | | | | --- | --- | --- | | Benchmark | 2015-04-01 (4fd929b43121) | 2015-10-01 (30b7138fe12b) | | python\_startup | 16.4 ms | 17.2 ms: 1.05x slower (+5%) | | python\_startup\_no\_site | 8.65 ms | 8.90 ms: 1.03x slower (+3%) | 2016-04-01 .. 2017-01-01: | | | | | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2017-01-01 (67e1aa0b58be) | | python\_startup | 17.3 ms | 14.5 ms: 1.20x faster (-16%) | | python\_startup\_no\_site | 8.89 ms | 8.39 ms: 1.06x faster (-6%) | ### regex\_compile | | | | | | --- | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2016-07-01 (355048970b2a) | 2016-10-01 (78a111c7d867) | | regex\_compile | 339 ms | 309 ms: 1.10x faster (-9%) | 383 ms: 1.13x slower (+13%) | ### telco | | | | | | | | --- | --- | --- | --- | --- | --- | | Benchmark | 2016-01-01 (899b72cee21c) | 2016-04-01 (6b6abd4cf10e) | 2016-07-01 (355048970b2a) | 2016-10-01 (78a111c7d867) | 2017-03-31 (cdcac039fb44) | | telco | 19.6 ms | 19.2 ms: 1.02x faster (-2%) | 18.3 ms: 1.08x faster (-7%) | 15.1 ms: 1.30x faster (-23%) | 13.9 ms: 1.41x faster (-29%) | ### scimark 2016-10-01 .. 2017-03-31: | | | | | | --- | --- | --- | --- | | Benchmark | 2016-10-01 (78a111c7d867) | 2017-01-01 (67e1aa0b58be) | 2017-03-31 (cdcac039fb44) | | scimark\_lu | 423 ms | 378 ms: 1.12x faster (-11%) | 318 ms: 1.33x faster (-25%) | | scimark\_sor | 426 ms | 403 ms: 1.06x faster (-5%) | 375 ms: 1.14x faster (-12%) | ### sqlalchemy\_declarative | | | | | --- | --- | --- | | Benchmark | 2014-10-01 (5a789f7eaf81) | 2015-10-01 (30b7138fe12b) | | sqlalchemy\_declarative | 345 ms | 301 ms: 1.15x faster (-13%) | ### sympy 2016-04-01 .. 2016-10-01: | | | | | | --- | --- | --- | --- | | Benchmark | 2016-04-01 (6b6abd4cf10e) | 2016-07-01 (355048970b2a) | 2016-10-01 (78a111c7d867) | | sympy\_expand | 1.10 sec | 1.01 sec: 1.09x faster (-8%) | 942 ms: 1.17x faster (-14%) | | sympy\_integrate | 46.6 ms | 42.9 ms: 1.09x faster (-8%) | 41.2 ms: 1.13x faster (-11%) | | sympy\_sum | 247 ms | 233 ms: 1.06x faster (-6%) | 199 ms: 1.24x faster (-19%) | | sympy\_str | 483 ms | 454 ms: 1.07x faster (-6%) | 427 ms: 1.13x faster (-12%) | ### xml\_etree\_generate | | | | | | | | --- | --- | --- | --- | --- | --- | | Benchmark | 2015-04-01 (4fd929b43121) | 2015-07-01 (d7982beca93c) | 2015-10-01 (30b7138fe12b) | 2016-01-01 (899b72cee21c) | 2016-07-01 (355048970b2a) | | xml\_etree\_generate | 282 ms | 267 ms: 1.06x faster (-5%) | 256 ms: 1.10x faster (-9%) | 237 ms: 1.19x faster (-16%) | 212 ms: 1.33x faster (-25%) | ## CHANGELOG ### Version 1.14.0 - • Bump **pyperf** to 2.10.0 - • Add base64 module benchmark (b64, b32, b16, a85, b85) - • Add FastAPI HTTP benchmark - • Add YAML parsing benchmark - • Respect rigorous setting in benchmark configuration files ### Version 1.13.0 (2025-10-27) - • Re-enable xdsl benchmark - • Remove code for Python 3.7 and previous versions - • CI updates: \* fix workflow ubuntu-latest \* remove unused Azure Pipelines config \* update checkout version \* setup dependabot \* update setup-python version ### Version 1.12.0 (2025-10-24) - • Add xdsl benchmark (but disabled by default due to a regression) - • Add a new quadtree nbody simulation using the Barnes Hut algorithm - • Add argparse benchmark - • Add NetworkX benchmarks - • Add Sphinx benchmark - • Add decimal module benchmark - • Add BPE tokeniser benchmark - • Skip dask benchmark on Windows for Python 3.13 - • Bump dask\[distributed\] to 2024.10.1 for Windows compatibility - • Bump greenlet to 3.2.4 for compatibility with 3.13+ - • Bump Chameleon to 4.6.0 - • Bump tornado to 6.5.0 - • Bump pyperf to 2.9.0 - • Bump sqlglot to V2 - • Bump mypy to 1.18.2 - • Fix check for editable mode - • Fix **--same-loops** handling - • Fix: pass **--warmup** and **--timeout** flags to pyperf - • Add **-V**/**--version** CLI argument - • Add **--hook** CLI argument - • Run ruff format and check under pre-commit and GitHub Actions - • Add support for experimental JIT builds - • Add Arm64 CI - • Add Free-threading CI - • Make Windows a non-experimental platform in CI - • Add support Python 3.14 and drop Python 3.7, 3.8, 3.9 ### Version 1.11.0 (2024-03-09) - • Add a --same-loops option to the run command to use the exact same number of loops as a previous run (without recalibrating). - • Bump pyperf to 2.6.3 - • Fix the django\_template benchmark for compatibilty with 3.13 - • Fix benchmark.conf.sample ### Version 1.10.0 (2023-10-22) - • Add benchmark for asyncio\_webockets - • Expose --min-time from pyperf to pyperformance CLI - • Bump coverage to 7.3.2 for compatibilty with 3.13 - • Bump greenlet to 3.0.0rc3 for compatibilty with 3.13 ### Version 1.0.9 (2023-06-14) - • Vendor lib2to3 for Python 3.13+ - • Add TaskGroups variants to async\_tree benchmarks ### Version 1.0.8 (2023-06-02) - • Move the main requirements.txt file to pyperformance/requirements so that dependabot can only run on that one file - • Update dependencies of benchmarks not to specify setuptools - • On older versions of Python, skip benchmarks that use features introduced in newer Python versions - • Support **--inherit-environ** when reusing a venv - • Use tomllib/tomli over toml - • Update MANIFEST.in to include cert files for asyncio\_tcp\_ssl benchmark - • Fix undefined variable issue when raising VenvPipInstallFailedError - • Add mypy config; run mypy in CI - • Fix typo of str.partition from \_pyproject\_toml.py - • Add version of Richards benchmark that uses super() - • Add a benchmark for runtime-checkable protocols - • Extend async tree benchmarks to cover eager task execution ### Version 1.0.7 (2023-04-22) - • Upgrade pyperf from 2.5.0 to 2.6.0 - • Clean unused imports and other small code details - • Migrage to the pyproject.toml based project - • Fix the django\_template benchmark due to lack of distutils - • Add benchmark for toml - • Add benchmark for comprehensions - • Add benchmark for asyncio\_tcp\_ssl - • Add benchmark for asyncio\_tcp - • Add benchmark for Dask scheduler - • Add the gc benchmarks to the MANIFEST file ### Version 1.0.6 (2022-11-20) - • Upgrade pyperf from 2.4.1 to 2.5.0 - • Add a benchmark to measure gc traversal - • Add jobs field in compile section to specify make -j param - • Add benchmark for Docutils - • Add async\_generators benchmark - • Add benchmark for IPC - • Fix Manifest Group - • Fix installing dev build of pyperformance inside compile/compile\_all - • Always upload, even when some benchmarks fail - • Add sqlglot benchmarks - • Support reporting geometric mean by tags - • Allow for specifying local wheels and sdists as dependencies - • Add a benchmark based on *python -m pprint* - • Add mdp back into the default group - • Add coroutines benchmark - • Reduce noise in generators benchmark - • Add benchmark for deepcopy - • Add coverage benchmark - • Add generators benchmark - • Add benchmark for async tree workloads - • Support relative paths to manifest files - • Add support for multiple benchmark groups in a manifest - • Fix --inherit-environ issue - • Use working Genshi 0.7.7 ### Version 1.0.4 (2022-01-25) - • Re-release support for user-defined benchmark after fixing problem with virtual environments. ### Version 1.0.3 (2021-12-20) - • Support user-defined benchmark suites. ### Version 1.0.2 (2021-05-11) - • Disable the henshi benchmark temporarily since is no longer compatible with Python 3.11. - • Reenable html5lib benchmark: html5lib 1.1 has been released. - • Update requirements. - • Replace Travis CI with GitHub Actions. - • The development branch **master** was renamed to **main**. See \ for the rationale. ### Version 1.0.1 (2020-03-26) - • Drop usage of the six module since Python 2 is no longer supported. Remove Python 2 specific code. - • Update dependencies: - • django: 3.0 =\> 3.0.4 - • dulwich: 0.19.14 =\> 0.19.15 - • mako: 1.1.0 = \> 1.1.2 - • mercurial: 5.1.1 =\> 5.3.1 - • psutil: 5.6.7 =\> 5.7.0 - • pyperf: 1.7.0 =\> 2.0.0 - • sqlalchemy: 1.3.12 =\> 1.3.15 - • sympy: 1.5 =\> 1.5.1 - • tornado: 6.0.3 =\> 6.0.4 - • Remove six, html5lib and mercurial requirements. - • pip-tools (pip-compile) is now used to update dependencies ### Version 1.0.0 (2019-12-17) - • Enable pyflate benchmarks on Python 3. - • Remove **spambayes** benchmark: it is not compatible with Python 3. - • Remove **2n3**:benchmark group. - • Drop Python 2.7 support: old Django and Tornado versions are not compatible with incoming Python 3.9. - • Disable html5lib benchmark temporarily, since it's no longer compatible with Python 3.9. - • Update requirements: - • Django: 1.11.22 =\> 3.0 - • Mako: 1.0.14 =\> 1.1.0 - • SQLAlchemy: 1.3.6 =\> 1.3.12 - • certifi: 2019.6.16 =\> 2019.11.28 - • docutils: 0.15.1 =\> 0.15.2 - • dulwich: 0.19.11 =\> 0.19.14 - • mercurial: 5.0.2 =\> 5.1.1 - • psutil: 5.6. =\> 5.6.7 - • pyperf: 1.6.1 =\> 1.7.0 - • six: 1.12. =\> 1.13.0 - • sympy: 1.4 =\> 1.5 ### Version 0.9.1 (2019-07-29) - • Enable hg\_startup on Python 3 - • Fix compatibility with Python 3.8 beta 2 - • Update requirements: - • certifi: 2019.3.9 =\> 2019.6.16 - • Chameleon: 3.6.1 =\> 3.6.2 - • Django: 1.11.20 =\> 1.11.22 - • docutils: 0.14 =\> 0.15.1.post1 - • Mako: 1.0.10 =\> 1.0.14 - • mercurial: 5.0 =\> 5.0.2 - • pathlib2: 2.3.3 =\> 2.3.4 - • psutil: 5.6.2 =\> 5.6.3 - • SQLAlchemy: 1.3.4 =\> 1.3.6 ### Version 0.9.0 (2019-05-29) - • Project renamed from "performance" to "pyperformance" - • Upgrade pyperf from version 1.6.0 to 1.6.1. The project has been renamed from "perf" to "pyperf". Update imports. - • Issue #54: Update Genshi to 0.7.3. It is now compatible with Python 3.8. - • Update requirements: - • Mako: 1.0.9= \> 1.0.10 - • SQLAlchemy: 1.3.3 =\> 1.3.4 ### Version 0.8.0 (2019-05-10) - • compile command: Add "pkg\_only" option to benchmark.conf. Add support for native libraries that are installed but not on path. Patch by Robert Grimm. - • Update Travis configuration: use trusty image, use pip cache. Patch by Inada Naoki. - • Upgrade tornado to 5.1.1. Patch by Inada Naoki. - • Fix compile command on Mac OS: no program extension. Patch by Anthony Shaw. - • Update requirements: - • Chameleon: 3.4 =\> 3.6.1 - • Django: 1.11.16 =\> 1.11.20 - • Genshi: 0.7.1 =\> 0.7.2 - • Mako: 1.0.7 =\> 1.0.9 - • MarkupSafe: 1.0 =\> 1.1.1 - • SQLAlchemy: 1.2.12 =\> 1.3.3 - • certifi: 2018.10.15 =\> 2019.3.9 - • dulwich: 0.19.6 =\> 0.19.11 - • mercurial: 4.7.2 =\> 5.0 - • mpmath: 1.0.0 =\> 1.1.0 - • pathlib2: 2.3.2 =\> 2.3.3 - • perf: 1.5.1 =\> 1.6.0 - • psutil: 5.4.7 =\> 5.6.2 - • six: 1.11.0 =\> 1.12.0 - • sympy: 1.3 =\> 1.4 - • tornado: 4.5.3 =\> 5.1.1 ### Version 0.7.0 (2018-10-16) - • python\_startup: Add **--exit** option. - • Update requirements: - • certifi: 2017.11.5 =\> 2018.10.15 - • Chameleon: 3.2 =\> 3.4 - • Django: 1.11.9 =\> 1.11.16 - • dulwich: 0.18.6 =\> 0.19.6 - • Genshi: 0.7 =\> 0.7.1 - • mercurial: 4.4.2 =\> 4.7.2 - • pathlib2: 2.3.0 =\> 2.3.2 - • psutil: 5.4.3 =\> 5.4.7 - • SQLAlchemy: 1.2.0 =\> 1.2.12 - • sympy: 1.1.1 =\> 1.3 - • Fix issue #40 for pip 10 and newer: Remove indirect dependencies. Indirect dependencies were used to install cffi, but Mercurial 4.0 doesn't depend on cffi anymore. ### Version 0.6.1 (2018-01-11) - • Fix inherit-environ: propagate to recursive invocations of **performance** in **compile** and **compile\_all** commands. - • Fix the **--track-memory** option thanks to the update to perf 1.5. - • Update requirements - • certifi: 2017.4.17 =\> 2017.11.5 - • Chameleon: 3.1 =\> 3.2 - • Django: 1.11.3 =\> 1.11.9 - • docutils: 0.13.1 =\> 0.14 - • dulwich: 0.17.3 =\> 0.18.6 - • html5lib: 0.999999999 =\> 1.0.1 - • Mako: 1.0.6 =\> 1.0.7 - • mercurial: 4.2.2 =\> 4.4.2 - • mpmath: 0.19 =\> 1.0.0 - • perf: 1.4 =\> 1.5.1 (fix **--track-memory** option) - • psutil: 5.2.2 =\> 5.4.3 - • pyaes: 1.6.0 =\> 1.6.1 - • six: 1.10.0 =\> 1.11.0 - • SQLAlchemy: 1.1.11 =\> 1.2.0 - • sympy: 1.0 =\> 1.1.1 - • tornado: 4.5.1 =\> 4.5.3 ### Version 0.6.0 (2017-07-06) - • Change **warn** to **warning** in *bm\_logging.py*. In Python 3, Logger.warn() calls warnings.warn() to log a deprecation warning, so is slower than Logger.warning(). - • Add again the **logging\_silent** microbenchmark suite. - • compile command: update the Git repository before getting the revision - • Update requirements - • perf: 1.3 =\> 1.4 (fix parse\_cpu\_list(): strip also NUL characters) - • Django: 1.11.1 =\> 1.11.3 - • mercurial: 4.2 =\> 4.2.2 - • pathlib2: 2.2.1 =\> 2.3.0 - • SQLAlchemy: 1.1.10 =\> 1.1.11 ### Version 0.5.5 (2017-05-29) - • On the 2.x branch on CPython, **compile** now pass **--enable-unicode=ucs4** to the **configure** script on all platforms, except on Windows which uses UTF-16 because of its 16-bit wchar\_t. - • The **float** benchmark now uses **\_\_slots\_\_** on the **Point** class. - • Remove the following microbenchmarks. They have been moved to the pymicrobench \ project because they are too short, not representative of real applications and are too unstable. - • **pybench** microbenchmark suite - • **call\_simple** - • **call\_method** - • **call\_method\_unknown** - • **call\_method\_slots** - • **logging\_silent**: values are faster than 1 ns on PyPy with 2^27 loops! (and around 0.7 us on CPython) - • Update requirements - • Django: 1.11 =\> 1.11.1 - • SQLAlchemy: 1.1.9 =\> 1.1.10 - • certifi: 2017.1.23 =\> 2017.4.17 - • perf: 1.2 =\> 1.3 - • mercurial: 4.1.2 =\> 4.2 - • tornado: 4.4.3 =\> 4.5.1 ### Version 0.5.4 (2017-04-10) - • Create a new documentation at: \ - • Add "CPython results, 2017" to the doc: significant performance changes, significant optimizations, timeline, etc. - • The **show** command doesn't need to create a virtual env anymore. - • Add new commands: - • **pyperformance compile**: compile, install and benchmark - • **pyperformance compile\_all**: benchmark multiple branches and revisions of Python - • **pyperformance upload**: upload a JSON file to a Codespeed - • setup.py: add dependencies to **perf** and **six** modules. - • bm\_xml\_etree now uses "\_pure\_python" in benchmark names if the accelerator is explicitly disabled. - • Upgrade requirements: - • Django: 1.10.6 -\> 1.11 - • SQLAlchemy: 1.1.6 -\> 1.1.9 - • mercurial: 4.1.1 -\> 4.1.2 - • perf: 1.1 =\> 1.2 - • psutil: 5.2.1 -\> 5.2.2 - • tornado: 4.4.2 -\> 4.4.3 - • webencodings: 0.5 -\> 0.5.1 - • perf 1.2 now calibrates the number of warmups on PyPy. - • On Python 3.5a0: force pip 7.1.2 and setuptools 18.5: \ ### Version 0.5.3 (2017-03-27) - • Upgrade Dulwich to 0.17.3 to support PyPy older than 5.6: see \ - • Fix ResourceWarning warnings: close explicitly files and sockets. - • scripts: replace Mercurial commands with Git commands. - • Upgrade requirements: - • dulwich: 0.17.1 =\> 0.17.3 - • perf: 1.0 =\> 1.1 - • psutil: 5.2.0 =\> 5.2.1 ### Version 0.5.2 (2017-03-17) - • Upgrade requirements: - • certifi: 2016.9.26 =\> 2017.1.23 - • Chameleon: 3.0 =\> 3.1 - • Django: 1.10.5 =\> 1.10.6 - • MarkupSafe: 0.23 =\> 1.0 - • dulwich: 0.16.3 =\> 0.17.1 - • mercurial: 4.0.2 =\> 4.1.1 - • pathlib2: 2.2.0 =\> 2.2.1 - • perf: 0.9.3 =\> 1.0 - • psutil: 5.0.1 =\> 5.2.0 - • SQLAlchemy: 1.1.4 =\> 1.1.6 ### Version 0.5.1 (2017-01-16) - • Fix Windows support (upgrade perf from 0.9.0 to 0.9.3) - • Upgrade requirements: - • Chameleon: 2.25 =\> 3.0 - • Django: 1.10.3 =\> 1.10.5 - • docutils: 0.12 =\> 0.13.1 - • dulwich: 0.15.0 =\> 0.16.3 - • mercurial: 4.0.0 =\> 4.0.2 - • perf: 0.9.0 =\> 0.9.3 - • psutil: 5.0.0 =\> 5.0.1 ### Version 0.5.0 (2016-11-16) - • Add **mdp** benchmark: battle with damages and topological sorting of nodes in a graph - • The **default** benchmark group now include all benchmarks but **pybench** - • If a benchmark fails, log an error, continue to execute following benchmarks, but exit with error code 1. - • Remove deprecated benchmarks: **threading\_threaded\_count** and **threading\_iterative\_count**. It wasn't possible to run them anyway. - • **dulwich** requirement is now optional since its installation fails on Windows. - • Upgrade requirements: - • Mako: 1.0.5 =\> 1.0.6 - • Mercurial: 3.9.2 =\> 4.0.0 - • SQLAlchemy: 1.1.3 =\> 1.1.4 - • backports-abc: 0.4 =\> 0.5 ### Version 0.4.0 (2016-11-07) - • Add **sqlalchemy\_imperative** benchmark: it wasn't registered properly - • The **list** command now only lists the benchmark that the **run** command will run. The **list** command gets a new **-b/--benchmarks** option. - • Rewrite the code creating the virtual environment to test correctly pip. Download and run **get-pip.py** if pip installation failed. - • Upgrade requirements: - • perf: 0.8.2 =\> 0.9.0 - • Django: 1.10.2 =\> 1.10.3 - • Mako: 1.0.4 =\> 1.0.5 - • psutil: 4.3.1 =\> 5.0.0 - • SQLAlchemy: 1.1.2 =\> 1.1.3 - • Remove **virtualenv** dependency ### Version 0.3.2 (2016-10-19) - • Fix setup.py: include also **performance/benchmarks/data/asyncio.git/** ### Version 0.3.1 (2016-10-19) - • Add **regex\_dna** benchmark - • The **run** command now fails with an error if no benchmark was run. - • genshi, logging, scimark, sympy and xml\_etree scripts now run all sub-benchmarks by default - • Rewrite pybench using perf: remove the old legacy code to calibrate and run benchmarks, reuse perf.Runner API. - • Change heuristic to create the virtual environment, tried commands: - • **python -m venv** - • **python -m virtualenv** - • **virtualenv -p python** - • The creation of the virtual environment now ensures that pip works to detect "python3 -m venv" which doesn't install pip. - • Upgrade perf dependency from 0.7.12 to 0.8.2: update all benchmarks to the new perf 0.8 API (which introduces incompatible changes) - • Update SQLAlchemy from 1.1.1 to 1.1.2 ### Version 0.3.0 (2016-10-11) New benchmarks: - • Add **crypto\_pyaes**: Benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the pyaes module (version 1.6.0). Add **pyaes** dependency. - • Add **sympy**: Benchmark on SymPy. Add **scipy** dependency. - • Add **scimark** benchmark - • Add **deltablue**: DeltaBlue benchmark - • Add **dulwich\_log**: Iterate on commits of the asyncio Git repository using the Dulwich module. Add **dulwich** (and **mpmath**) dependencies. - • Add **pyflate**: Pyflate benchmark, tar/bzip2 decompressor in pure Python - • Add **sqlite\_synth** benchmark: Benchmark Python aggregate for SQLite - • Add **genshi** benchmark: Render template to XML or plain text using the Genshi module. Add **Genshi** dependency. - • Add **sqlalchemy\_declarative** and **sqlalchemy\_imperative** benchmarks: SQLAlchemy Declarative and Imperative benchmarks using SQLite. Add **SQLAlchemy** dependency. Enhancements: - • **compare** command now fails if the performance versions are different - • **nbody**: add **--reference** and **--iterations** command line options. - • **chaos**: add **--width**, **--height**, **--thickness**, **--filename** and **--rng-seed** command line options - • **django\_template**: add **--table-size** command line option - • **json\_dumps**: add **--cases** command line option - • **pidigits**: add **--digits** command line option - • **raytrace**: add **--width**, **--height** and **--filename** command line options - • Port **html5lib** benchmark to Python 3 - • Enable **pickle\_pure\_python** and **unpickle\_pure\_python** on Python 3 (code was already compatible with Python 3) - • Creating the virtual environment doesn't inherit environment variables (especially **PYTHONPATH**) by default anymore: **--inherit-environ** command line option must now be used explicitly. Bugfixes: - • **chaos** benchmark now also reset the **random** module at each sample to get more reproductible benchmark results - • Logging benchmarks now truncate the in-memory stream before each benchmark run Rename benchmarks: - • Rename benchmarks to get a consistent name between the command line and benchmark name in the JSON file. - • Rename pickle benchmarks: - • **slowpickle** becomes **pickle\_pure\_python** - • **slowunpickle** becomes **unpickle\_pure\_python** - • **fastpickle** becomes **pickle** - • **fastunpickle** becomes **unpickle** - • Rename ElementTree benchmarks: replace **etree\_** prefix with **xml\_etree\_**. - • Rename **hexiom2** to **hexiom\_level25** and explicitly pass **--level=25** parameter - • Rename **json\_load** to **json\_loads** - • Rename **json\_dump\_v2** to **json\_dumps** (and remove the deprecated **json\_dump** benchmark) - • Rename **normal\_startup** to **python\_startup**, and **startup\_nosite** to **python\_startup\_no\_site** - • Rename **threaded\_count** to **threading\_threaded\_count**, rename **iterative\_count** to **threading\_iterative\_count** - • Rename logging benchmarks: - • **silent\_logging** to **logging\_silent** - • **simple\_logging** to **logging\_simple** - • **formatted\_logging** to **logging\_format** Minor changes: - • Update dependencies - • Remove broken **--args** command line option. ### Version 0.2.2 (2016-09-19) - • Add a new **show** command to display a benchmark file - • Issue #11: Display Python version in compare. Display also the performance version. - • CPython issue #26383; csv output: don't truncate digits for timings shorter than 1 us - • compare: Use sample unit of benchmarks, format values in the table output using the unit - • compare: Fix the table output if benchmarks only contain a single sample - • Remove unused -C/--control\_label and -E/--experiment\_label options - • Update perf dependency to 0.7.11 to get Benchmark.get\_unit() and BenchmarkSuite.get\_metadata() ### Version 0.2.1 (2016-09-10) - • Add **--csv** option to the **compare** command - • Fix **compare -O table** output format - • Freeze indirect dependencies in requirements.txt - • **run**: add **--track-memory** option to track the memory peak usage - • Update perf dependency to 0.7.8 to support memory tracking and the new **--inherit-environ** command line option - • If **virtualenv** command fail, try another command to create the virtual environment: catch **virtualenv** error - • The first command to upgrade pip to version **\>= 6.0** now uses the **pip** binary rather than **python -m pip** to support pip 1.0 which doesn't support **python -m pip** CLI. - • Update Django (1.10.1), Mercurial (3.9.1) and psutil (4.3.1) - • Rename **--inherit\_env** command line option to **--inherit-environ** and fix it ### Version 0.2 (2016-09-01) - • Update Django dependency to 1.10 - • Update Chameleon dependency to 2.24 - • Add the **--venv** command line option - • Convert Python startup, Mercurial startup and 2to3 benchmarks to perf scripts (bm\_startup.py, bm\_hg\_startup.py and bm\_2to3.py) - • Pass the **--affinity** option to perf scripts rather than using the **taskset** command - • Put more installer and optional requirements into **performance/requirements.txt** - • Cached **.pyc** files are no more removed before running a benchmark. Use **venv recreate** command to update a virtual environment if required. - • The broken **--track\_memory** option has been removed. It will be added back when it will be fixed. - • Add performance version to metadata - • Upgrade perf dependency to 0.7.5 to get **Benchmark.update\_metadata()** ### Version 0.1.2 (2016-08-27) - • Windows is now supported - • Add a new **venv** command to show, create, recrete or remove the virtual environment. - • Fix pybench benchmark (update to perf 0.7.4 API) - • performance now tries to install the **psutil** module on CPython for better system metrics in metadata and CPU pinning on Python 2. - • The creation of the virtual environment now also tries **virtualenv** and **venv** Python modules, not only the virtualenv command. - • The development version of performance now installs performance with "pip install -e \" - • The GitHub project was renamed from **python/benchmarks** to **python/performance**. ### Version 0.1.1 (2016-08-24) - • Fix the creation of the virtual environment - • Rename pybenchmarks script to pyperformance - • Add -p/--python command line option - • Add \_\_main\_\_ module to be able to run: python3 -m performance ### Version 0.1 (2016-08-24) - • First release after the conversion to the perf module and move to GitHub - • Removed benchmarks - • django\_v2, django\_v3 - • rietveld - • spitfire (and psyco): Spitfire is not available on PyPI - • pystone - • gcbench - • tuple\_gc\_hell ### History Projected moved to \ in August 2016. Files reorganized, benchmarks patched to use the perf module to run benchmark in multiple processes. Project started in December 2008 by Collin Winter and Jeffrey Yasskin for the Unladen Swallow project. The project was hosted at \ until Feb 2016 Other Python Benchmarks: - • CPython: speed.python.org \ uses pyperf, pyperformance and Codespeed \ (Django web application) - • PyPy: speed.pypy.org \ uses PyPy benchmarks \ - • Pyston: pyston-perf \ and speed.pyston.org \ - • Numba benchmarks \ - • Cython: Cython Demos/benchmarks \ - • pythran: numpy-benchmarks \ See also the Python speed mailing list \ and the Python pyperf module \ (used by pyperformance). pyperformance is not tuned for PyPy yet: use the PyPy benchmarks project \ instead to measure PyPy performances. Image generated by bm\_raytrace (pure Python raytrace): \[image: Pure Python raytracer\] \[image\] ## Author Victor Stinner ## Copyright 2017, Victor Stinner