Below is a table that summarizes installation and run status of the Phoronix Test Suite CPU suite.
Some background context of how this fits and where I hope to head from here…
Where I’d like to head is comparing micro-architectural features (e.g. caches, branch predictors,…), NUMA properties, OS features/policies (e.g. interrupt pinning) and other aspects with a set of benchmarks. There is however, a bit of circular dependency here. To make good comparisons, one needs good benchmarks and to get good benchmarks one either needs to have a representative sample or at least a good understanding of the range.
I don’t necessarily have these benchmarks up front, so my initial idea is to “prime the pump” by comparing a set of semi-random benchmarks on their properties and iterate with both the analysis and eventually adding other workloads.
The Phoronix Test Suite is a reasonable starting point to look at an initial set of codes to start comparing because of a few reasons:
- It is free, with ~200 total tests including ~90 listed as “processor”
- There is a test harness to run tests, report results, etc
On the downsides, the tests are of varying quality e.g. a number won’t build or install for variety of reasons and other have missing download files. Also, it seems many of the tests are smaller in terms of run-time, execution size, etc.
However, that is part of the idea off priming the pump to start. To make this start I wanted a number of tests – not so many that it becomes unwieldy and not so few that it becomes harder to generalize. Hence, I picked the “cpu” suite, a set of 25 benchmarks banded together.
Results of the initial install + run exercise are listed below. Of the 25, I had problems getting four benchmarks to quickly run out of the box. So I’ll skip these for now. A quick diagnosis of what prevents these four from running:
- graphics-magick fails to download with missing tarball
- gcrypt code doesn’t compile on Ubuntu 17.10, diagnosis suggests some code cleanup is required
- pgbench has security checks to prevent running as root; while might workaround by running the test as a mortal I also have some of my tools requiring a root run, skip for now
- NAS parallel benchmarks require correct install-time variables for MPI. Can be done but skip for now.
Note what also seems true is some of these microbenchmarks have known compiler optimizations (e.g. john-the-ripper) or tuning (e.g. ffmpeg, stream) needed to get absolute highest scores. The values below are for the benchmark as it comes “out of the box”, with some optimizations still pending.
|pts/etqw-demo||cpu||yes||yes||41.80||required 32-bit libraries|
|pts/john-the-ripper||cpu||yes||yes||5937 blowfish, 20593667 des, 203603 MD5||Seems to run when I run it by itself, but not when run as part of the cpu suite.|
|pts/gcrypt||cpu||yes||no||--||gcrypt libgcrypt-1.4.4/tests benchmark not found; compile errors with source|
|pts/pgbench||cpu||yes||no||--||The test run did not produce a result; doesn't run as root|
|pts/npb||cpu||yes||no||--||Test quit with non-zero exit status; most likely MPI environment variables not configured for the run|
|pts/stream||cpu||yes||yes||copy 19452.82, scale 14243.04, triad 16108.24, add 16135,70|