Phoronix article – Ubuntu 18.04 vs 16.04 (2018-05-02)
On May 2nd, Phoronix published a benchmark comparison of 18.04 vs. 16.04 on multiple systems. It was intended to compare “out of the box” experiences.
As the article notes, similarities and differences between these systems include
- gcc 7.3 vs. gcc 5.4
- linux kernel 4.15 vs. kernel 4.12
- Slight differences in Spectre mitigation
- Newer Mesa, Python versions
While I don’t have the hardware or the OS versions, this post makes some notes characterizing the benchmarks to understand what is being measured as well as possible differences. As best I can tell below, benchmarks seem most sensitive to “build” and “runtime” environments. It is also an excuse to do some deeper dives on the benchmarks themselves to understand what is being measured.
NOTE: Slowly adding to analysis in the table below:
| Benchmark | Phoronix Observations | My Observations | Analysis |
|---|---|---|---|
| go-benchmark: json | Go version 1.10 vs. 1.6 seems to make a big difference. | Parallel, high On_CPU, stalls in front-end (31%) and not backend/speculation. 10 seconds elapsed. | Analysis |
| go-benchmark: garbage | Go version 1.10 vs. 1.6 seems to make a big difference. | Parallel, high On_CPU, stalls in front end (31%), 18 seconds elapsed. | Analysis |
| go-benchmark: build | Go version 1.10 vs. 1.6 causes build times to go up slightly in some cases. | Build is volatile, not necessarily always On_CPU, both front-end and backend stalls | Analysis |
| java-scimark | Similar between 16.04 and 18.04 | Code is compiled Java byte codes (javac -O) not a test of interpretive Java. Stalls vary by workload, but most in the back end. | Analysis |
| java-gradle-perf | Similar between 16.04 and 18.04 | Many small short-lived processes. On_CPU of ~40% and IPC of ~1 with front end stalls and speculative misses. | Analysis |
| botan: AES-256 | Several systems faster between 16.04 and 18.04 | Single-threaded, 100% core with largest issue backend stalls | Analysis |
| x264 | Small improvements for AMD between 16.04 and 18.04, suspect GCC changes. | On_CPU of ~85% with almost equal front end and back end stall issues. | Analysis |
| himeno | AMD degradations between 16.04 and 18.04 | On CPU of 100%, single-threaded, backend stall limited, | Analysis |
| ebizzy | Large AMD improvements from 16.04 to 18.04 | Tests kernel memory system. Interesting uops/instructions ratio on Intel. | Analysis |
| build-linux-kernel | Compile time goes up from Ubuntu 16.04 (gcc 5) to Ubuntu 18.04 (gcc 7). | Runs ~29000 processes, many short-lived. Frontend stalls limit. Fairly parallel with On_CPU of 87.5% | Analysis |
| encode-mp3 | Just slightly faster in 18.04 than 16.04. | Single-threaded, medium high-IPC, runs single process for 30 seconds. | Analysis |
| hackbench | Hackbench consistently improved from 16.04 to 18.04 | System level, multiple tasks spawned with IPC communications; tests kernel scheduler. | Analysis |
| stress-ng:socket | Socket performance somewhat lower in Ubuntu 18.04 than 16.04 | System level stress test of kernel socket interfaces. | Analysis |
| stress-ng:message passing | Mixed results, few systems including AMD showing better performance in Ubuntu 18.04 than 16.04 but also some Xeon systems showing worse performance. | System level stress test of kernel message passing interfaces. | Analysis |
| sqlite | Mostly the same with one or two systems much quicker (final times very close on all systems; initial times are what seems to vary). | Testing limited by filesystem performance with very little CPU used. Many involuntary switches and hence reaction of OS is important. | Analysis |
| fio | Significant improvements for Ryzen read and write tests from Ubuntu 16.04 to 18.04 | Testing limited by disk, very list CPU used. Many involuntary switches and hence reaction of OS is important. | Analysis |
| compilebench | Fairly similar with slight variations. | Testing limited by filesystem performance with very little CPU used. Many involuntary switches and hence reaction of OS is important. | Analysis |
| apache | Apache benchmark slightly lower on Ubuntu 18.04 than 16.04 | Driver process and 140 http backend server processes to receive requests. 25% of total time is system time and 15% is user time with rest idle. Low IPC with many frontend stalls. | Analysis |
| nginx | Apache benchmark slightly lower on Ubuntu 18.04 than 16.04 | Driver process and single-threaded backend server. Backend server seems to be independent of the test driver and hence not seen in the process tree. | Analysis |
| phpbench | Improvements going from Ubuntu 16.04 to 18.04 | Higher IPC, higher for Intel than AMD. Single threaded. | Analysis |
| pybench | Pybench performance sped up slightly on Intel systems from 16.04 to 18.04 and slowed down slightly on AMD. | Relatively high IPC, single threaded. | Analysis |
The following benchmarks are of the graphics subsystem and were not analyzed further (and they also didn’t seem to differ much between Ubuntu 16.04 and 18.04)
- openarena – graphics based game; measures frames/second
- tesseract – graphics based game; measures frames/second
- paraview – graphics based test
Overall, looks like more of a comparison of the build/execution environment which can be useful to note the sensitivities here as part of the benchmark setup.
Comments
Phoronix article – Ubuntu 18.04 vs 16.04 (2018-05-02) — No Comments
HTML tags allowed in your comment: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>