In this posting I summarize top-down performance counter analysis to evaluate workloads and show how this can be measured on Haswell using likwid-perfctr and perf. In part 2 to follow, I’ll describe how top down metrics have been added to wspy.

Top Down Analysis
The top down analysis approach is based on a paper and slides by Yasin Ahmad. This technique is also described on Intel’s web site.

It also turns out this is implemented in the perf(1) tool for Intel platforms with the –topdown option including some generic counters that have been added.

Before reading about this technique my initial approach was going to be to first measure/calculate/research the costs of various events e.g. cache misses, tlb misses, branch misses, memory access and then use performance counters to measure how frequently these events occur. Using the frequency and cost and comparing across workloads I might better understand what factors are limiting particular workloads. While I might still take some of this approach later, the top-down method makes a lot of sense as a start.

There are several reasons why first costing events and then adding frequencies can be difficult on a multi-core, super-scalar, parallelized and speculated micro-architecture.

• Finding costs is not always straightforward, since almost by definition the micro-architecture is trying to minimize their effects. Hence, it can be tough to make experiments that actually measure what you expect
• Many operations occur in parallel. This means that forward progress could be stalled for more than one reason. It might be possible that other progress gets made during a stall. For example, in a hyper-threaded program the opposite thread might take advantage of a stall.
• While there is no shortage of events and counters to example, it can also be difficult to know which of these events are most relevant.

The main idea behind top-down performance counter analysis is to first characterize the workload based on key metrics. These metrics are based on a small number of counters in the Intel architecture that see how well the processor pipelines are being used. With this overall pipeline analysis one can first characterize bottlenecks into several categories (front-end, back-end, retiring, speculation). These categories can then be used to further guide subsequent analysis based on the specific category.

Measurement Techniques
Slide #13 of Ahmed Yasin’s slides defines the following event names based on five counters:

TotalSlots = 4 * CPU_CLK_UNHALTED.THREAD
SlotsIssued = UOPS_ISSUED.ANY
SlotsRetired = UOPS_RETIRED.RETIRE_SLOTS
FetchBubbles = IDQ_UOPS_NOT_DELIVERED.CORE
RecoveryBubbles = 4 * INT_MISC.RECOVERY_CYCLES

these events are then used to compute the following metrics:

Frontend Bound = FetchBubbles / TotalSlots
Bad Speculation = (SlotsIssues - SlotsRetired  + RecoveryBubbles)/ TotalSlots
Retiring = SlotsRetired / TotalSlots
Backend Bound = 1 - (FrontendBound + BadSpeculation + Retiring) 

There are four measurement configurations of interest: likwid, perf, wspy periodic timer and wspy process tree.

In a previous blog post about likwid-perfctr I printed the results from likwid-perfctr runs benchmarks of a Phoronix CPU suite. There isn’t a metric exactly like this one, though a few are close. For example, the CYCLE_ACTIVITY group reports the percentage of cycles spent waiting on stalls due to data traffic. So I created a new performance group and placed the file in

/usr/share/likwid/perfgroups/haswell/TOPDOWN.txt

Following is the contents of the new file I created:

SHORT Top down cycle allocation

EVENTSET
FIXC0 INSTR_RETIRED_ANY
FIXC1 CPU_CLK_UNHALTED_CORE
PMC0 UOPS_ISSUED_ANY
PMC1 UOPS_RETIRED_RETIRE_SLOTS
PMC2 IDQ_UOPS_NOT_DELIVERED_CORE
PMC3 INT_MISC_RECOVERY_CYCLES

METRICS
IPC FIXC0/FIXC1
Total Slots 4*FIXC1
Slots Retired PMC1
Fetch Bubbles PMC2
Recovery Bubbles 4*PMC3
Front End [%] PMC2/(4*FIXC1)*100
Speculation [%] (PMC0-PMC1+(4*PMC3))/(4*FIXC1)*100
Retiring [%] PMC1/(4*FIXC1)*100
Back End [%] (1-((PMC2+PMC0+(4*PMC3))/(4*FIXC1)))*100

LONG
Front End [%] = IDQ_UOPS_NOT_DELIVERED_CORE/(4*CPU_CLK_UNHALTED_CORE)*100
Speculation [%] = (UOPS_ISSUED_ANY-UOPS_RETIRED_RETIRE_SLOTS+(4*INT_MISC_RECOV\
ERY_CYCLES))/(4*CPU_CLK_UNHALTED_CORE)*100
Retiring [%] = UOPS_RETIRED_RETIRE_SLOTS/(4*CPU_CLK_UNHALTED_CORE)*100
Back End [%] = (1-((IDQ_UOPS_NOT_DELIVERED_CORE+UOPS_ISSUED_ANY+(4*INT_MISC_RE\
COVERY_CYCLES))/(4*CPU_CLK_UNHALTED_CORE)))*100
--
This performance group measures cycles to determine percentage of time spent i\
n front end, back end, retiring and speculation.

The results files of this run across the Phoronix CPU benchmarks are linked below:
padman etqw-demo graphics-magick john-the-ripper ttsiod-renderer compress-pbzip2 compress-7zip encode-mp3 encode-flac x264 ffmpeg openssl himeno pgbench apache c-ray povray smallpt tachyon crafty tscp mafft stream

These are also summarized in the following table. It is also useful to note the sorts of events to investigate further when particular areas are high

• Frontend Bound: fetch latency (iTLB, iCache, Branch Resteers) and fetch bandwidth
• Bad Speculation: Branch mispredicts and machine clears
• Retiring: Floating point arithmetic, micro-sequencer
• Backend Bound: memory bound (L1, L2, L3, external memory), core bound (divider, execution port utilization)

After implementing this for likwid-perfctr, I noticed that my Intel platforms actually had some general architecture breakpoints named topdown-fetch-bubble, topdown-slots-issues, topdown-slots-retired and topdown-total-slots. Looking further, I saw this method had been implemented for perf(1

       --topdown
           Print top down level 1 metrics if supported by the CPU. This allows
           to determine bottle necks in the CPU pipeline for CPU bound
           workloads, by breaking the cycles consumed down into frontend
           bound, backend bound, bad speculation and retiring.

root@pasto:~# perf stat -a --topdown gcc -o hello hello.c

On an in order Atom machine (no speculation), I did a quick test:

root@pasto:~# perf stat -a --topdown gcc -o hello hello.c

 Performance counter stats for 'system wide':

                  retiring             frontend bound       backend bound/bad spec 
S0-C0           1     16.3%               50.3%                               
S0-C1           1     10.6%               58.6%                               
S0-C2           1     23.7%               29.9%                               
S0-C3           1     20.8%               21.4%                               
S0-C4           1     19.3%               45.2%                               
S0-C5           1     37.5%               32.5%                               
S0-C6           1     15.3%               54.6%                               
S0-C7           1     21.8%               44.1%                               

       0.116470972 seconds time elapsed

Excellent, nice to see it here. Also here is the lwn article on when this was added to perf along with some notes about how hyper-threading is treated and a reference to the pmu_tools that provide additional topdown metrics used to drill down further.

As a whole, I the technique seems intriguing. However, I also need to calibrate this further with some of the underlying metrics. A further next step is also to implement this with wspy. That will let me watch how these metrics might vary over time as an application runs. They might also let me see how these values build hierarchically in a tree of processes.

A different avenue is to see if there are some equivalent metrics that might provide a similar high-level overview on AMD or ARM processors.