As described in top down performance counter analysis part 1, top down analysis is an approach that uses key performance counters to characterize an application and then successively drills down with further refinement. On Intel x86 processors, this first level refinement characterizes the applications by looking at the overall usage of the pipeline. Topdown is implemented in the perf(1) tool and in the previous post, I also was able to extend likwid-perfctr to add a TOPDOWN performance group for 1st level characterization.

In my last post, I got topdown performance counter metrics added to the wspy tool when it is run in the “core” mode. In other words, instead of following processes, wspy reports against all processes that run on a core. In this posting, I have now also added it when performance counters are run per process. The resultant process trees are now decorated with the topdown metrics.

One interesting thing I noticed in the last post was for hyper-threading that the front-end cycles are sometimes unevenly divided, resulting in being attributed to one thread or another. I have also noticed this within the process tree as well. As a result, I don’t trust the metrics printed unless they are on a set of processes that run across all cores since otherwise, the accounting between processes might be suspect.

Here is an example of a process tree for c-ray that is now adorned with topdown metrics. I’ll trust the top level metrics (that sum for all child processes and run on all cores) but don’t give as much weight to lower down processes.

      [22744] c-ray        cpu=5 ipc=1.44 on_cpu=0.99 on_core=7.92 retire=0.740 frontend=0.104 spec=0.008 backend=0.148 elapsed=26.51 user=0.00 system=0.00 pcount=130
        (22745) c-ray-mt     cpu=7 ipc=1.44 on_cpu=0.99 on_core=7.92 retire=0.740 frontend=0.104 spec=0.008 backend=0.148 elapsed=26.51 user=0.08 system=0.00 pcount=129
          (22746) c-ray-mt     cpu=4 ipc=1.43 on_cpu=0.01 on_core=0.10 retire=0.732 frontend=0.194 spec=0.006 backend=0.068 elapsed=14.23 user=1.42 system=0.00 pcount=1
          (22747) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.09 retire=0.737 frontend=0.016 spec=0.006 backend=0.241 elapsed=15.23 user=1.40 system=0.00 pcount=1
          (22748) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.10 retire=0.732 frontend=0.193 spec=0.006 backend=0.069 elapsed=15.22 user=1.57 system=0.00 pcount=1
          (22749) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.11 retire=0.737 frontend=0.018 spec=0.006 backend=0.239 elapsed=13.27 user=1.42 system=0.00 pcount=1
          (22750) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.738 frontend=0.016 spec=0.006 backend=0.240 elapsed=22.58 user=1.44 system=0.00 pcount=1
          (22751) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.733 frontend=0.195 spec=0.006 backend=0.066 elapsed=24.58 user=1.62 system=0.00 pcount=1
          (22752) c-ray-mt     cpu=2 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.017 spec=0.006 backend=0.236 elapsed=24.47 user=1.46 system=0.00 pcount=1
          (22753) c-ray-mt     cpu=1 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.734 frontend=0.194 spec=0.006 backend=0.065 elapsed=25.70 user=1.64 system=0.00 pcount=1
          (22754) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.019 spec=0.006 backend=0.233 elapsed=23.03 user=1.43 system=0.00 pcount=1
          (22755) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.734 frontend=0.195 spec=0.006 backend=0.064 elapsed=23.84 user=1.45 system=0.00 pcount=1
          (22756) c-ray-mt     cpu=2 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.017 spec=0.007 backend=0.234 elapsed=25.86 user=1.62 system=0.00 pcount=1
          (22757) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.10 retire=0.738 frontend=0.016 spec=0.007 backend=0.239 elapsed=14.57 user=1.48 system=0.00 pcount=1
          (22758) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.11 retire=0.733 frontend=0.195 spec=0.007 backend=0.066 elapsed=13.22 user=1.47 system=0.00 pcount=1
          (22759) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.021 spec=0.007 backend=0.233 elapsed=25.86 user=1.61 system=0.00 pcount=1
          (22760) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.733 frontend=0.196 spec=0.006 backend=0.065 elapsed=23.66 user=1.43 system=0.00 pcount=1
          (22761) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.017 spec=0.007 backend=0.237 elapsed=24.42 user=1.57 system=0.00 pcount=1
          (22762) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.018 spec=0.007 backend=0.235 elapsed=22.28 user=1.40 system=0.00 pcount=1
          (22763) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.733 frontend=0.196 spec=0.007 backend=0.065 elapsed=23.65 user=1.43 system=0.00 pcount=1
          (22764) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.734 frontend=0.195 spec=0.007 backend=0.064 elapsed=25.59 user=1.62 system=0.00 pcount=1
          (22765) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.018 spec=0.007 backend=0.232 elapsed=23.46 user=1.46 system=0.00 pcount=1
          (22766) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.734 frontend=0.193 spec=0.007 backend=0.065 elapsed=24.61 user=1.51 system=0.00 pcount=1
          (22767) c-ray-mt     cpu=5 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.737 frontend=0.131 spec=0.008 backend=0.124 elapsed=25.93 user=1.70 system=0.00 pcount=1
          (22768) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.741 frontend=0.019 spec=0.008 backend=0.232 elapsed=23.02 user=1.52 system=0.00 pcount=1
          (22769) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.735 frontend=0.194 spec=0.007 backend=0.064 elapsed=25.80 user=1.66 system=0.00 pcount=1
          (22770) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.736 frontend=0.193 spec=0.007 backend=0.064 elapsed=24.47 user=1.50 system=0.00 pcount=1
          (22771) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.017 spec=0.008 backend=0.235 elapsed=25.15 user=1.53 system=0.00 pcount=1
          (22772) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.735 frontend=0.192 spec=0.007 backend=0.065 elapsed=25.41 user=1.72 system=0.00 pcount=1
          (22773) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.734 frontend=0.192 spec=0.007 backend=0.066 elapsed=23.98 user=1.56 system=0.00 pcount=1
          (22774) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.017 spec=0.008 backend=0.234 elapsed=25.39 user=1.56 system=0.00 pcount=1
          (22775) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.742 frontend=0.017 spec=0.008 backend=0.233 elapsed=26.11 user=1.76 system=0.00 pcount=1
          (22776) c-ray-mt     cpu=5 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.189 spec=0.008 backend=0.065 elapsed=24.75 user=1.62 system=0.00 pcount=1
          (22777) c-ray-mt     cpu=1 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.021 spec=0.007 backend=0.229 elapsed=25.21 user=1.70 system=0.00 pcount=1
          (22778) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.734 frontend=0.194 spec=0.007 backend=0.064 elapsed=23.76 user=1.51 system=0.00 pcount=1
          (22779) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.017 spec=0.008 backend=0.233 elapsed=24.21 user=1.55 system=0.00 pcount=1
          (22780) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.744 frontend=0.017 spec=0.008 backend=0.231 elapsed=25.66 user=1.72 system=0.00 pcount=1
          (22781) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.017 spec=0.008 backend=0.233 elapsed=25.44 user=1.57 system=0.00 pcount=1
          (22782) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.743 frontend=0.017 spec=0.008 backend=0.232 elapsed=24.66 user=1.59 system=0.00 pcount=1
          (22783) c-ray-mt     cpu=3 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.147 spec=0.008 backend=0.106 elapsed=26.05 user=1.81 system=0.00 pcount=1
          (22784) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.736 frontend=0.190 spec=0.008 backend=0.065 elapsed=25.24 user=1.64 system=0.00 pcount=1
          (22785) c-ray-mt     cpu=7 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.744 frontend=0.026 spec=0.008 backend=0.222 elapsed=26.23 user=1.79 system=0.00 pcount=1
          (22786) c-ray-mt     cpu=1 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.735 frontend=0.190 spec=0.008 backend=0.067 elapsed=25.17 user=1.63 system=0.00 pcount=1
          (22787) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.06 retire=0.744 frontend=0.022 spec=0.008 backend=0.226 elapsed=24.93 user=1.59 system=0.00 pcount=1
          (22788) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.040 spec=0.008 backend=0.208 elapsed=26.12 user=1.78 system=0.00 pcount=1
          (22789) c-ray-mt     cpu=4 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.737 frontend=0.191 spec=0.008 backend=0.064 elapsed=25.04 user=1.61 system=0.00 pcount=1
          (22790) c-ray-mt     cpu=4 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.738 frontend=0.186 spec=0.008 backend=0.069 elapsed=24.65 user=1.56 system=0.00 pcount=1
          (22791) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.017 spec=0.008 backend=0.232 elapsed=25.82 user=1.78 system=0.00 pcount=1
          (22792) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.743 frontend=0.018 spec=0.009 backend=0.231 elapsed=24.76 user=1.60 system=0.00 pcount=1
          (22793) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.742 frontend=0.017 spec=0.008 backend=0.233 elapsed=25.70 user=1.73 system=0.00 pcount=1
          (22794) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.017 spec=0.008 backend=0.236 elapsed=24.06 user=1.52 system=0.00 pcount=1
          (22795) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.018 spec=0.008 backend=0.234 elapsed=23.77 user=1.50 system=0.00 pcount=1
          (22796) c-ray-mt     cpu=7 ipc=1.42 on_cpu=0.01 on_core=0.07 retire=0.731 frontend=0.194 spec=0.008 backend=0.068 elapsed=25.53 user=1.68 system=0.00 pcount=1
          (22797) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.732 frontend=0.193 spec=0.008 backend=0.067 elapsed=24.08 user=1.52 system=0.00 pcount=1
          (22798) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.732 frontend=0.194 spec=0.008 backend=0.066 elapsed=24.15 user=1.54 system=0.00 pcount=1
          (22799) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.741 frontend=0.017 spec=0.008 backend=0.234 elapsed=25.48 user=1.69 system=0.00 pcount=1
          (22800) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.017 spec=0.008 backend=0.234 elapsed=23.97 user=1.53 system=0.00 pcount=1
          (22801) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.017 spec=0.008 backend=0.236 elapsed=25.64 user=1.72 system=0.00 pcount=1
          (22802) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.017 spec=0.008 backend=0.233 elapsed=24.80 user=1.58 system=0.00 pcount=1
          (22803) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.017 spec=0.008 backend=0.232 elapsed=24.93 user=1.60 system=0.00 pcount=1
          (22804) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.019 spec=0.009 backend=0.229 elapsed=26.22 user=1.83 system=0.00 pcount=1
          (22805) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.743 frontend=0.017 spec=0.009 backend=0.231 elapsed=26.11 user=1.69 system=0.00 pcount=1
          (22806) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.734 frontend=0.191 spec=0.008 backend=0.067 elapsed=25.41 user=1.66 system=0.00 pcount=1
          (22807) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.736 frontend=0.190 spec=0.008 backend=0.066 elapsed=26.04 user=1.85 system=0.00 pcount=1
          (22808) c-ray-mt     cpu=7 ipc=1.42 on_cpu=0.01 on_core=0.07 retire=0.735 frontend=0.191 spec=0.008 backend=0.066 elapsed=25.72 user=1.71 system=0.00 pcount=1
          (22809) c-ray-mt     cpu=0 ipc=1.48 on_cpu=0.01 on_core=0.07 retire=0.762 frontend=0.038 spec=0.009 backend=0.191 elapsed=26.41 user=1.89 system=0.00 pcount=1
          (22810) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.188 spec=0.009 backend=0.064 elapsed=25.68 user=1.75 system=0.00 pcount=1
          (22811) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.744 frontend=0.020 spec=0.009 backend=0.227 elapsed=25.67 user=1.78 system=0.00 pcount=1
          (22812) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.189 spec=0.009 backend=0.064 elapsed=26.22 user=1.94 system=0.00 pcount=1
          (22813) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.737 frontend=0.190 spec=0.009 backend=0.064 elapsed=25.66 user=1.73 system=0.00 pcount=1
          (22814) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.736 frontend=0.189 spec=0.009 backend=0.066 elapsed=25.75 user=1.72 system=0.00 pcount=1
          (22815) c-ray-mt     cpu=3 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.018 spec=0.009 backend=0.228 elapsed=26.32 user=1.89 system=0.00 pcount=1
          (22816) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.020 spec=0.010 backend=0.225 elapsed=25.19 user=1.73 system=0.00 pcount=1
          (22817) c-ray-mt     cpu=2 ipc=1.46 on_cpu=0.01 on_core=0.07 retire=0.758 frontend=0.179 spec=0.009 backend=0.054 elapsed=26.41 user=1.93 system=0.00 pcount=1
          (22818) c-ray-mt     cpu=3 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.168 spec=0.009 backend=0.083 elapsed=26.02 user=1.82 system=0.00 pcount=1
          (22819) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.188 spec=0.009 backend=0.064 elapsed=26.03 user=1.83 system=0.00 pcount=1
          (22820) c-ray-mt     cpu=1 ipc=1.47 on_cpu=0.01 on_core=0.07 retire=0.761 frontend=0.058 spec=0.009 backend=0.172 elapsed=26.37 user=1.96 system=0.00 pcount=1
          (22821) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.029 spec=0.009 backend=0.217 elapsed=25.95 user=1.79 system=0.00 pcount=1
          (22822) c-ray-mt     cpu=4 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.742 frontend=0.187 spec=0.009 backend=0.063 elapsed=26.07 user=1.77 system=0.00 pcount=1
          (22823) c-ray-mt     cpu=7 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.023 spec=0.009 backend=0.224 elapsed=26.29 user=1.93 system=0.00 pcount=1
          (22824) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.744 frontend=0.018 spec=0.009 backend=0.229 elapsed=26.00 user=1.76 system=0.00 pcount=1
          (22825) c-ray-mt     cpu=7 ipc=1.49 on_cpu=0.01 on_core=0.07 retire=0.768 frontend=0.174 spec=0.009 backend=0.049 elapsed=26.40 user=1.93 system=0.00 pcount=1
          (22826) c-ray-mt     cpu=2 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.740 frontend=0.180 spec=0.009 backend=0.071 elapsed=26.30 user=1.82 system=0.00 pcount=1
          (22827) c-ray-mt     cpu=2 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.180 spec=0.008 backend=0.072 elapsed=26.14 user=1.77 system=0.00 pcount=1
          (22828) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.189 spec=0.008 backend=0.065 elapsed=26.12 user=1.92 system=0.00 pcount=1
          (22829) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.189 spec=0.008 backend=0.065 elapsed=25.67 user=1.70 system=0.00 pcount=1
          (22830) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.747 frontend=0.021 spec=0.008 backend=0.224 elapsed=25.34 user=1.72 system=0.00 pcount=1
          (22831) c-ray-mt     cpu=0 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.748 frontend=0.018 spec=0.008 backend=0.226 elapsed=26.33 user=1.93 system=0.00 pcount=1
          (22832) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.746 frontend=0.018 spec=0.008 backend=0.228 elapsed=25.64 user=1.69 system=0.00 pcount=1
          (22833) c-ray-mt     cpu=4 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.747 frontend=0.021 spec=0.008 backend=0.224 elapsed=26.29 user=1.87 system=0.00 pcount=1
          (22834) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.185 spec=0.007 backend=0.070 elapsed=25.73 user=1.70 system=0.00 pcount=1
          (22835) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.019 spec=0.007 backend=0.229 elapsed=25.60 user=1.67 system=0.00 pcount=1
          (22836) c-ray-mt     cpu=5 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.119 spec=0.008 backend=0.129 elapsed=26.32 user=1.93 system=0.00 pcount=1
          (22837) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.747 frontend=0.018 spec=0.008 backend=0.227 elapsed=26.01 user=1.76 system=0.00 pcount=1
          (22838) c-ray-mt     cpu=5 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.748 frontend=0.025 spec=0.008 backend=0.219 elapsed=26.14 user=1.79 system=0.00 pcount=1
          (22839) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.187 spec=0.008 backend=0.063 elapsed=26.33 user=1.96 system=0.00 pcount=1
          (22840) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.186 spec=0.008 backend=0.063 elapsed=25.76 user=1.71 system=0.00 pcount=1
          (22841) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.741 frontend=0.188 spec=0.008 backend=0.063 elapsed=26.04 user=1.78 system=0.00 pcount=1
          (22842) c-ray-mt     cpu=3 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.170 spec=0.008 backend=0.083 elapsed=25.43 user=1.64 system=0.00 pcount=1
          (22843) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.021 spec=0.008 backend=0.229 elapsed=24.78 user=1.64 system=0.00 pcount=1
          (22844) c-ray-mt     cpu=7 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.740 frontend=0.173 spec=0.008 backend=0.080 elapsed=26.26 user=1.81 system=0.00 pcount=1
          (22845) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.189 spec=0.008 backend=0.063 elapsed=24.76 user=1.58 system=0.00 pcount=1
          (22846) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.189 spec=0.007 backend=0.063 elapsed=24.26 user=1.54 system=0.00 pcount=1
          (22847) c-ray-mt     cpu=0 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.744 frontend=0.018 spec=0.007 backend=0.231 elapsed=25.71 user=1.70 system=0.00 pcount=1
          (22848) c-ray-mt     cpu=4 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.191 spec=0.007 backend=0.064 elapsed=24.20 user=1.53 system=0.00 pcount=1
          (22849) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.190 spec=0.007 backend=0.063 elapsed=25.77 user=1.72 system=0.00 pcount=1
          (22850) c-ray-mt     cpu=6 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.737 frontend=0.190 spec=0.007 backend=0.065 elapsed=24.50 user=1.50 system=0.00 pcount=1
          (22851) c-ray-mt     cpu=5 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.188 spec=0.007 backend=0.065 elapsed=22.50 user=1.51 system=0.00 pcount=1
          (22852) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.179 spec=0.007 backend=0.076 elapsed=25.98 user=1.69 system=0.00 pcount=1
          (22853) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.099 spec=0.007 backend=0.152 elapsed=25.10 user=1.54 system=0.00 pcount=1
          (22854) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.741 frontend=0.189 spec=0.007 backend=0.063 elapsed=24.42 user=1.55 system=0.00 pcount=1
          (22855) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.740 frontend=0.189 spec=0.007 backend=0.063 elapsed=25.67 user=1.70 system=0.00 pcount=1
          (22856) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.741 frontend=0.020 spec=0.007 backend=0.233 elapsed=22.66 user=1.50 system=0.00 pcount=1
          (22857) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.191 spec=0.007 backend=0.064 elapsed=25.44 user=1.67 system=0.00 pcount=1
          (22858) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.743 frontend=0.025 spec=0.007 backend=0.226 elapsed=23.23 user=1.50 system=0.00 pcount=1
          (22859) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.742 frontend=0.017 spec=0.007 backend=0.234 elapsed=24.66 user=1.48 system=0.00 pcount=1
          (22860) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.742 frontend=0.019 spec=0.007 backend=0.233 elapsed=24.67 user=1.67 system=0.00 pcount=1
          (22861) c-ray-mt     cpu=1 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.745 frontend=0.019 spec=0.007 backend=0.229 elapsed=22.29 user=1.47 system=0.00 pcount=1
          (22862) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.739 frontend=0.191 spec=0.006 backend=0.064 elapsed=22.65 user=1.42 system=0.00 pcount=1
          (22863) c-ray-mt     cpu=4 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.737 frontend=0.192 spec=0.006 backend=0.064 elapsed=24.43 user=1.55 system=0.00 pcount=1
          (22864) c-ray-mt     cpu=3 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.743 frontend=0.018 spec=0.006 backend=0.233 elapsed=20.66 user=1.39 system=0.00 pcount=1
          (22865) c-ray-mt     cpu=7 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.733 frontend=0.192 spec=0.006 backend=0.068 elapsed=23.29 user=1.57 system=0.00 pcount=1
          (22866) c-ray-mt     cpu=2 ipc=1.44 on_cpu=0.01 on_core=0.06 retire=0.740 frontend=0.018 spec=0.006 backend=0.236 elapsed=23.86 user=1.42 system=0.00 pcount=1
          (22867) c-ray-mt     cpu=1 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.742 frontend=0.022 spec=0.006 backend=0.230 elapsed=21.47 user=1.41 system=0.00 pcount=1
          (22868) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.735 frontend=0.192 spec=0.006 backend=0.066 elapsed=24.41 user=1.60 system=0.00 pcount=1
          (22869) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.06 retire=0.733 frontend=0.194 spec=0.006 backend=0.067 elapsed=22.14 user=1.42 system=0.00 pcount=1
          (22870) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.738 frontend=0.019 spec=0.006 backend=0.237 elapsed=20.36 user=1.40 system=0.00 pcount=1
          (22871) c-ray-mt     cpu=1 ipc=1.45 on_cpu=0.01 on_core=0.07 retire=0.740 frontend=0.020 spec=0.006 backend=0.235 elapsed=22.83 user=1.57 system=0.00 pcount=1
          (22872) c-ray-mt     cpu=5 ipc=1.43 on_cpu=0.01 on_core=0.07 retire=0.734 frontend=0.194 spec=0.006 backend=0.067 elapsed=21.55 user=1.44 system=0.00 pcount=1
          (22873) c-ray-mt     cpu=1 ipc=1.44 on_cpu=0.01 on_core=0.07 retire=0.739 frontend=0.021 spec=0.006 backend=0.234 elapsed=24.22 user=1.59 system=0.00 pcount=1

Another thing that is fairly apparent is that displaying all this information while decorating a process tree may not be the best presentation form for this data. Hence, I’ve also enhanced wspy to dump the process tree information also in a *.csv file as a first step to creating a display tool that can print and produce more metrics on demand as well as include more summary metrics.