wspy is both a code base and a tool for performance analysis.

As a code base, I modify wspy to run particular experiments and measurements. This lets me experiment trying particular techniques and interfaces to create a quick and dirty tool for a particular analysis. Occasionally I’ll find a situation it isn’t quite robust e.g. run out of file descriptors. As that happens or as I need new experiments I’m changing and updating the code.

The code is hosted at github.com under an MIT license, though I am the only contributor. However, using github makes it easy to pull the code onto multiple (internet connected) systems and also reuse the experimental code if it proves useful.

As a tool, wspy has three main types of data collection.

1. A periodic sampling of overall system state such as CPU usage or disk information with hooks to add future items like network usage or memory..
2. A monitoring of the processes in a workload to create a “process tree” for both analysis and output.
3. A sampling of hardware performance counters. Initially focused on metrics such as IPC, but generic enough to measure most any event.

These are described in more detail below along with some examples.

The code for wspy keeps evolving, but a recent usage dump of wspy to show major options is:

./wspy: fatal error: usage: ./wspy [options] ...
	--cpustats, --no-cpustats      	CPU usage tracing from /proc/stat
	--diskstats,--no-diskstats     	Disk usage tracing from /proc/diskstats
	--memstats, --no-memstats      	Memory usage tracing from /proc/meminfo
	--netstats, --no-netstats      	Network usage tracing from /proc/net/dev
	--processtree, --no-processtree	Generate process tree
	--processtree-engine   	Set engine to either ftrace or ptrace
	--perfcounters, --no-perfcounters	Collect basic perf counters
	--perfcounter-model            	Set perfcounters to either core or process
	--counterinfo                  	Read directory of counter definitions
	--show-counters,--no-show-counters	Show available counters
	--set-counters :	Set list of counters to measure
	--uid , -u           	run as user
	--set-cpumask         	bind child to list of cores
	--interval              	set timer in ms (default 1000)
	--zip            	create zip archive of results
	--root , -r        	set name for process tree root
	--config                 	read configuration file
	--output                 	redirect output to 
	--error-output           	redirect errors and debug to 
	--debug, -d                    	internal debugging flag

In addition to the options, there is a config file mechanism where any additional options can be given as “command options”. This lets me easily create both a default configuration and particular configuration files for a specific analysis.

The following sections describe what is done so far in each of three areas: overall statistics, tree analysis and performance counters.

Overall Statistics
wspy has a periodic counter with default interval of once per second but configurable with the –interval option.

I’ve added a –cpustats option that uses this counter to periodically look at /proc/stat. This file records overall time the process spends scheduled in user time, system time and running the idle loop. This information is also used in tools like top(1) to show %user %system. The –cpustats option dumps a set of *.csv files that can then be graphed with gnuplot.

I’ve also added a –diskstats option to look at /proc/diskstats.

I parse options for –memstats (/proc/statm) and –netstats, but haven’t implemented these yet.

Process Tree Analysis
wspy has implemented four different methods of gathering process tree information. These came while collecting slightly different information and also troubleshooting. These four methods can be selected using the –processtree-engine option and include:

–processtree-engine ftrace; uses the kernel /sys/kernel/debug/tracing facility to read OS scheduler events for fork/exec/exit and uses these to contruct an in-memory tree. This mechanism typically requires root access.

–processtree-engine ptrace; uses the process tracing mechanism also used by gdb(1) and strace(1) to follow a tree of processes and intercept fork/exec/exit events. This mechanism usually allows one to examine a process /proc information before the process exits.

–processtree-engine tracecmd; invokes the external tool trace-cmd to read kernel ftrace events.

–proocesstree-engine ptrace2; experimental improvements to ptrace that no longer keep an in-memory tree but instead dump a *csv file of process information to parse into a tree later. Experimenting with making this more lightweight and versatile than the original ptrace implementation though both are useful.

Performance counters
Performance counter tools inventory available counters, instrument the application and then display the results.

Inventory information comes from three sources: (1) the /sys/kernel/devices pseudo directories have a more generic set of counters such as “instructions” and “cpu-cycles”. These are read to create a list of available counters (2) the perf_event_open(2) man page documents a set of generic events. These are also made available and (3) the –counterinfo option can read a list of counters in same format as /sys/kernel/devices. As an example, I’ve written a small program to parse Excel spreadsheets published by Intel into this format so I use many of the hundreds of Intel counters on my processor.

The application can be instrumented in two methods. These can be chosen using the –perfcounter-model option.

–perfcounter-model core; measures core by core for any process that runs on that core. Different counters can be measured on each core, or they can all be measured the same.

–perfcounter-model process; instruments processes when they start and reads the counters when they exit. This allows me to collect a tree of counter information for further analysis.

Multiple analysis methods can be built using these options. Initially I’ve focused on higher level metrics including IPC and the topdown retiring/front end/back end/speculation counters or specific queries like memory controller reads/writes. As needed this analysis will expand from further analyis.

Overall, wspy does basic things I need while serving as a base to keep extending and growing for doing additional analysis. Unless stated or obviously otherwise, graphs and metrics plots on this site come using wspy.

Next steps: Add some example usage and outputs to this page.