When you embark on a journey to squeeze the last bit of performance from your application, simply looking at the list of hotspots is not enough. Unless you have a crystal ball or an accurate model of an entire CPU in your head, you need additional support to understand what the performance bottlenecks are.

Some developers rely on their intuition and proceed with random experiments, trying to force various compiler optimizations like loop unrolling, vectorization, inlining, you name it. Indeed, sometimes you can be lucky and enjoy a portion of compliments from your colleagues and maybe even claim an unofficial title of performance guru on your team. But usually, you need to have a very good intuition and luck. In this book, we don’t teach you how to be lucky. Instead, we show methods that have proved to be working in practice.

Top-Down Microarchitecture Analysis (TMA)

Proposed in 2014. This approach is so effective that it turns out to be code agnostic. You can do TMA analysis without taking a look at the code.

You'll wait all microperations to fall into the left most bracket.

There might actually be classes of events which fall under each of the buckets:

The concept is to drill down (identify and see that its memory bound events, then run only the memory bound events to determine which event dominates.

You can multiplex PMCs to check multiple metrics at the same time. Provided "the workload is steady".

After drilling down you can smaple on the specific event to get more pointed results.

Perf defaults to this taxonomy:

$ perf stat-- ./benchmark.exe
...
TopdownL1 (cpu_core) # 53.4 % tma_backend_bound
<==
#
0.2 % tma_bad_speculation
# 13.8 % tma_frontend_bound
# 32.5 % tma_retiring

You can use toplev to descend one level

$ ~/pmu-tools/toplev.py--core S0-C0-l2-v--no-desc taskset-c 0 ./benchmark.exe
...
# Level 1
S0-C0 Frontend_Bound:
13.92 % Slots
S0-C0 Bad_Speculation:
0.23 % Slots
S0-C0 Backend_Bound:
53.39 % Slots
S0-C0 Retiring:
32.49 % Slots
# Level 2
S0-C0 Frontend_Bound.FE_Latency:
12.11 % Slots
S0-C0 Frontend_Bound.FE_Bandwidth:
1.84 % Slots
S0-C0 Bad_Speculation.Branch_Mispred: 0.22 % Slots
S0-C0 Bad_Speculation.Machine_Clears: 0.01 % Slots
S0-C0 Backend_Bound.Memory_Bound:
44.59 % Slots <==
S0-C0 Backend_Bound.Core_Bound:
8.80 % Slots
S0-C0 Retiring.Base:
24.83 % Slots
S0-C0 Retiring.Microcode_Sequencer: 7.65 % Slots

Then, afterwards:

--core S0-C0: limit output report to this core

-c 0: only test on this core

--l2 is obviously an instructino to restrict collection to level 2 metrics

In our case, to find memory accesses that contribute to such a high value of the DRAM_Bound metric (miss in the L3 cache), we should sample on MEM_LOAD_RETIRED.L3_MISS_PS precise event. Here is the example command:

if foo is the problem, you can kill the dummy work time by using prefetches

Watch out for such cases and remember that TMA identifies CPU performance bottlenecks but doesn’t correlate them with the performance of your program. You will find it out once you do the necessary experiments.

A few final thoughts before we move on... As we mentioned at the beginning of this chapter, using TMA on code that has major performance flaws is not recommended because it will likely steer you in the wrong direction, and instead of fixing real high-level performance problems, you will be tuning bad code, which is just a waste of time. Similarly, make sure the environment doesn’t get in the way of profiling. For example, if you drop the filesystem cache and run the benchmark under TMA, it will likely show that your application is Memory Bound, which may in fact be false when the filesystem cache is warmed up.

TMA might also suggest when its time to upgrade your hardware (after you exhaust all software related optimisations).

Branch Recording

Last Branch Record

Works the same way as other PMIs (overflow then log). Is a ring buffer of the 32 most recent branch outcomes.

3 64 bit values:

1. Source address of branch
2. Destination address of branch
3. Metadata
   3a. mispredict (if mispredict, 'Y')
   3b. cycle count (if mispredict, 'N')

As its a record, it can estimate branch probabilities and help us choose which branch path to optimise.

Recall types of branches:

JNE - loop back edge

JNS - conditional branch

CALL - calling from the function

add rbx, 0x4 - returning from the function

LBR allows auto filtering

LBR also allows call-stack mode, where pairs of CALL and RETURNS will always be popped together. Then your LBR isn't flooded with leaf function calls.

perf record-b-e cycles-- ./benchmark.exe #this is for LBR stacks
perf record--call-graph lbr #this drops the branch mispredictions and cycles data

You can also export LBR for custom analysis

Users can export raw LBR stacks for custom analysis. Below is the Linux perf
command you can use to dump the contents of collected branch stacks:
$ perf record-b-e cycles-- ./benchmark.exe
$ perf script-F brstack &> dump.txt

On AMD hardware you can make similar analysis

$ AMDuProfCLI collect--branch-filter-o /tmp/ ./AMDTClassicMatMul-bin

$ perf record--call-graph lbr-- ./a.exe
$ perf report-n--stdio

--call-graph lbr

note that counts for samples cycles spent in funciton, not call frequency. It does still tell you which the hottest function in the program is, though.

perf record -b (hardware based sampling)

+: Lowers overhead sampling, enablesmore precise instruction capture

Each call captures a LBR, each with 32 branches and since there are 670 of them you get 21440 data points to analyse.

In tools like Linux perf, using --call-graph lbr is just a hardware-assisted mechanism to unwind the call stack during statistical sampling (usually sampling on CPU cycles).

it samples an event (like cycles), unwinds the call stack from the LBR ring buffer, and accumulates sample counts.
While the raw Last Branch Record (LBR) hardware tracks every recent branch (and thus basic blocks), the perf --call-graph lbr mode specifically configures the LBR to filter for function CALL and RET instructions to reconstruct the call chain cheaply without frame pointers.
If you want actual basic block profiling and execution branch frequencies, you use LBR branch sampling modes like perf record -b instead of --call-graph lbr.

Precision And Timing

Owing to Intel's LBR structure...

And with that data, then with respect to each block you can find...

Of course those two hoops seem to correspond to cache fetch latencies, so this specific block might benefit from a little bit of microfetching (in the specific example it involved a block of memory that wouldn't fit in L1).

it is feasible to plot basic block latencies not only for tiny microbenchmarks but for real-world applications as well. Currently, LBR is the most precise cycle-accurate source of timing information on Intel systems.

$ perf record-e cycles-b-- ./7zip.exe b
$ perf report-n--sort symbol_from,symbol_to-F +cycles,srcline_from,srcline_to--stdio
# Samples: 658K of event 'cycles'
# Event count (approx.): 658240

Intel Processor Traces (PT)

Popular in HFTs. But its not very popular outside of there, as its seen as "overkill" and using LBR sampling gets the job done most of the time.

Hardware-Based Sampling Features

Intel's PEBS - Processor Event Based Sampling

In Intel they have the PEBS system to replace costly interrupts with gentle traps that will simultanaeously store the relevant info into its buffer during a PMC overload while restting the counter. Only when the buffer overflows then an inevitable interrupt occurs.

In PEBs if you want to profile memory access behaviour you can sample one every 100,000 (mind the latency) and use Data Adress Profiling (DLA) (yes the acronym doesnt match the starting letters). DLA enable auto filtering at collection time.

AMD's IBS - Instruction Based Sampling

Fixed counters that are enabled or disabled with little overhead. Arranged by CPU frontend/backend (i.e. by instruction fetch and decode versus LSU unit level errors)

$ perf record-a-e ibs_op/cnt_ctl=1,l3missonly=1/-- benchmark.exe
$ perf record-a-e ibs_fetch/l3missonly=0/-- benchmark.exe
$ perf report

Skid

Distance between IP causing a PMC overflow, and the IP where the event is tagged.

For AMD IBS and Arm SPE since they make use of hardware captures, its precise by design (...?)

PEBS has support for only some but not all the events being precise.

$ perf record-e cycles:pp-- ./a.exe