Benchmarking

Intel CPU analysis with VTune, PCM and perf: distinguish cache misses from memory stalls

Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to DRAM.

A cache-miss count is the start of a question. It does not tell you whether the request reached DRAM, whether another cache supplied the line, or whether the processor overlapped the delay with useful work.

For an Intel CPU investigation, connect the application’s slow phase to core behavior and then to the memory subsystem. VTune can attribute activity to code and threads. Intel Performance Counter Monitor supplies useful platform-level views. Linux perf can provide a compact repeatable measurement when the selected events are supported and permitted.

Inventory the CPU set the application can actually use

Record processor model, stepping, topology, firmware settings relevant to the experiment, kernel and profiler versions. Include the batch allocation and the process’s allowed CPUs. A node with many logical CPUs does not necessarily grant a job access to all of them.

Start with read-only topology observations:

Illustrative session
Inventory the CPU set the application can actually use A deliberately small topology example. Use the actual socket, core and NUMA mapping when placing a profiling run. Illustrative example; not a captured production run. Open full size · Scroll sideways to inspect the terminal.
Read the illustrated session as text
$ lscpu -e=CPU,CORE,SOCKET,NODE
CPU CORE SOCKET NODE
  0    0      0    0
  1    1      0    0
  2    2      1    1
  3    3      1    1
Read and copy the original commands
lscpu
lscpu -e=CPU,CORE,SOCKET,NODE
numactl --hardware

Map threads to physical cores, SMT siblings and NUMA domains. Do not infer memory locality from socket count alone. A processor platform can expose several locality domains within a socket, and firmware policy affects the topology presented to the operating system.

Retain dataset size and initialization behavior. Memory placement established by the threads that initialize data can persist into the timed phase, making an apparently identical compute loop behave differently.

Establish a small process-level baseline

Use the uninstrumented application for elapsed time and correctness. Then add a bounded observation such as:

perf list
perf stat -r 3 -e cycles,instructions -- ./solver input.dat

solver and its input are placeholders. The perf stat manual documents event counting and reporting. Confirm event availability, permissions and the scope of each result on the actual system.

Counted instructions and cycles provide context, not a scientific throughput metric. A vector instruction can perform different work from a scalar one, and a more efficient algorithm may retire fewer instructions while taking less time. Keep application output rate beside IPC rather than optimizing the ratio in isolation.

Locate the hot phase before opening every metric

A compact VTune collection can begin with:

vtune -collect hotspots -result-dir vtune-hotspots -- ./solver input.dat
vtune -collect memory-access -result-dir vtune-memory -- ./solver input.dat

Use separate runs and matching inputs. Collection availability, data detail and overhead depend on the target CPU, permissions and installed collectors. The VTune command-line guide describes analysis types and target syntax.

Identify which functions, threads and intervals dominate the phase. Sampling is evidence about observed execution, with attribution limits. A source line receiving samples may be near the expensive operation rather than a complete explanation of why it is expensive.

Distinguish core execution limits from memory limits

Use the supported microarchitecture view to form a hypothesis: front-end delivery, speculation, execution resources, memory latency or bandwidth may be involved. The event formulas and categories depend on processor generation and tool version.

Keep counts with their denominators. If calculating misses per thousand retired instructions, the miss event and instruction count need compatible scope and interval. A socket-wide count divided by one thread’s instructions produces a number with no useful interpretation.

Check whether events were multiplexed, unsupported or scaled. Collecting too many events can reduce how long each is observed. Repeat a narrower set when the result is central to the conclusion, especially if activity varies across phases.

Use uncore observations to test the memory explanation

Intel PCM includes tools such as pcm-memory and pcm-numa for supported platforms. These can help examine memory-channel traffic and locality-related behavior beyond one process’s sampled hot lines.

Keep platform scope explicit. A socket’s memory traffic includes other work, operating-system activity and potentially other users. Collect during an isolated maintenance allocation where possible, or retain enough context to describe the contamination.

Avoid running competing counter tools simultaneously without reviewing their interaction. PMU ownership and collection mechanisms can conflict or reduce available measurements. Coordinate the experiment through the site’s normal profiling policy instead of weakening machine-wide access controls to make every tool run.

Separate locality, bandwidth and sharing

Use a controlled comparison rather than diagnosing all three from a single miss ratio:

Hypothesis Focused experiment Evidence to retain
Remote placement hurts latency Compare supported local and remote placement with the same CPU set and initialization policy Application time, placement evidence, locality and traffic observations
Memory bandwidth is saturated Vary active cores while keeping the useful work comparable Output rate, memory traffic, per-thread behavior and variation
Threads contend on cache lines Change layout or ownership of the suspected shared objects Correctness, line-sharing evidence, coherence-related observations and elapsed time

Keep each experiment within the assigned resources. A lower thread count can reduce both bandwidth demand and synchronization, so success at that setting does not identify the cause without supporting evidence.

Prove false sharing rather than inferring it from proximity

Two threads updating independent values on the same cache line can create coherence traffic. That is different from necessary communication through a genuinely shared value. Find which threads write which locations and inspect the actual object layout.

Intel’s false-sharing example illustrates the kind of attribution required. Its measured speedups belong to that example, not to an arbitrary research code. Padding or partitioning is an experiment whose additional footprint and locality effects must also be measured.

A reduction in a coherence-related event is useful, but accept the change only after the application remains correct and completes the intended work faster or with a documented resource benefit.

Finish with a model-specific conclusion

Save the CPU model, event definitions, scope, collection configuration, workload phase and placement with the report. Explain which observation supported the cause and which controlled change tested it.

Repeat the uninstrumented workload and a representative larger case. A cache optimization that helps a tiny working set may lose its advantage when data grows or another rank shares the socket.

For the equivalent analysis boundaries on AMD systems, see AMD CPU core, cache and IBS analysis. Comparable research questions do not make vendor PMU events interchangeable.

All articles