OProfile manual


Table of Contents

1. Introduction
1. Applications of OProfile
2. System requirements
3. Internet resources
4. Installation
5. Uninstalling OProfile
2. Overview
1. Getting started
2. Tools summary
3. Controlling the profiler
1. Using opcontrol
1.1. Examples
1.2. Specifying performance counter events
2. Using oprof_start
3. Configuration details
3.1. Hardware performance counters
3.2. OProfile in RTC mode
3.3. OProfile in timer interrupt mode
3.4. Pentium 4 support
3.5. Intel Itanium 2 support
3.6. Dangerous counter settings
4. Other features
4.1. pid/pgrp filter
4.2. Unloading the kernel module
4. Obtaining results
1. Image summaries and symbol summaries
2. Outputting annotated source
3. gprof-compatible output
4. Profile specifications
4.1. Examples
4.2. Profile specification parameters
4.3. Locating and managing binary images
5. Usage of opreport
5.1. Merging separate profiles
5.2. Options for opreport
6. Usage of opannotate
7. Usage of opgprof
5. Interpreting profiling results
1. Profiling interrupt latency
2. Kernel profiling
2.1. Interrupt masking
2.2. Idle time
2.3. Exiting tasks
2.4. Profiling kernel modules
3. Inaccuracies in annotated source
3.1. Side effects of optimizations
3.2. Prologues and epilogues
3.3. Inlined functions
3.4. Inaccuracy in line number information
4. Assembly functions
5. Other discrepancies
6. Profiling overhead
7. Acknowledgments

This manual applies to OProfile version 0.6cvs. OProfile is a profiling system for Linux 2.2/2.4/2.6 systems on a number of architectures. It is capable of profiling all parts of a running system, from the kernel (including modules and interrupt handlers) to shared libraries to binaries. It runs transparently in the background collecting information at a low overhead. These features make it ideal for profiling entire systems to determine bottle necks in real-world systems.

Many CPUs provide "performance counters", hardware registers that can count "events"; for example, cache misses, or CPU cycles. OProfile provides profiles of code based on the number of these occurring events: repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded. This information is aggregated into profiles for each binary image.

Some hardware setups do not allow OProfile to use performance counters: in these cases, no events are available, and OProfile operates in timer/RTC mode, as described in later chapters.

Linux kernel 2.2/2.4

OProfile uses a kernel module that can be compiled for 2.2.11 or later and 2.4. Versions 2.4.10 or above are recommended, and required if you use the boot-time kernel option nosmp. AMD Hammer support requires a recent (>= 2.4.19) kernel with the line EXPORT_SYMBOL(do_fork); present in kernel/ksyms.c. Such a kernel is present in the x86-64.org CVS repository. 2.5 kernels are supported with the in-kernel OProfile driver.

modutils 2.4.6 or above

You should have installed modutils 2.4.6 or higher (in fact earlier versions work well in almost all cases).

Supported architecture

For Intel IA32, a CPU with either a P6 generation or Pentium 4 core is required. In marketing terms this translates to anything between an Intel Pentium Pro (not Pentium Classics) and a Pentium 4 / Xeon, including all Celerons. The AMD Athlon, Duron, and Hammer CPUs are also supported. Other IA32 CPU types only support the RTC mode of OProfile; please see later in this manual for details. Hyper-threaded Pentium IVs are not supported in 2.4. For 2.4 kernels, the Intel IA-64 CPUs are also supported. For 2.5 kernels, there is additionally support for Alpha processors, and sparc64, ppc64, and PA-RISC in timer mode.

Uniprocessor or SMP

SMP machines are fully supported.

Required libraries

These libraries are required : popt, bfd, liberty (debian users: libiberty is provided in binutils-dev package), dl, plus the standard C++ libraries.

Bash version 2

The opcontrol script requires bash version 2 at least to be installed as /bin/bash or /bin/bash2

OProfile GUI

The use of the GUI to start the profiler requires the Qt 2 library. Qt 3 should also work.

ELF

Probably not too strenuous a requirement, but older A.OUT binaries/libraries are not supported.

K&R coding style

OK, so it's not really a requirement, but I wish it was...

Web page

There is a web page (which you may be reading now) at http://oprofile.sf.net/.

Download

You can download a source tarball or get anonymous CVS at the sourceforge page, http://sf.net/projects/oprofile/.

Mailing list

There is a low-traffic OProfile-specific mailing list, details at http://sf.net/mail/?group_id=16191.

Bug tracker

There is a bug tracker for OProfile at SourceForge, http://sf.net/tracker/?group_id=16191&atid=116191.

IRC channel

Several OProfile developers and users sometimes hang out on channel #oprofile on the freenode network.

First you need to build OProfile and install it. ./configure, make, make install is often all you need, but note these arguments to ./configure :

You'll need to have a configured kernel source for the current kernel to build the module for 2.4 kernels. It is also recommended that if you have a uniprocessor machine, you enable the local APIC / IO_APIC support for your kernel (this is automatically enabled for SMP kernels). On machines with power management, such as laptops, the power management must be turned off when using OProfile with 2.4 kernels. The power management software in the BIOS cannot handle the non-maskable interrupts (NMIs) used by OProfile for data collection. If you use the NMI watchdog, be aware that the watchdog is disabled when profiling starts, and not re-enabled until the OProfile module is removed (or, in 2.5, when OProfile is not running). If you compile OProfile for a 2.2 kernel you must be root to compile the module. If you are using 2.5 kernels or higher, you do not need kernel source, as long as the OProfile driver is enabled; additionally, you should not need to disable power management.

Please note that you must save or have available the vmlinux file generated during a kernel compile, as OProfile needs it (you can use --no-vmlinux, but this will prevent kernel profiling).

In this section we describe the configuration and control of the profiling system with opcontrol in more depth. The opcontrol script has a default setup, but you can alter this with the options given below. In particular, if your hardware supports performance counters, you can configure them. There are a number of counters (for example, counter 0 and counter 1 on the Pentium III). Each of these counters can be programmed with an event to count, such as cache misses or MMX operations. The event chosen for each counter is reflected in the profile data collected by OProfile: functions and binaries at the top of the profiles reflect that most of the chosen events happened within that code.

Additionally, each counter has a "count" value: this corresponds to how detailed the profile is. The lower the value, the more frequently profile samples are taken. A counter can choose to sample only kernel code, user-space code, or both (both is the default). Finally, some events have a "unit mask" - this is a value that further restricts the types of event that are counted. The event types and unit masks for your CPU are listed by opcontrol --list-events.

The opcontrol script provides the following actions :

--init

Loads the OProfile module if required and makes the OProfile driver interface available.

--setup

Followed by list arguments for profiling set up. List of arguments saved in /home/root/.oprofile/daemonrc. Giving this option is not necessary; you can just directly pass one of the setup options, e.g. opcontrol --no-vmlinux.

--start-daemon

Start the oprofile daemon without starting actual profiling. The profiling can then be started using --start. This is useful for avoiding measuring the cost of daemon startup, as --start is a simple write to a file in oprofilefs. Not available in 2.2/2.4 kernels.

--start

Start data collection with either arguments provided by --setup or information saved in /home/root/.oprofile/daemonrc. Specifying the addition --verbose makes the daemon generate lots of debug data whilst it is running.

--dump

Force a flush of the collected profiling data to the daemon.

--stop

Stop data collection (this separate step is not possible with 2.2 or 2.4 kernels).

--shutdown

Stop data collection and remove daemon.

--reset

Clears out data from current session, but leaves saved sessions.

--save=session_name

Save data from current session to session_name.

--deinit

Shuts down daemon. Unload the OProfile module and oprofilefs.

--list-events

List event types and unit masks.

--help

Generate usage messages.

There are a number of possible settings, of which, only --vmlinux (or --no-vmlinux) is required. These settings are stored in ~/.oprofile/daemonrc.

--buffer-size=num

Number of samples in kernel buffer.

--event=[eventspec]

Use the given performance counter event to profile. See Section 1.2 below.

--pid-filter=pid

Only profile the given process PID (only available for 2.4 version). Set to "none" to re-enable profiling of all PIDs.

--pgrp-filter=pgrp

Only profile the given process tty group (only avilable for 2.4 version). Set to "none" to re-enable profiling of all PGRPs.

--separate=[none,library,kernel,all]

By default, every profile is stored in a single file. Thus, for example, samples in the C library are all accredited to the /lib/libc.o profile. However, you choose to create separate sample files by specifying one of the below options.

none No profile separation (default)
library Create per-application profiles for libraries
kernel Create per-application profiles for the kernel and kernel modules
all Both of the above options

Note that --separate=kernel also turns on --separate=library. When using --separate=kernel, samples in hardware interrupts, soft-irqs, or other asynchronous kernel contexts are credited to the task currently running. This means you will see seemingly nonsense profiles such as /bin/bash showing samples for the PPP modules, etc.

On 2.2/2.4 only kernel threads already started when profiling begins are correctly profiled; newly started kernel thread samples are credited to the vmlinux (kernel) profile. On 2.5 there is no kernel thread profiling, all these samples are credited to the vmlinux profile.

--vmlinux=file

vmlinux kernel image.

--no-vmlinux

Use this when you don't have a kernel vmlinux file, and you don't want to profile the kernel. This still counts the total number of kernel samples, but can't give symbol-based results for the kernel or any modules.

The oprof_start application provides a convenient way to start the profiler. Note that oprof_start is just a wrapper around the opcontrol script, so it does not provide more services than the script itself.

After oprof_start is started you can select the event type for each counter; the sampling rate and other related parameters are explained in Section 1. The "Configuration" section allows you to set general parameters such as the buffer size, kernel filename etc. The counter setup interface should be self-explanatory; Section 3.1 and related links contain information on using unit masks.

A status line shows the current status of the profiler: how long it has been running, and the average number of interrupts received per second and the total, over all processors. Note that quitting oprof_start does not stop the profiler.

Your configuration is saved in the same file as opcontrol uses; that is, ~/.oprofile/daemonrc.

Note

Your CPU type may not include the requisite support for hardware performance counters, in which case you must use OProfile in RTC mode in 2.4 (see Section 3.2), or timer mode in 2.5 (see Section 3.3). You do not really need to read this section unless you are interested in using events other than the default event chosen by OProfile.

The hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available from http://developer.intel.com/. The AMD Athlon/Duron implementation is detailed in http://www.amd.com/products/cpg/athlon/techdocs/pdf/22007.pdf. These processors are capable of delivering an interrupt when a counter overflows. This is the basic mechanism on which OProfile is based. The delivery mode is NMI, so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called, the current PC value and the current task are recorded into the profiling structure. This allows the overflow event to be attached to a specific assembly instruction in a binary image. The daemon receives this data from the kernel, and writes it to the sample files.

If we use an event such as CPU_CLK_UNHALTED or INST_RETIRED (GLOBAL_POWER_EVENTS or INSTR_RETIRED, respectively, on the Pentium 4), we can use the overflow counts as an estimate of actual time spent in each part of code. Alternatively we can profile interesting data such as the cache behaviour of routines with the other available counters.

However there are several caveats. First, there are those issues listed in the Intel manual. There is a delay between the counter overflow and the interrupt delivery that can skew results on a small scale - this means you cannot rely on the profiles at the instruction level as being perfectly accurate. If you are using an "event-mode" counter such as the cache counters, a count registered against it doesn't mean that it is responsible for that event. However, it implies that the counter overflowed in the dynamic vicinity of that instruction, to within a few instructions. Further details on this problem can be found in Chapter 5 and also in the Digital paper "ProfileMe: A Hardware Performance Counter".

Each counter has several configuration parameters. First, there is the unit mask: this simply further specifies what to count. Second, there is the counter value, discussed below. Third, there is a parameter whether to increment counts whilst in kernel or user space. You can configure these separately for each counter.

After each overflow event, the counter will be re-initialized such that another overflow will occur after this many events have been counted. Thus, higher values mean less-detailed profiling, and lower values mean more detail, but higher overhead. Picking a good value for this parameter is, unfortunately, somewhat of a black art. It is of course dependent on the event you have chosen. Specifying too large a value will mean not enough interrupts are generated to give a realistic profile (though this problem can be ameliorated by profiling for longer). Specifying too small a value can lead to higher performance overhead.

OK, so the profiler has been running, but it's not much use unless we can get some data out. Fairly often, OProfile does a little too good a job of keeping overhead low, and no data reaches the profiler. This can happen on lightly-loaded machines. Remember you can force a dump at any time with :

opcontrol --dump

Remember to do this before complaining there is no profiling data ! Now that we've got some data, it has to be processed. That's the job of opreport, opannotate, or opgprof.

The opreport utility is the primary utility you will use for getting formatted data out of OProfile. It produces two types of data: image summaries and symbol summaries. An image summary lists the number of samples for individual binary images such as libraries or applications. Symbol summaries provide per-symbol profile data. In the following example, we're getting an image summary for the whole system:

[moz@lambent oprofile]$ opreport --long-filenames
CPU: PIII, speed 863.195 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 23150
   905898 59.7415 /usr/lib/gcc-lib/i386-redhat-linux/3.2/cc1plus
   214320 14.1338 /boot/2.5.69/vmlinux
   103450  6.8222 /lib/i686/libc-2.3.2.so
    60160  3.9674 /usr/local/bin/madplay
    31769  2.0951 /usr/local/oprofile-pp/bin/oprofiled
    26550  1.7509 /usr/lib/libartsflow.so.1.0.0
    23906  1.5765 /usr/bin/as
    18770  1.2378 /oprofile
    15528  1.0240 /usr/lib/qt-3.0.5/lib/libqt-mt.so.3.0.5
    11979  0.7900 /usr/X11R6/bin/XFree86
    11328  0.7471 /bin/bash
    ...

If we had specified --symbols in the previous command, we would have gotten a symbol summary of all the images across the entire system. We can restrict this to only part of the system profile; for example, below is a symbol summary of the OProfile daemon. Note that as we used opcontrol --separate=all, symbols from images that oprofiled has used are also shown.

[moz@lambent oprofile]$ opreport -l `which oprofiled` 2>/dev/null | more
CPU: PIII, speed 863.195 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 23150
vma      samples  %           image name               symbol name
0804be10 14971    28.1993     oprofiled                odb_insert
0804afdc 7144     13.4564     oprofiled                pop_buffer_value
c01daea0 6113     11.5144     vmlinux                  __copy_to_user_ll
0804b060 2816      5.3042     oprofiled                opd_put_sample
0804b4a0 2147      4.0441     oprofiled                opd_process_samples
0804acf4 1855      3.4941     oprofiled                opd_put_image_sample
0804ad84 1766      3.3264     oprofiled                opd_find_image
0804a5ec 1084      2.0418     oprofiled                opd_find_module
0804ba5c 741       1.3957     oprofiled                odb_hash_add_node
...

These are the two basic ways you are most likely to use regularly, but opreport can do a lot more than that. For more details, see Section 5.

The opannotate utility generates annotated source files or assembly listings, optionally mixed with source. If you want to see the source file, the profiled application needs to have debug information, and the source must be available through this debug information. For GCC, you must use the -g option when you are compiling. If the binary doesn't contain sufficient debug information, you can still use opannotate --assembly to get annotated assembly.

Note that for the reason explained in Section 3.1 the results can be inaccurate. The debug information itself can add other problems; for example, the line number for a symbol can be incorrect. Assembly instructions can be re-ordered and moved by the compiler, and this can lead to crediting source lines with samples not really "owned" by this line. Also see Chapter 5.

You can output the annotation to one single file, containing all the source found using the --source. You can use this in conjunction with --assembly to get combined source/assembly output.

You can also output a directory of annotated source files that maintains the structure of the original sources. Each line in the annotated source is prepended with the samples for that line. Additionally, each symbol is annotated giving details for the symbol as a whole. An example:

[moz@lambent moz]$ opannotate --source --output-dir=annotated --source-dir=/home/moz/src/oprofile-pp/ /usr/local/oprofile-pp/bin/oprofiled
[moz@lambent moz]$ ls  annotated/daemon/
opd_cookie.h  opd_image.c  opd_kernel.c  opd_sample_files.c  oprofiled.c

Line numbers are maintained in the source files, but each file has a footer appended describing the profiling details. The actual annotation looks something like this :

...
               :static uint64_t pop_buffer_value(struct transient * trans)
 11510  1.9661 :{ /* pop_buffer_value total:  89901 15.3566 */
               :        uint64_t val;
               :
 10227  1.7469 :        if (!trans->remaining) {
               :                fprintf(stderr, "BUG: popping empty buffer !\n");
               :                exit(EXIT_FAILURE);
               :        }
               :
               :        val = get_buffer_value(trans->buffer, 0);
  2281  0.3896 :        trans->remaining--;
  2296  0.3922 :        trans->buffer += kernel_pointer_size;
               :        return val;
 10454  1.7857 :}
...

The first number on each line is the number of samples, whilst the second is the relative percentage of total samples. There are a number of options for specifying the output; for more details, see Section 6.

All of the analysis tools take a profile specification. This is a set of definitions that describe which actual profiles should be examined. The simplest profile specification is empty: this will match all the available profile files for the current session (this is what happens when you do opreport).

Specification parameters are of the form name:value[,value]. For example, if I wanted to get a combined symbol summary for /bin/myprog and /bin/myprog2, I could do opreport -l image:/bin/myprog,/bin/myprog2. As a special case, you don't actually need to specify the image: part here: anything left on the command line is assumed to be an image: name. Similarly, if no session: is specified, then session:current is assumed ("current" is a special name of the current / last profiling session).

In addition to the comma-separated list shown above, some of the specification parameters can take glob-style values. For example, if I want to see image summaries for all binaries profiled in /usr/bin/, I could do opreport image:/usr/bin/\*. Note the necessity to escape the special character from the shell.

For a profile specification to produce useful data, all selected profiles must match in the basic event details. It's not possible to meaningfully compare profiles from two different counter types. So some parameters like event: do not allow a comma-separated list or glob, as collating results is not possible. This implies you might need to specify these parameters if you've profiled with multiple event types, so that the tools can disambiguate which profiles you actually mean.

session: sessionlist

A comma-separated list of session names to resolve in. Absence of this tag, unlike all others, means "the current session", equivalent to specifying "session:current".

session-exclude: sessionlist

A comma-separated list of sessions to exclude.

image: imagelist

A comma-separated list of image names to resolve. Each entry may be relative path, glob-style name, or full path, e.g.

opreport 'image:/usr/bin/oprofiled,*op*,./oprofpp'
image-exclude: imagelist

Same as image:, but the matching images are excluded.

lib-image: imagelist

Same as image:, but only for images that are for a particular primary binary image (namely, an application). This only makes sense to use if you're using --separate. This includes kernel modules and the kernel when using --separate=kernel.

lib-image-exclude: imagelist

Same as lib-image:, but the matching images are excluded.

event: eventname

The symbolic event name to match on, e.g. event:DATA_MEM_REFS. When using the timer interrupt, the event is always "TIMER".

count: eventcount

The event count to match on, e.g. event:DATA_MEM_REFS count:30000. Note that this value refers the setting used for opcontrol only, and has nothing to do with the sample counts in the profile data itself. When using the timer interrupt, the count is always 0 (indicating it cannot be set).

unit-mask: maskvalue

The unit mask value of the event to match on, e.g. unit-mask:1.

binary: file

Give results only for the given file. This can only be used with sample-file:.

sample-file: file

Give results only for the given sample file. This can only be used with binary:.

--accumulated / -c

Accumulate sample and percentage counts in the symbol list.

--debug-info / -g

Show source file and line for each symbol.

--details / -d

Show per-instruction details for all selected symbols.

--exclude-dependent / -x

Do not include application-specific images for libraries, kernel modules and the kernel. This option only makes sense if the profile session used --separate.

--exclude-symbols / -e [symbols]

Exclude all the symbols in the given comma-separated list.

--global-percent

Make dependent image summary percentages relative to the whole profile, not just the application image.

--help / -? / --usage

Show help message.

--image-path / -p [paths]

Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.5 and upwards.

--include-symbols / -i [symbols]

Only include symbols in the given comma-separated list.

--long-filenames / -l

Output full paths instead of basenames.

--merge / -m [cpu,pid,lib]

Merge any profiles separated in a --separate session.

--no-demangle

Do not attempt to demangle C++ symbols.

--no-header

Don't output a header detailing profiling parameters.

--output-file / -o [file]

Output to the given file instead of stdout.

--reverse-sort / -r

Reverse the sort from the default.

--smart-demangle / -D

Use pattern-matching to make C++ symbol demangling more readable.

--sort / -s [vma,sample,symbol,debug,image]

Sort the list of symbols by, respectively, symbol address, number of samples, symbol name, debug filename and line number, binary image filename.

--symbols / -l

List per-symbol information instead of a binary image summary.

--threshold / -t [percentage]

Only output data for symbols that have more than the given percentage of total samples.

--verbose / -V

Give verbose debugging output.

--version / -v

Show version.

--assembly / -a

Output annotated assembly. If this is combined with --source, then mixed source / assembly annotations are output.

--exclude-dependent / -x

Do not include application-specific images for libraries, kernel modules and the kernel. This option only makes sense if the profile session used --separate.

--exclude-file [files]

Exclude all files in the given comma-separated list of glob patterns.

--exclude-symbols / -e [symbols]

Exclude all the symbols in the given comma-separated list.

--help / -? / --usage

Show help message.

--image-path / -p [paths]

Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.5 and upwards.

--include-file [files]

Only include files in the given comma-separated list of glob patterns.

--include-symbols / -i [symbols]

Only include symbols in the given comma-separated list.

--no-demangle

Do not attempt to demangle C++ symbols.

--objdump-params [params]

Pass the given parameters as extra values when calling objdump.

--output-dir / -o [dir]

Output directory. This makes opannotate output one annotated file for each source file.

--source / -s

Output annotated source. This requires debugging information to be available for the binaries.

--source-dir / -d [dir]

Top source directory for the source files.

--smart-demangle / -D

Use pattern-matching to make C++ symbol demangling more readable.

--threshold / -t [percentage]

Only output data for symbols that have more than the given percentage of total samples.

--verbose / -V

Give verbose debugging output.

--version / -v

Show version.

The standard caveats of profiling apply in interpreting the results from OProfile: profile realistic situations, profile different scenarios, profile for as long as a time as possible, avoid system-specific artifacts, don't trust the profile data too much. Also bear in mind the comments on the performance counters above - you cannot rely on totally accurate instruction-level profiling. However, for almost all circumstances the data can be useful. Ideally a utility such as Intel's VTUNE would be available to allow careful instruction-level analysis; go hassle Intel for this, not me ;)

This is an example of how the latency of delivery of profiling interrupts can impact the reliability of the profiling data. This is pretty much a worst-case-scenario example: these problems are fairly rare.

double fun(double a, double b, double c)
{
 double result = 0;
 for (int i = 0 ; i < 10000; ++i) {
  result += a;
  result *= b;
  result /= c;
 }
 return result;
}

Here the last instruction of the loop is very costly, and you would expect the result reflecting that - but (cutting the instructions inside the loop):

$ opannotate -a -t 10 ./a.out

     88 15.38% : 8048337:       fadd   %st(3),%st
     48 8.391% : 8048339:       fmul   %st(2),%st
     68 11.88% : 804833b:       fdiv   %st(1),%st
    368 64.33% : 804833d:       inc    %eax
               : 804833e:       cmp    $0x270f,%eax
               : 8048343:       jle    8048337

The problem comes from the x86 hardware; when the counter overflows the IRQ is asserted but the hardware has features that can delay the NMI interrupt: x86 hardware is synchronous (i.e. cannot interrupt during an instruction); there is also a latency when the IRQ is asserted, and the multiple execution units and the out-of-order model of modern x86 CPUs also causes problems. This is the same function, with annotation :

$ opannotate -s -t 10 ./a.out

               :double fun(double a, double b, double c)
               :{ /* _Z3funddd total:     572 100.0% */
               : double result = 0;
    368 64.33% : for (int i = 0 ; i < 10000; ++i) {
     88 15.38% :  result += a;
     48 8.391% :  result *= b;
     68 11.88% :  result /= c;
               : }
               : return result;
               :}

The conclusion: don't trust samples coming at the end of a loop, particularly if the last instruction generated by the compiler is costly. This case can also occur for branches. Always bear in mind that samples can be delayed by a few cycles from its real position. That's a hardware problem and OProfile can do nothing about it.

You may see that a function is credited with a certain number of samples, but the listing does not add up to the correct total. To pick a real example :

               :internal_sk_buff_alloc_security(struct sk_buff *skb)
 353 2.342%    :{ /* internal_sk_buff_alloc_security total: 1882 12.48% */
               :
               :        sk_buff_security_t *sksec;
  15 0.0995%   :        int rc = 0;
               :
  10 0.06633%  :        sksec = skb->lsm_security;
 468 3.104%    :        if (sksec && sksec->magic == DSI_MAGIC) {
               :                goto out;
               :        }
               :
               :        sksec = (sk_buff_security_t *) get_sk_buff_memory(skb);
   3 0.0199%   :        if (!sksec) {
  38 0.2521%   :                rc = -ENOMEM;
               :                goto out;
  10 0.06633%  :        }
               :        memset(sksec, 0, sizeof (sk_buff_security_t));
  44 0.2919%   :        sksec->magic = DSI_MAGIC;
  32 0.2123%   :        sksec->skb = skb;
  45 0.2985%   :        sksec->sid = DSI_SID_NORMAL;
  31 0.2056%   :        skb->lsm_security = sksec;
               :
               :      out:
               :
 146 0.9685%   :        return rc;
               :
  98 0.6501%   :}

Here, the function is credited with 1,882 samples, but the annotations below do not account for this. This is usually because of inline functions - the compiler marks such code with debug entries for the inline function definition, and this is where opannotate annotates such samples. In the case above, memset is the most likely candidate for this problem. Examining the mixed source/assembly output can help identify such results.

Furthermore, for some languages the compiler can implicitly generate functions, such as default copy constructors. Such functions are labelled by the compiler as having a line number of 0, which means the source annotation can be confusing.

Depending on your compiler you can fall into the following problem:

struct big_object { int a[500]; };

int main()
{
	big_object a, b;
	for (int i = 0 ; i != 1000 * 1000; ++i)
		b = a;
	return 0;
}

Compiled with gcc 3.0.4 the annotated source is clearly inaccurate:

               :int main()
               :{  /* main total: 7871 100% */
               :        big_object a, b;
               :        for (int i = 0 ; i != 1000 * 1000; ++i)
               :                b = a;
 7871 100%     :        return 0;
               :}

The problem here is distinct from the IRQ latency problem; the debug line number information is not precise enough; again, looking at output of opannoatate -as can help.

               :int main()
               :{
               :        big_object a, b;
               :        for (int i = 0 ; i != 1000 * 1000; ++i)
               : 80484c0:       push   %ebp
               : 80484c1:       mov    %esp,%ebp
               : 80484c3:       sub    $0xfac,%esp
               : 80484c9:       push   %edi
               : 80484ca:       push   %esi
               : 80484cb:       push   %ebx
               :                b = a;
               : 80484cc:       lea    0xfffff060(%ebp),%edx
               : 80484d2:       lea    0xfffff830(%ebp),%eax
               : 80484d8:       mov    $0xf423f,%ebx
               : 80484dd:       lea    0x0(%esi),%esi
               :        return 0;
    3 0.03811% : 80484e0:       mov    %edx,%edi
               : 80484e2:       mov    %eax,%esi
    1 0.0127%  : 80484e4:       cld
    8 0.1016%  : 80484e5:       mov    $0x1f4,%ecx
 7850 99.73%   : 80484ea:       repz movsl %ds:(%esi),%es:(%edi)
    9 0.1143%  : 80484ec:       dec    %ebx
               : 80484ed:       jns    80484e0
               : 80484ef:       xor    %eax,%eax
               : 80484f1:       pop    %ebx
               : 80484f2:       pop    %esi
               : 80484f3:       pop    %edi
               : 80484f4:       leave
               : 80484f5:       ret

So here it's clear that copying is correctly credited with of all the samples, but the line number information is misplaced. objdump -dS exposes the same problem. Note that maintaining accurate debug information for compilers when optimizing is difficult, so this problem is not suprising. The problem of debug information accuracy is also dependent on the binutils version used; some BFD library versions contain a work-around for known problems of gcc, some others do not. This is unfortunate but we must live with that, since profiling is pointless when you disable optimisation (which would give better debugging entries).

One of the major design criteria for OProfile was low overhead. In many cases profiling is hardly noticeable in terms of overhead (I regularly leave it turned on all the time). It achieves this by judicious use of kernel-side data structures to reduce the collection overhead to a bare runtime minimum. There are several things that unfortunately complicate the issue, so there are cases where the overhead is noticeable.

The worst-case scenario is where there are many short-lived processes. This can be seen in a kernel compile, for instance. Even in this worst case overhead is low compared to other profilers; only very detailed profiling of these workloads has an overhead of higher than 5%. Actual performance data is presented in CVS. In fact most situations have much fewer numbers of processes, leading to far better performance.

Some graphs of performance characteristics of OProfile are available on the website - see Section 3.

Thanks to (in no particular order) : Arjan van de Ven, Rik van Riel, Juan Quintela, Philippe Elie, Phillipp Rumpf, Tigran Aivazian, Alex Brown, Alisdair Rawsthorne, Bob Montgomery, Ray Bryant, H.J. Lu, Jeff Esper, Will Cohen, Graydon Hoare, Cliff Woolley, Alex Tsariounov, Al Stone, Jason Yeh, Randolph Chung, Anton Blanchard, Richard Henderson, Andries Brouwer, Bryan Rittmeyer, Richard Reich (rreich@rdrtech.com), Dave Jones, Charles Filtness; and finally Pulp, for "Intro".