---
name: "funcslower-bpfcc"
section: "8"
platform: "linux"
package: "bpfcc-tools"
version: "0.31.0+ds-7"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/funcslower-bpfcc"
---

# funcslower-bpfcc(8)

Trace slow kernel or user function calls.

## NAME

funcslower - Trace slow kernel or user function calls.

## SYNOPSIS

**funcslower \[-hf\] \[-p PID\] \[-U \| -K\] \[-m MIN\_MS\] \[-u MIN\_US\] \[-a ARGUMENTS\] \[-T\] \[-t\] \[-v\] function \[function ...\]**

## DESCRIPTION

This script traces a kernel or user function's entry and return points, and prints a message when the function's latency exceeded the specified threshold. Multiple functions are supported, and you can mix kernel functions with user functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS

CONFIG\_BPF and bcc.

## OPTIONS

\-p PID Trace this PID only.

- \-m MIN\_NS

  Minimum duration to trace, in milliseconds. Default is 1 ms.

- \-u MIN\_US

  Minimum duration to trace, in microseconds.

- \-a ARGUMENTS

  Print the function's arguments, up to 6.

- \-T

  Print a HH:MM:SS timestamp with each entry.

- \-t

  Print a seconds timestamp with each entry, at microsecond resolution.

- \-f

  Print output in folded stack format.

- \-U

  Show stacks from user space only (no kernel space stacks).

- \-K

  Show stacks from kernel space only (no user space stacks).

- \-v

  Print the resulting BPF program, for debugging purposes.

- function

  The function to trace -- multiple functions are supported. If a plain function name is provided, the function is assumed to be a kernel function. For user functions, provide the library name and the function name, e.g. bash:readline or c:malloc.

## EXAMPLES

- Trace vfs\_write calls slower than 1ms:

  \# **funcslower vfs\_write**

- Trace open() calls in libc slower than 10us:

  \# **funcslower -u 10 c:open**

- Trace both malloc() and free() slower than 10us, in pid 135 only:

  \# **funcslower -p 135 -u 10 c:malloc c:free**

- Trace the write syscall and print its first 4 arguments:

  \# **funcslower -a 4 SyS\_write**

- Trace opens from libc and print the user and kernel stack frames:

  \# **funcslower -UK c:open**

## FIELDS

- TIME

  Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds at microsecond-accuracy from the first event seen.

- COMM

  Process name.

- PID

  Process ID.

- LAT

  Latency of the operation in either microseconds (us) or milliseconds (ms).

- RVAL

  The return value from the function. Often useful for diagnosing a relationship between slow and failed function calls.

- FUNC

  The function name, followed by its arguments if requested.

## OVERHEAD

Depending on the function(s) being traced, overhead can become severe. For  example, tracing a common function like malloc() can slow down a C/C++ program by a factor of 2 or more. On the other hand, tracing a low-frequency event like the SyS\_setreuid() function will probably not be as prohibitive, and in fact negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation,  because they summarize data in-kernel and have a much lower overhead than this tool. To get a general idea of the number of times a particular function is called (and estimate the overhead), use the funccount tool, e.g.:

\# funccount c:open

## SOURCE

This is from bcc.

-

  https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion \_examples.txt file containing example usage, output, and commentary for this tool.

## OS

Linux

## STABILITY

Unstable - in development.

## AUTHOR

Sasha Goldshtein

## SEE ALSO

[funccount(8)](/funccount/8.md), funclatency(8), argdist(8), [trace(8)](/trace/8.md)