--- name: "bpftrace.bt" section: "8" platform: "linux" package: "bpftrace" version: "0.8+git60-gccac69c2239b-2" release: "Debian 10 (Buster)" platforms: ["linux"] url: "https://new.carta.tech/bpftrace.bt" --- # bpftrace.bt(8) the eBPF tracing language & frontend ## NAME **BPFtrace** - the eBPF tracing language & frontend ## SYNOPSIS bpftrace \[*OPTIONS*\] *FILE*\ bpftrace \[*OPTIONS*\] -e ´program code´ ## DESCRIPTION BPFtrace is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent Linux kernels (4.x). BPFtrace uses: - • *LLVM* as a backend to compile scripts to BPF-bytecode - • *BCC* for interacting with the Linux BPF system - As well as the existing Linux tracing capabilities: | | | | | --- | --- | --- | | | kernel | userland | | static | *tracepoints* | USDT\* probes | | dynamic | *kprobes* | uprobes | \*USDT = user-level statically defined tracing The BPFtrace language is inspired by awk and C, and predecessor tracers such as DTrace and SystemTap. See **EXAMPLES** and **ONELINERS** if you are impatient.\ See **PROBE TYPES** and **BUILTINS (variables/functions)** for the bpftrace language elements. ## OPTIONS - **-l \[searchterm\]** List probes. - **-e ´PROGRAM´** Execute PROGRAM. - **-p PID** Enable USDT probes on PID. Will terminate bpftrace on PID termination. Note this is not a global PID filter on probes. - **-c CMD** Helper to run CMD. Equivalent to manually running CMD and then giving passing the PID to -p. This is useful to ensure you've traced at least the duration CMD's execution. - **-v** Verbose messages. - **-d** Debug info on dry run. - **-dd** Verbose debug info on dry run. ## EXAMPLES - **bpftrace -l ´\*sleep\*´** List probes containing "sleep". - **bpftrace -e ´kprobe:do\_nanosleep { printf("PID %d sleeping\\n", pid); }´** Trace processes calling sleep. - **bpftrace -c ´sleep 5´ -e ´kprobe:do\_nanosleep { printf("PID %d sleeping\\n", pid); }´** run "sleep 5" in a new process and then trace processes calling sleep. - **bpftrace -e ´tracepoint:raw\_syscalls:sys\_enter { @\[comm\]=count(); }´** Count syscalls by process name. ## ONELINERS For brevity, just the the actual BPF code is shown below.\ Usage: **bpftrace -e ´bpf-code´** - New processes with arguments: **tracepoint:syscalls:sys\_enter\_execve { join(args-\>argv); }** - Files opened by process: **tracepoint:syscalls:sys\_enter\_open { printf("%s %s\\n", comm, str(args-\>filename)); }** - Syscall count by program: **tracepoint:raw\_syscalls:sys\_enter { @\[comm\] = count(); }** - Syscall count by syscall: **tracepoint:syscalls:sys\_enter\_\* { @\[probe\] = count(); }** - Syscall count by process: **tracepoint:raw\_syscalls:sys\_enter { @\[pid, comm\] = count(); }** - Read bytes by process: **tracepoint:syscalls:sys\_exit\_read /args-\>ret/ { @\[comm\] = sum(args-\>ret); }** - Read size distribution by process: **tracepoint:syscalls:sys\_exit\_read { @\[comm\] = hist(args-\>ret); }** - Disk size by process: **tracepoint:block:block\_rq\_issue { printf("%d %s %d\\n", pid, comm, args-\>bytes); }** - Pages paged in by process: **software:major-faults:1 { @\[comm\] = count(); }** - Page faults by process: **software:faults:1 { @\[comm\] = count(); }** - Profile user-level stacks at 99 Hertz, for PID 189: **profile:hz:99 /pid == 189/ { @\[ustack\] = count(); }** ## PROBE TYPES ### KPROBES Attach a BPFtrace script to a kernel function, to be executed when that function is called: **kprobe:vfs\_read { ... }** ### UPROBES Attach script to a userland function: **uprobe:/bin/bash:readline { ... }** ### TRACEPOINTS Attach script to a statically defined tracepoint in the kernel: **tracepoint:sched:sched\_switch { ... }** Tracepoints are guaranteed to be stable between kernel versions, unlike kprobes. ### SOFTWARE Attach script to kernel software events, executing once every provided count or use a default: **software:faults:100** **software:faults:** ### HARDWARE Attach script to hardware events (PMCs), executing once every provided count or use a default: **hardware:cache-references:1000000** **hardware:cache-references:** ### PROFILE Run the script on all CPUs at specified time intervals: **profile:hz:99 { ... }** **profile:s:1 { ... }** **profile:ms:20 { ... }** **profile:us:1500 { ... }** ### INTERVAL Run the script once per interval, for printing interval output: **interval:s:1 { ... }** **interval:ms:20 { ... }** ### MULTIPLE ATTACHMENT POINTS A single probe can be attached to multiple events: **kprobe:vfs\_read,kprobe:vfs\_write { ... }** ### WILDCARDS Some probe types allow wildcards to be used when attaching a probe: **kprobe:vfs\_\* { ... }** ### PREDICATES Define conditions for which a probe should be executed: **kprobe:sys\_open / uid == 0 / { ... }** ## BUILTINS The following variables and functions are available for use in bpftrace scripts: ### VARIABLES - **pid** Process ID (kernel tgid) - **tid** Thread ID (kernel pid) - **cgroup** Cgroup ID of the current process - **uid** User ID - **gid** Group ID - **nsecs** Nanosecond timestamp - **cpu** Processor ID - **comm** Process name - **stack** Kernel stack trace - **ustack** User stack trace - **arg0**, **arg1**, ... etc. Arguments to the function being traced - **retval** Return value from function being traced - **func** Name of the function currently being traced - **probe** Full name of the probe - **curtask** Current task\_struct as a u64. - **rand** Random number of type u32. ### FUNCTIONS - **hist(int n)** Produce a log2 histogram of values of **n** - **lhist(int n, int min, int max, int step)** Produce a linear histogram of values of **n** - **count()** Count the number of times this function is called - **sum(int n)** Sum this value - **min(int n)** Record the minimum value seen - **max(int n)** Record the maximum value seen - **avg(int n)** Average this value - **stats(int n)** Return the count, average, and total for this value - **delete(@x)** Delete the map element passed in as an argument - **str(char \*s)** Returns the string pointed to by **s** - **printf(char \*fmt, ...)** Print formatted to stdout - **print(@x\[, int top \[, int div\]\])** Print a map, with optional top entry count and divisor - **clear(@x)** Delete all key/values from a map - **sym(void \*p)** Resolve kernel address - **usym(void \*p)** Resolve user space address - **kaddr(char \*name)** Resolve kernel symbol name - **uaddr(char \*name)** Resolve user space symbol name - **reg(char \*name)** Returns the value stored in the named register - **join(char \*arr\[\])** Prints the string array - **time(char \*fmt)** Print the current time - **system(char \*fmt)** Execute shell command - **exit()** Quit bpftrace ## FURTHER READING The official documentation can be found here:\ https://github.com/iovisor/bpftrace/blob/master/docs ## HISTORY The first official talk by Alastair on bpftrace happened at the Tracing Summit in Edinburgh, Oct 25th 2018. ## AUTHOR Created by Alastair Robertson.\ Manpage by Stephan Schuberth. ## SEE ALSO **man -k bcc**, after having installed the *bpfcc-tools* package under Ubuntu. ## CONTRIBUTING Prior to contributing new tools, read the official checklist at:\ https://github.com/iovisor/bpftrace/blob/master/CONTRIBUTING-TOOLS.md