---
name: "lket"
section: "5"
platform: "linux"
package: "systemtap"
version: "0.0.20061028-2"
release: "Debian 4.0 (Etch)"
platforms: ["linux"]
url: "https://new.carta.tech/lket"
---

# lket(5)

Linux Kernel Event Trace tool based on SystemTap

## NAME

LKET - Linux Kernel Event Trace tool based on SystemTap

## DESCRIPTION

The Linux Kernel Event Trace (LKET) tool is an extension to the tapsets library available on SystemTap. Its goal is to utilize the dynamic probing capabilities provided through SystemTap to create a set of standard hooks that probe pre-defined places in the kernel. It can be used to collect important information that can be used as a starting point to analyze  a performance problem in the system.

The LKET tapsets are designed to only trace the events selected by the user. Once the data has been collected, it is then post-processed  according to the need of the user. Trace data can be processed in  various different ways to generate simple to complex reports.

## EVENT HOOKS

The following sections enumerate the variety of event hooks implemented in LKET and their trace data format. The trace data generated by different event hooks contain common data as well as some data specific to that event hook.

the INT8, INT16, INT32, INT64 and STRING appeared in trace data format represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated string respectively.

The data common(i.e. *common\_data* in the following subsecions) to all event hooks is:

**timestamp(INT64),(tid\<\<32 \| groupID\<\<24 \| hookID\<\<16 \| cpu\_id\<\<8)(INT64)**

Each event hook group is a collection of those hooks that have similarities of what they could trace. And the ID of each event hook (HookID) is defined in the context of its corresponding group.

### EVENT REGISTER

Event register is not actually an event. It is used to log the  metadata of the trace data, including the extra trace data appended by user. See **EVENT REGISTER** and **CUSTOMIZED TRACE DATA** for more details.

- **register\_sys\_event**

  This is a function used to register event hooks available in LKET.  It should be called from register\_event.stp:register\_sys\_events().

- **register\_user\_event**

  This is a function used to log the metadata of the extra trace data appended by user for a specific event. It should be called in the probe *register\_event*

### SYSTEM CALLS

You could use  *addevent.syscall* to trace the entry and return of all system calls. It contains two sub event hooks:

- **addevent.syscall.entry**

  Trace entry of all system calls.

  Data format is:

  *common\_data, syscall\_name(STRING)*

- **addevent.syscall.return**

  Trace return of all system calls.

  Data format is:

  *common\_data, syscall\_name(STRING)*

### PROCESS CREATION

This group contains three sub event hooks. You could use *addevent.process* to trace fork and execve of processes(note that process\_snapshot() won't be included).

- **process\_snapshot()**

  This event hook isn't a probe definition but a function. It is called by LKET silently to take a snapshot of all running processes.

  Data format is:

  *common\_data, tid(INT32), pid(INT32), ppid(INT32), process\_name(STRING)*

- **addevent.process.fork**

  Trace fork of processes

  Data format is:

  *common\_data, new\_tid(INT32), new\_pid(INT32), ppid(INT32)*

- **addevent.process.execve**

  Trace execve of new processes

  Data format is:

  *common\_data, pid(INT32), new\_process\_name(STRING)*

### SIGNAL

You could use *addevent.signal* to trace signal activities. It contains the following events:

- **addevent.signal.send.entry**

  Trace when a signal is sent to a process

  Data format is: *common\_data, sig(INT8), shared(INT8), send2queue(INT8), pid(INT32)*

- **addevent.signal.send.return**

  Trace when returning from sending signal

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.syskill.entry**

  Trace when sys\_kill is called to send a signal to a process.

  Data format is: *common\_data, pid(INT32), sig(INT8)*

- **addevent.signal.syskill.return**

  Trace when return from sys\_kill

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.systgkill.entry**

  Trace when sys\_tgkill is called to send a signal to one specific thread

  Data format is: *common\_data, tid(INT32), pid(INT32), sig(INT8)*

- **addevent.signal.systgkill.return**

  Trace when returning from sys\_tgkill

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.systkill.entry**

  Trace when sys\_tkill is called to send a signal to a single process.

  Data format is: *common\_data, pid(INT32), sig(INT8)*

- **addevent.signal.systkill.return**

  Trace when returning from sys\_tkill.

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.pending.entry**

  Trace when examine the set of signals that are pending for delivery.

  Data format is: *common\_data, sigset\_addr(INT32), setsize(INT32)*

- **addevent.signal.pending.return**

  Trace when returning from signal.pending

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.do\_action.entry**

  Trace when a thread is about to examine and change a signal action

  Data format is: *common\_data, sig(INT8), handler(INT64)*

- **addevent.signal.do\_action.return**

  Trace when returning from signal.do\_action

  Data format is: *common\_data, return(INT8)*

- **addevent.signal.procmask.entry**

  Trace when a thread is about to examine and change blocked signals

  Data format is: *common\_data, how(INT8), sigset(INT64)*

- **addevent.signal.procmask.return**

  Trace when returning from signal.procmask

  Data format is *common\_data, return(INT8)*

- **addevent.signal.flush.entry**

  Trace when flush all pending signals for a task

  Data format is: *common\_data, pid(INT32)*

### IO SCHEDULER ACTIVITIES

You could use *addevent.ioscheduler* to trace the IO scheduler activities. It contains three sub event hooks:

- **addevent.ioscheduler.elv\_add\_request**

  Trace when a request is added to the request queue

  Data format is:

  *common\_data, elevator\_name(STRING), disk\_major(INT8), disk\_minor(INT8),* *request\_addr(INT64), request\_flags(INT64)*

- **addevent.ioscheduler.elv\_next\_request.entry**

  Trace when try to retrieve a request from request queue

  Data format is:

  *common\_data, elevator\_name(STRING)*

- **addevent.ioscheduler.elv\_next\_request.return**

  Trace when return from retrieving a request from request queue

  Data format is:

  *common\_data, disk\_major(INT8), disk\_minor(INT8),* *request\_addr(INT64), request\_flags(INT64)*

- **addevent.ioscheduler.elv\_completed\_request**

  Trace when a request is completed

  Data format is:

  *common\_data, elevator\_name(STRING), disk\_major(INT8), disk\_minor(INT8),* *request\_addr(INT64), request\_flags(INT64)*

### TASK SCHEDULE ACTIVITIES

You could use *addevent.tskdispatch* to trace the task scheduler activities. It contains two sub event hooks:

- **addevent.tskdispatch.ctxswitch**

  Trace the process context switch

  Data format is:

  *common\_data, prev\_pid(INT32), next\_pid(INT32), prev\_state(INT8)*

- **addevent.tskdispatch.cpuidle**

  Trace when cpu goes idle

  Data format is:

  *common\_data, current\_pid(INT32)*

### SCSI ACTIVITIES

You could use *addevent.scsi* to trace the scsi layer activities. It contains four sub event hooks:

- **addevent.scsi.ioentry**

  mid-layer prepares a IO request

  Data format is:

  *common\_data, disk\_major(INT8), disk\_minor(INT8), device\_state(INT8), request\_addr(INT64)*

- **addevent.scsi.iodispatching**

  Dispatch a command to the low-level driver

  Data format is:

  *common\_data, host(INT8), channel(INT8), lun(INT8), dev\_id(INT8),* *device\_state(INT8), data\_direction(INT8), reqbuf\_addr(INT64),* *reqbuf\_len(INT32), request\_addr(INT64)*

- **addevent.scsi.iodone**

  I/O is done by low-level driver

  Data format is:

  *common\_data, host(INT8), channel(INT8), lun(INT8), dev\_id(INT8),* *device\_state(INT8), data\_direction(INT8), request\_addr(INT64)*

- **addevent.scsi.iocompleted**

  mid-layer processed the completed IO

  Data format is:

  *common\_data, host(INT8), channel(INT8), lun(INT8), dev\_id(INT8),* *device\_state(INT8), data\_direction(INT8), request\_addr(INT64),* *bytes\_done(INT32)*

### PAGE FAULT

You could use  *addevent.pagefault* to trace page fault events. It contains only one sub event hooks:

- **addevent.pagefault**

  Data format is:

  *common\_data, memory\_address(INT64), write\_access(INT8)*

### NETWORK DEVICE ACTIVITIES

You could use *addevent.netdev* to trace the network device  activities. It contains two sub event hooks:

- **addevent.netdev.receive**

  network device receives a packet

  Data format is:

  *common\_data, netdev\_name(STRING), data\_length(INT32), protocol(INT16),* *buffer\_length(INT32)*

- **addevent.netdev.transmit**

  A packet will be sent out by network device

  Data format is:

  *common\_data, netdev\_name(STRING), data\_length(INT32), protocol(INT16),* *buffer\_length(INT32)*

### IO SYSCALLS

You could use *addevent.iosyscall* to trace the detail activities of io related system calls.  It contains 16 entry hooks and 16 corresponding return hooks.

All the return hooks will only log the common\_data and the return value. So in the following subsections, only the entry  hooks will be listed:

- **addevent.iosyscall.open.entry**

  the entry of sys\_open

  Data format is:

  *common\_data, filename(STRING), flags(INT32), mode(INT32)*

- **addevent.iosyscall.close.entry**

  the entry of sys\_close

  Data format is:

  *common\_data, fd(INT64)*

- **addevent.iosyscall.read.entry**

  the entry of sys\_read

  Data format is:

  *common\_data, fd(INT64), buf\_addr(INT64), count(INT64)*

- **addevent.iosyscall.write.entry**

  the entry of sys\_write

  Data format is:

  *common\_data, fd(INT64), buf\_addr(INT64), count(INT64)*

- **addevent.iosyscall.readv.entry**

  the entry of sys\_readv

  Data format is:

  *common\_data, fd(INT64), vector\_addr(INT64), count(INT64)*

- **addevent.iosyscall.writev.entry**

  the entry of sys\_writev

  Data format is:

  *common\_data, fd(INT64), vector\_addr(INT64), count(INT64)*

- **addevent.iosyscall.pread64.entry**

  the entry of sys\_pread64

  Data format is:

  *common\_data, fd(INT64), buff\_addr(INT64), count(INT64), offset(INT64)*

- **addevent.iosyscall.pwrite64.entry**

  the entry of sys\_pwrite64

  Data format is:

  *common\_data, fd(INT64), buff\_addr(INT64), count(INT64), offset(INT64)*

- **addevent.iosyscall.readahead.entry**

  the entry of sys\_readahead

  Data format is:

  *common\_data, fd(INT64), offset(INT64), count(INT64)*

- **addevent.iosyscall.senfile.entry**

  the entry of sys\_sendfile and sys\_sendfile64

  Data format is:

  *common\_data, out\_fd(INT64), in\_fd(INT64), offset\_uaddr(INT64), count(INT64)*

- **addevent.iosyscall.lseek.entry**

  the entry of sys\_lseek

  Data format is:

  *common\_data, fd(INT64), offset(INT64), whence(INT8)*

- **addevent.iosyscall.llseek.entry**

  the entry of sys\_llseek

  Data format is:

  *common\_data, fd(INT64), offset\_high(INT64), offset\_low(INT64),* *result\_addr(INT64), whence(INT8)*

- **addevent.iosyscall.sync.entry**

  the entry of sys\_sync

  Data format is:

  *common\_data*

- **addevent.iosyscall.fsync.entry**

  the entry of sys\_fsync

  Data format is:

  *common\_data, fd(INT64)*

- **addevent.iosyscall.fdatasync.entry**

  the entry of sys\_fdatasync

  Data format is:

  *common\_data, fd(INT64)*

- **addevent.iosyscall.flock.entry**

  the entry of sys\_flock

  Data format is:

  *common\_data, fd(INT64), operation(INT32)*

### Asynchronous IO

You could use *addevent.aio* to trace the detail activities of AIO related calls(most of them are AIO system calls).  It contains 6 entry hooks and 6 corresponding return hooks.

All the return hooks will only log the common\_data and the return value. So in the following subsections, only the entry  hooks will be listed:

- **addevent.aio.io\_setup.entry**

  Fired by calling io\_setup from user space. The corresponding system call is sys\_io\_setup, which will create an aio\_context capable of receiving at least maxevents.

  Data format is:

  *common\_data, nr\_events(INT32), ctxp\_uaddr(INT64)*

- **addevent.aio.io\_submit.entry**

  Fired by calling io\_submit from user space. The corresponding system call is sys\_io\_submit which will queue the nr iocbs  pointed to by iocbpp\_uaddr for processing.

  Data format is:

  *common\_data, ctx\_id(INT64), nr(INT32), iocbpp\_uaddr(INT64)*

- **addevent.aio.io\_submit\_one.entry**

  Called by sys\_io\_submit. It will iterate iocbpp and process them one by one

  Data format is:

  *common\_data, ctx(INT64), user\_iocb\_uaddr(INT64), aio\_lio\_opcode(INT16),* *aio\_reqprio(INT16), aio\_fildes(INT32), aio\_buf(INT64), aio\_nbytes(INT64),* *aio\_offset(INT64)*

- **addevent.aio.io\_getevents.entry**

  Fired by calling io\_getevents from user space. The corresponding system call is sys\_io\_getevents, which will attempt to read at least min\_nr events and up to nr events from  the completion queue for the aio\_context specified by ctx\_id.

  Data format is:

  *common\_data, ctx\_id(INT64), min\_nr(INT32), nr(INT32), events\_uaddr(INT64),* *tv\_sec(INT32), tv\_nsec(INT32)*

- **addevent.aio.io\_destroy.entry**

  Fired by calling io\_destroy from user space. The corresponding system call is sys\_io\_destroy, which will destroy the aio\_context specified.

  Data format is:

  *common\_data, ctx(INT64)*

- **addevent.aio.io\_cancel.entry**

  Fired by calling io\_cancel from user space. The corresponding system call is sys\_io\_cancel, which will attempt to cancel an  iocb previously passed to io\_submit.

  Data format is:

  *common\_data, ctx\_id(INT64), iocb\_uaddr(INT64), result\_uaddr(INT64)*

### SUNRPC

You could use *addevent.sunrpc* to trace the details of SUNRPC activities. It is now divided into three groups: high-level client operation event hooks (addevent.sunrpc.clnt), high-level server operation event hooks (addevent.sunrpc.svc) and RPC scheduler operation event hooks (addevent.sunrpc.sched).  It contains 19 entry hooks and 19 corresponding return hooks.

All the return hooks will only log the common\_data and the return value. So in the following subsections, only the entry hooks will be listed:

- **addevent.sunrpc.clnt.create\_client.entry**

  Fires when an RPC client is to be created

  Data format is:

  *common\_data, servername(STRING), prog(INT64), vers(INT8),* *prot(INT16), port(INT16), authflavor(INT8)*

- **addevent.sunrpc.clnt.clone\_client.entry**

  Fires when the RPC client structure is to be cloned

  Data format is:

  *common\_data, servername(STRING), prog(INT64), vers(INT8),* *prot(INT16), port(INT16), authflavor(INT8)*

- **addevent.sunrpc.clnt.shutdown\_client.entry**

  Fires when an RPC client is to be shut down

  Data format is:

  *common\_data, servername(STRING), prog(INT64), clones(INT16),* *tasks(INT16), rpccnt(INT32)*

- **addevent.sunrpc.clnt.bind\_new\_program.entry**

  Fires when a new RPC program is to be bound an existing client

  Data format is:

  *common\_data, servername(STRING), old\_prog(INT64), old\_vers(INT8),* *prog(INT64), vers(INT8)*

- **addevent.sunrpc.clnt.call\_sync.entry**

  Fires when an RPC procedure is to be called synchronously

  Data format is:

  *common\_data, servername(STRING), prog(INT64), vers(INT8),* *proc(INT64), flags(INT64)*

- **addevent.sunrpc.clnt.call\_async.entry**

  Fires when an RPC procedure is to be called asynchronously

  Data format is:

  *common\_data, servername(STRING), prog(INT64), vers(INT8),* *proc(INT64), flags(INT64)*

- **addevent.sunrpc.clnt.restart\_call.entry**

  Fires when want to restart a task

  Data format is:

  *common\_data, tk\_pid(INT64), tk\_flags(INT64)*

- **addevent.sunrpc.svc.register.entry**

  Fires when an RPC service is to be registered with the local portmapper

  Data format is:

  *common\_data, sv\_name(STRING), prog(INT64), prot(INT16),* *port(INT32)*

- **addevent.sunrpc.svc.create.entry**

  Fires when an RPC service is to be created

  Data format is:

  *common\_data, prog(INT64), pg\_nvers(INT8), bufsize(INT32)*

- **addevent.sunrpc.svc.destroy.entry**

  Fires when an RPC service is to be destroyed

  Data format is:

  *common\_data, sv\_name(STRING), sv\_prog(INT64), sv\_nrthreads(INT32)*

- **addevent.sunrpc.svc.process.entry**

  Fires when an RPC request is to be processed

  Data format is:

  *common\_data, sv\_name(STRING), sv\_prog(INT64), peer\_ip(INT64),* *rq\_xid(INT64), rq\_prog(INT64), rq\_vers(INT8), rq\_proc(INT8)*

- **addevent.sunrpc.svc.authorise.entry**

  Fires when an RPC request is to be authorised

  Data format is:

  *common\_data, sv\_name(STRING), peer\_ip(INT64), rq\_xid(INT64),* *rq\_prog(INT64), rq\_vers(INT8), rq\_proc(INT64)*

- **addevent.sunrpc.svc.recv.entry**

  Fires when receiving the next request on any socket

  Data format is:

  *common\_data, sv\_name(STRING), timeout(INT64)*

- **addevent.sunrpc.svc.send.entry**

  Fires when want to return reply to the client

  Data format is:

  *sv\_name(STRING), peer\_ip(INT64), rq\_xid(INT64), rq\_prog(INT64),* *rq\_vers(INT8), rq\_proc(INT64)*

- **addevent.sunrpc.svc.drop.entry**

  Fires when a request is to be dropped

  Data format is:

  *common\_data, sv\_name(STRING), peer\_ip(INT64), rq\_xid(INT64),* *rq\_prog(INT64), rq\_vers(INT8), rq\_proc(INT64)*

- **addevent.sunrpc.sched.new\_task.entry**

  Fires when creating a new task for the specified client

  Data format is:

  *common\_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64),* *flags(INT64)*

- **addevent.sunrpc.sched.release\_task.entry**

  Fires when releasing a task

  Data format is:

  *common\_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64),* *flags(INT64)*

- **addevent.sunrpc.sched.execute.entry**

  Fires when an RPC request is to be executed

  Data format is:

  *common\_data, xid(INT64), prog(INT64), vers(INT8), prot(INT64),* *tk\_pid(INT64), tk\_flags(INT64)*

- **addevent.sunrpc.sched.delay.entry**

  Fires when want to delay an RPC request

  Data format is:

  *common\_data, xid(INT64), prog(INT64), tk\_pid(INT64),* *tk\_flags(INT64), delay(INT64)*

### NFS

You could use *addevent.nfs* to trace the detail activities of nfs on client side.  It divided into three groups: nfs file operation event hooks(addevent.nfs.fop), nfs address space operation event hooks(addevent.nfs.aop), nfs proc event hooks(addevent.nfs.proc). It contains 36 entry hooks and 33 corresponding return hooks

All the return hooks will only log the common\_data and the return value. So in the following subsections, only the entry  hooks will be listed:

- **addevent.nfs.fop.llseek.entry**

  the entry of nfs\_file\_llseek

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *offset(INT32), origin(INR8)*

- **addevent.nfs.fop.read.entry**

  the entry of do\_sync\_read

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *buf\_addr(INT64), count(INT64) , offset(INT64)*

- **addevent.nfs.fop.write.entry**

  the entry of do\_sync\_write

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *buf\_addr(INT64), count(INT64) , offset(INT64)*

- **addevent.nfs.fop.aio\_read.entry**

  the entry of nfs\_file\_read

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *buf\_addr(INT64), count(INT64) , offset(INT64)*

- **addevent.nfs.fop.aio\_write.entry**

  the entry of nfs\_file\_read

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *buf\_addr(INT64), count(INT64) , offset(INT64)*

- **addevent.nfs.fop.mmap.entry**

  the entry of nfs\_file\_mmap

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *vm\_start(INT64), vm\_end(INT64) , vm\_flags(INT32)*

- **addevent.nfs.fop.open.entry**

  the entry of nfs\_file\_open

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *flag(INT32), filename(STRING)*

- **addevent.nfs.fop.flush.entry**

  the entry of nfs\_file\_flush

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *ndirty(INT32)*

- **addevent.nfs.fop.release.entry**

  the entry of nfs\_file\_release

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *mode(INT16)*

- **addevent.nfs.fop.fsync.entry**

  the entry of nfs\_fsync

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *ndirty(INT32)*

- **addevent.nfs.fop.lock.entry**

  the entry of nfs\_lock

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *fl\_start(INT64), fl\_end(INT64), fl\_type(INT8), fl\_flag(INT8), cmd(INT32)*

- **addevent.nfs.fop.sendfile.entry**

  the entry of nfs\_file\_sendfile

  Data format is:

  *common\_data, major\_device(INT8), minor\_devide(INT8), fileid(INT32),* *count(INT64), ppos(INT64)*

- **addevent.nfs.fop.checkflags.entry**

  the entry of nfs\_check\_flags

  Data format is:

  *flag(INT32)*

- **addevent.nfs.aop.readpage.entry**

  the entry of nfs\_readpage

  Data format is:

  *fileid(INT64), rsize(INT32), page\_address(INT64), page\_index(INT64)*

- **addevent.nfs.aop.readpages.entry**

  the entry of nfs\_readpages

  Data format is:

  *fileid(INT64), rpages(INT32), nr\_pages(INT32)*

- **addevent.nfs.aop.writepage.entry**

  the entry of nfs\_writepage

  Data format is:

  *fileid(INT64), wsize(INT32), page\_address(INT64), page\_index(INT64)*

- **addevent.nfs.aop.writepages.entry**

  the entry of nfs\_writepages

  Data format is:

  *fileid(INT64), wpages(INT32), nr\_to\_write(INT64)*

- **addevent.nfs.aop.prepare\_write.entry**

  the entry of nfs\_prepare\_write

  Data format is:

  *fileid(INT64), page\_address(INT64), page\_index(INT64)*

- **addevent.nfs.aop.commit\_write.entry**

  the entry of nfs\_commit\_write

  Data format is:

  *fileid(INT64), page\_address(INT64), page\_index(INT64),offset(INT32),count(INT32)*

- **addevent.nfs.aop.set\_page\_dirty.entry**

  the entry of \_\_set\_page\_dirty\_nobuffers

  Data format is:

  *page\_address(INT64), page\_flag(INT8)*

- **addevent.nfs.aop.release\_page.entry**

  the entry of nfs\_release\_page

  Data format is:

  *page\_address(INT64), page\_index(INT64)*

- **addevent.nfs.proc.lookup.entry**

  the entry of nfs\_proc\_lookup , nfs3\_proc\_lookup and nfs4\_proc\_lookup

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *filename(STRING)*

- **addevent.nfs.proc.read.entry**

  the entry of nfs\_proc\_read, nfs3\_proc\_read and nfs4\_proc\_read

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *count(INT32),offset(INT64)*

- **addevent.nfs.proc.write.entry**

  the entry of nfs\_proc\_write, nfs3\_proc\_write and nfs4\_proc\_write

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *count(INT32),offset(INT64)*

- **addevent.nfs.proc.commit.entry**

  Fires when client writes the buffered data to disk,the buffered data is asynchronously written by client before . The commit function works in sync way,not exist in NFSV2

  the entry of  nfs3\_proc\_commit and nfs4\_proc\_commit

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *count(INT32),offset(INT64)*

- **addevent.nfs.proc.read\_setup.entry**

  The read\_setup function is used to setup a read rpc task,not do a real read operation

  the entry of  nfs\_proc\_read\_setup, nfs3\_proc\_read\_setup and nfs4\_proc\_read\_setup

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *count(INT32),offset(INT64)*

- **addevent.nfs.proc.write\_setup.entry**

  The write\_setup function is used to setup a write rpc task,not do a real write operation

  the entry of  nfs\_proc\_write\_setup, nfs3\_proc\_write\_setup and nfs4\_proc\_write\_setup

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *how(INT8), count(INT32),offset(INT64)*

- **addevent.nfs.proc.commit\_setup.entry**

  The commit\_setup function is used to setup a commit rpc task,not do a real commit operation.It is not exist in NFSV2

  the entry of nfs3\_proc\_commit\_setup and nfs4\_proc\_commit\_setup

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *how(INT8), count(INT32),offset(INT64)*

- **addevent.nfs.proc.read\_done.entry**

  Fires when a read reply is received or some read error occur (timeout or socket shutdown)

  the entry of nfs\_read\_done, nfs3\_read\_done and nfs4\_read\_done

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *status(INT32), count(INT32)*

- **addevent.nfs.proc.write\_done.entry**

  Fires when a write reply is received or some write error occur (timeout or socket shutdown)

  the entry of nfs\_write\_done, nfs3\_write\_done and nfs4\_write\_done

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *status(INT32), count(INT32)*

- **addevent.nfs.proc.commit\_done.entry**

  Fires when a commit reply is received or some commit operation error occur (timeout or socket shutdown)

  the entry of nfs\_commit\_done, nfs3\_commit\_done and nfs4\_commit\_done

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *status(INT32), count(INT32)*

- **addevent.nfs.proc.open.entry**

  the entry of nfs\_open

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *filename(STRING), flag(INT32), mode(INT32)*

- **addevent.nfs.proc.release.entry**

  the entry of nfs\_release

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *filename(STRING), flag(INT32), mode(INT32)*

- **addevent.nfs.proc.create.entry**

  the entry of nfs\_proc\_create, nfs3\_proc\_create, nfs4\_proc\_create

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *filename(STRING), mode(INT32)*

- **addevent.nfs.proc.rename.entry**

  the entry of nfs\_proc\_rename, nfs3\_proc\_rename, nfs4\_proc\_rename

  Data format is:

  *version(INT8), major\_old(INT8), minor\_old(INT8), old\_fileid(INT64), old\_name(STRING),* *major\_new(INT8), minor\_new(INT8), new\_fileid(INT64), new\_name(STRING)*

- **addevent.nfs.proc.remove.entry**

  the entry of nfs\_proc\_remove, nfs3\_proc\_remove, nfs4\_proc\_remove

  Data format is:

  *major\_dev(INT8), minor\_dev(INT8), fileid(INT64), version(INT8),* *filename(STRING)*

### NFSD

You could use *addevent.nfsd* to trace the detail activities of nfs on server side.  It divided into two groups: nfsd operation event hooks(addevent.nfsd.op), nfsd proc event hooks(addevent.nfsd.proc). It contains 19 entry hooks and 19 corresponding return hooks

All the return hooks will only log the common\_data and the return value. So in the following subsections, only the entry  hooks will be listed:

- **addevent.nfsd.dispatch.entry**

  Fires when server receives a NFS operation from client

  the entry of nfsd\_dispatch

  Data format is:

  *proto(INT8), version(INT8), xid(INT32), proc(INT32),client\_ip(INT32)*

- **addevent.nfsd.open.entry**

  the entry of nfsd\_open

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *type(INT32), access(INT32)*

- **addevent.nfsd.read.entry**

  the entry of nfsd\_read

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *count(INT64), offset(INT64), iov\_len(INT64), vlen(INT64)*

- **addevent.nfsd.write.entry**

  the entry of nfsd\_write

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *count(INT64), offset(INT64), iov\_len(INT64), vlen(INT64)*

- **addevent.nfsd.lookup.entry**

  the entry of nfsd\_lookup

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *filename(STRING)*

- **addevent.nfsd.commit.entry**

  the entry of nfsd\_commit

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *count(INT64), offset(INT64)*

- **addevent.nfsd.create.entry**

  Fires when client creates a file(regular,dir,device,fifo) on server side, sometimes nfsd will call nfsd\_create\_v3 instead of this function

  the entry of nfsd\_create

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *filename(STRING), type(INT32), iap\_valid(INT16), iap\_mode(INT32)*

- **addevent.nfsd.createv3.entry**

  Fires when client creates a regular file or set file attributes on server side, only called by nfsd3\_proc\_create and nfsd4\_open(op\_claim\_type is NFS4\_OPEN\_CLAIM\_NULL)

  the entry of nfsd\_create\_v3

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *filename(STRING), createmode(INT8), iap\_valid(INT16), iap\_mode(INT32)*

- **addevent.nfsd.unlink.entry**

  the entry of nfsd\_unlink

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64),* *filename(STRING), type(INT32)*

- **addevent.nfsd.rename.entry**

  the entry of nfsd\_rename

  Data format is:

  *old\_fhsize(INT8), old\_fh0(INT64), old\_fh1(INT64), old\_fh2(INT64), old\_name(STRING)* *new\_fhsize(INT8), new\_fh0(INT64), new\_fh1(INT64), new\_fh2(INT64), new\_name(STRING)*

- **addevent.nfsd.close.entry**

  the entry of nfsd\_close

  Data format is:

  *filename(STRING)*

- **addevent.nfsd.proc.lookup.entry**

  the entry of nfsd\_proc\_lookup, nfsd3\_proc\_lookup

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *filename(STRING)*

- **addevent.nfsd.proc.read.entry**

  the entry of nfsd\_proc\_read, nfsd3\_proc\_read

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *count(INT64), offset(INT64), iov\_len(INT64), vlen(INT64)*

- **addevent.nfsd.proc.write.entry**

  the entry of nfsd\_proc\_write, nfsd3\_proc\_write

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *count(INT64), offset(INT64), iov\_len(INT64), vlen(INT64)*

- **addevent.nfsd.proc.commit.entry**

  the entry of nfsd\_proc\_commit, nfsd3\_proc\_commit

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *count(INT64), offset(INT64)*

- **addevent.nfsd.proc.commit.entry**

  the entry of nfsd4\_proc\_compound

  Data format is:

  *number(INT32)*

- **addevent.nfsd.proc.remove.entry**

  the entry of nfsd4\_proc\_compound

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *filename(STRING)*

- **addevent.nfsd.proc.rename.entry**

  the entry of nfsd\_proc\_rename, nfsd3\_proc\_rename

  Data format is:

  *old\_fhsize(INT8), old\_fh0(INT64), old\_fh1(INT64), old\_fh2(INT64), old\_name(STRING)* *new\_fhsize(INT8), new\_fh0(INT64), new\_fh1(INT64), new\_fh2(INT64), new\_name(STRING)*

- **addevent.nfsd.proc.create.entry**

  the entry of nfsd\_proc\_create, nfsd3\_proc\_create

  Data format is:

  *fh\_size(INT8), fhandle0(INT64), fhandle1(INT64), fhandle2(INT64), version(INT8)* *filename(STRING)*

## TRACE DATA FORMAT

By default, LKET will log the trace data in binary format.

To get a better performance for binary tracing, the "-b" option should be turned on for stap and thus -M option has to be added to stop staprun merging per-cpu files.

You could use the command  *lket-b2a* to convert the binary trace data  generated by LKET into readable data in ascii format.

*lket-b2a* uses the pre-cpu binary trace data files as inputs, and generates an output file named  *lket.out* You should use "stap -b -M" with LKET to get those pre-cpu files (stpd\_cpu\*) before using it.

If you want LKET to log trace data in ASCII format directly, you should:\

```
stap -D ASCII_TRACE ...
```

## EVENT REGISTER

LKET provides a way to log the metadata of the trace data by events registering.

Two function is provided:

-

  **register\_sys\_event(grpid:long, hookid:long, fmt:string, names:string)**

-

  **register\_user\_event(grpid:long, hookid:long, fmt:string, names:string)**

The  *grpid* and  *hookid* is the groupid and hookid of the event you want to register.

*fmt* contains a set of fomat tokens seperated by ":". The valid format tokens are: **UINT8,** **UINT16,** **UINT32,** **UINT64** and **STRING** which represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated respectively.

*names* contains a set of names seperated by ":". The names contains in *names* should match the format tokens contains in *fmt*

**register\_sys\_event** is used to register the newly added event hooks. For example, supposing you want to add a new event hook to trace the entry of sys\_open, and you want this event hook to log the fd, flag and mode paremeters for you. You should add:

```
register_sys_event(GROUP_IOSYSCALL, HOOKID_IOSYSCALL_OPEN_ENTRY,
	"STRING:INT32:INT32", "filename:flags:mode")
```

into the function **register\_sys\_events** in LKET/register\_event.stp

**register\_user\_event** is used for user to add extra trace data for a event hook. See the section **CUSTOMIZED TRACE DATA** for more detail

## CUSTOMIZED TRACE DATA

LKET defines a set of event hooks and will log the predefined trace data for you, but what if you want to trace extra  data for that event?

LKET provides a way to do this without modifying the codes in the tapset of that event hook. You can simply use printf to trace extra data. For example, supposing you want to trace sk\_buff-\>mac\_len and sk\_buff-\>priority besides the sk\_buff-\>len, sk\_buff-\>protocol and sk\_buff-\>truesize for the  **netdev** event hooks:

```
probe register_event
{
    register_user_event(GROUP_NETDEV, HOOKID_NETDEV_TRANSMIT,
        "INT32:INT32", "mac_len:priority")
}
probe addevent.netdev.transmit
{
    printf("%4b%4b", $skb->mac_len, $skb->priority)
}
```

## EXAMPLES

Here are some examples of using LKET:

- To turn on all event hooks:

  stap -e "probe addevent.\* {}" -bM

- To probe syscall:

  stap -e "probe addevent.syscall {}" -bM

- To only probe syscall.entry:

  stap -e "probe addevent.syscall.entry {}" -bM

- To probe netdev transmition and log extra data of mac\_len and priority:

  stap -e "probe addevent.netdev.transmit { printf(\\"%4b%4b\\", $skb-\>mac\_len, $skb-\>priority) }" -bM

## SEE ALSO

[*stap*(1)](/stap/1.md) *lket-b2a*(5)