My favorites ▼ | Sign in
Project Home Downloads Wiki Issues Source
Search
for
HOWTO  

Phase-QA, Phase-Deploy
Updated May 19, 2012 by teawa...@gmail.com

Update in 2012-05-20

What is KGTP

KGTP is a realtime and lightweight Linux debugger and tracer.
To use it, you don't need patch or rebuild the Linux kernel. Just build KGTP module and insmod it is OK.

It makes Linux Kernel supply a GDB remote debug interface. Then GDB in current machine or remote machine can debug and trace Linux through GDB tracepoint and some other functions without stopping the Linux Kernel.
And even if the board doesn't have GDB on it and doesn't have interface for remote debug. It can debug the Linux Kernel using offline debug (See HOWTO#/sys/kernel/debug/gtpframe_and_offline_debug).

KGTP supports X86-32, X86-64, MIPS and ARM.
KGTP is tested on Linux kernel 2.6.18 to upstream.
And it can work with Android (See HowToUseKGTPinAndroid).

For new user of KGTP, please go to see Quickstart.

Please go to UPDATE to get more info about KGTP update.

Get help or report issues about KGTP

You can post it to http://code.google.com/p/kgtp/issues/list, write Email to kgtp@freelists.org or write Email to teawater@gmail.com .
The KGTP team will try our best to help you.

Preparatory work before use KGTP

Linux kernel

If your system use the Linux kernel that is built by yourself

To use KGTP, your Linux kernel need open following options: 

General setup  --->
	[*] Kprobes

[*] Enable loadable module support  --->

Kernel hacking  --->
	[*] Debug Filesystem
	[*] Compile the kernel with debug info

If your system use the Linux kernel from distribution

You need install some Linux kernel package.

Ubuntu

Install the Linux kernel debug image

1) Add debug source to the sources list of Ubuntu.

  • Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal: 

    • echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
  • Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command: 
    • echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
  • Import the debug symbol archive signing key: 
    • sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
  • Then run: 
    • sudo apt-get update


2) Get Linux kernel debug image. 

sudo apt-get install linux-image-$(uname -r)-dbgsym

Then you can find Linux kernel debug image in "/usr/lib/debug/boot/vmlinux-$(uname -r)".

Install the Linux kernel headers
sudo apt-get install linux-headers-generic
Install the Linux kernel source
  • Install the source package: 
    • sudo apt-get install linux-source
  • Uncompress the source package: 
    • sudo mkdir -p /build/buildd/
sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2 -C /build/buildd/
sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//') /build/buildd/linux-$(uname -r | sed 's/-.*//')

Fedora

Install the Linux kernel debug image
sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo

Then you can find Linux kernel debug image in "/usr/lib/debug/lib/modules/$(uname -r)/vmlinux".

Install the Linux kernel devel package
sudo yum install kernel-devel-$(uname -r)

Others

You need install the Linux kernel debug image package and the Linux Kernel source.

Make sure current Linux kernel debug image is right

GDB open the right Linux kernel debug image is an very important because GDB will get the debug info and address info from it. So before you use KGTP, please do the check to make sure about it.
There are 2 ways to do the check, what I suggest is do both of them to make sure Linux kernel debug image is right.

Please note that if you determine you use the right Linux kernel debug image, but cannot pass these ways. Please see HOWTO#Handle_the_issue_that_Linux_kernel_debug_image's_address_in.

Use /proc/kallsyms

In the system that its Linux kernel is what you want to trace, use following command to get the address of sys_read and sys_write: 

sudo cat /proc/kallsyms | grep sys_read
ffffffff8117a520 T sys_read
sudo cat /proc/kallsyms | grep sys_write
ffffffff8117a5b0 T sys_write

Then we can get that the address of sys_read is 0xffffffff8117a520 and the address of sys_write is 0xffffffff8117a5b0.
After that use GDB get address of sys_read and sys_write from Linux kernel debug image: 

gdb ./vmlinux
(gdb) p sys_read
$1 = {long int (unsigned int, char *, size_t)} 0xffffffff8117a520 <sys_read>
(gdb) p sys_write
$2 = {long int (unsigned int, const char *, size_t)} 0xffffffff8117a5b0 <sys_write>

The address of sys_read and sys_write is same, so the Linux kernel debug image is right.

Use linux_banner

sudo gdb ./vmlinux
(gdb) p linux_banner
$1 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"

This linux_banner is the kernel info inside the Linux kernel debug image.
After that, connect to KGTP following the way in HOWTO#Make_GDB_connect_to_gtp connect to KGTP and print linux_banner again. 

(gdb) target remote /sys/kernel/debug/gtp
Remote debugging using /sys/kernel/debug/gtp
0x0000000000000000 in irq_stack_union ()
(gdb) p linux_banner
$2 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"

This linux_banner is the kernel info that Linux kernel that KGTP is tracing. If it is same with the prev kernel info, the Linux kernel debug image is right.

Handle the issue that Linux kernel debug image's address info is not same with Linux kernel when it running

In X86_32, you will found that the Linux kernel debug image's address info is not same with Linux kernel when it running through the ways in HOWTO#Make_sure_current_Linux_kernel_debug_image_is_right. And you determine the Linux kernel debug image is right.
This issue is because: 

Processor type and features  --->
	(0x1000000) Physical address where the kernel is loaded
	(0x100000) Alignment value to which kernel should be aligned

The values of these two options are different. Please note that the "Physical address where the kernel is loaded" is not showed in config sometimes. You can get its value through search "PHYSICAL_START".

You can handle this issue through change "Alignment value to which kernel should be aligned" same with "Physical address where the kernel is loaded".
This issue doesn't affect X86_64.

Get KGTP

Get KGTP through http

Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to download the package.

Get KGTP through svn

Some people have trouble with access to KGTP website. You can access kgtp through svn: 

svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only

kgtp-read-only/tags/ Present for each release of KGTP.
kgtp-read-only/trunk/ Present for the main trunk of KGTP.

Config KGTP

Following part is the default config of KGTP inside the Makefile. With this config, KGTP will build together with current kernel that running on this machine. 

KERNELDIR := /lib/modules/`uname -r`/build
CROSS_COMPILE :=

KERELDIR is set to the directory which holds the kernel you want to build for. By default, it is set to the kernel that you are running.
Please note that this directory should be Linux kernel build directory or linux-headers directory but not the source directory but not the Linux kernel source directory. And the Linux kernel build directory should be used after build successful.
CROSS_COMPILE is set to the prefix name of compiler that you want to build KGTP. Empty to compile with your default compiler.
ARCH is the architecture.

Or you can choose which kernel you want build with and which compiler you want use by change Makefile.
For example: 

KERNELDIR := /home/teawater/kernel/bamd64
CROSS_COMPILE :=x86_64-glibc_std-
ARCH := x86_64

KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will use x86_64-glibc_std-gcc.

Compile KGTP

Normal compile

cd kgtp/
make

Compile KGTP with some special config

Most of time, KGTP can auto select right options to build with Various versions of Linux kernel.
But if you want config special options with yourself, you can read following part:

With this option, KGTP will not auto select any build options. 

make AUTO=0

With this option, KGTP will use simple frame instead of KGTP ring buffer.
The simple frame doesn't support gtpframe_pipe. It just for debug KGTP. 

make AUTO=0 FRAME_SIMPLE=1

With this option, $clock will return rdtsc value instead of local_clock. 

make AUTO=0 CLOCK_CYCLE=1

With this option, KGTP will use procfs instead of debugfs. 

make AUTO=0 USE_PROC=1

The options can use together, for example: 

make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1

Install and uninstall KGTP

KGTP don't need to be install because it can insmod directly inside its directory (See HOWTO#Exec_it). But if you need, you can install it to your system.
Install: 

cd kgtp/
sudo make install

Uninstall: 

cd kgtp/
sudo make uninstall

Use KGTP with DKMS

You can use KGTP with DKMS if you want it.
Following commands will copy the files of KGTP to the directory that DKMS need. 

cd kgtp/
sudo make dkms

Then you can use DKMS commands to control KGTP. Please goto http://linux.dell.com/dkms/manpage.html to see how to use DKMS.

Use KGTP patch for Linux kernel

Most of time, you don't need KGTP patch because KGTP can build as a LKM and very easy to use. But to help some people include KGTP to them special Linux Kernel tree, KGTP supply patches for Linux kernel.
In the KGTP directory:

  • gtp_for_review.patch is the patch for Linux kernel upstream.
  • gtp_3.0_to_upstream.patch is the patch for Linux kernel from 3.0 to upstream.
  • gtp_2.6.39.patch is the patch for Linux kernel 2.6.39.
  • gtp_2.6.33_to_2.6.38.patch is the patch for Linux kernel from 2.6.33 to 2.6.38.
  • gtp_older_to_2.6.32.patch is the patch for Linux kernel 2.6.32 and older version.

Install GDB for KGTP

The GDB that older than 7.3 have some bugs of tracepoint. And some functions of GDB are not very well.
So if your GDB is older than 7.3 please go to https://code.google.com/p/gdbt/ to get howto install GDB for KGTP.
If you have issue about GDB please get help according to HOWTO#Report_issues_about_KGTP.

Howto use

Exec it

If you have installed KGTP in your system, you can: 

sudo modprobe gtp

Or you can use the kgtp module in the directory. 

cd kgtp/
sudo insmod gtp.ko

Make GDB connect to gtp

GDB on the current machine

sudo gdb ./vmlinux
(gdb) target remote /sys/kernel/debug/gtp
Remote debugging using /sys/kernel/debug/gtp
0x0000000000000000 in ?? ()

After that, you can begin to use GDB command trace and debug the Linux Kernel.

If GDB on remote machine

#Open the KGTP interface in current machine.
sudo su
nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp
(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old version netcat.)
#Let gdb connect to the port 1234
gdb ./vmlinux
(gdb) target remote xxx.xxx.xxx.xxx:1234

After that, you can begin to use GDB command trace and debug the Linux Kernel.

Add module symbols to GDB

Sometimes you need to add a Linux kernel module's symbols to GDB to debug it.
Add symbols with hand is not very easy, so KGTP package include an GDB python script "getmod.py" and a program "getmod" can help you.

How to use getmod.py

Connect to KGTP before use the getmod.py. 

(gdb) source ~/kgtp/getmod.py

Then this script will auto load the Linux kernel module's symbols to GDB.

How to use getmod

"getmod" is written by C so you can use it anywhere even if in an embedded environment.
For example: 

#Following command save Linux Kernel module info to the file ~/tmp/mi in GDB
#command format.
sudo getmod >~/tmp/mi
#in gdb part:
(gdb) source ~/tmp/mi
add symbol table from file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
	.text_addr = 0xf80de000
	.note.gnu.build-id_addr = 0xf80de088
	.exit.text_addr = 0xf80de074
	.init.text_addr = 0xf8118000
	.rodata.str1.1_addr = 0xf80de0ac
	.rodata_addr = 0xf80de0c0
	__mcount_loc_addr = 0xf80de9c0
	.data_addr = 0xf80de9e0
	.gnu.linkonce.this_module_addr = 0xf80dea00
#After this GDB command, all the Linux Kernel module info is loaded into GDB.

If you use remote debug or offline debug, maybe you need change the base directory. Following example is for it. 

#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r /home/teawater/kernel/b26
sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi

Access memory directly

After connect the KGTP, you can access most of memory directly.
For example, you can access to "jiffies_64" with following command: 

(gdb) p jiffies_64

Or you can access to the first entry of "static LIST_HEAD(modules)" with following command: 

(gdb) p *((struct module *)((char *)modules->next - ((size_t) &(((struct module *)0)->list))))

Or you can access to the CPU0 memory info of "DEFINE_PER_CPU(struct device , mce_device);": 

p *(struct device *)(__per_cpu_offset[0]+(uint64_t)(&mce_device))

GDB tracepoint

Tracepoint is that GDB define some addresses and some actions and put them to the target (KGTP). After tracepoint start, , KGTP will do these actions (Some of them will collect data and save them to tracepoint frame buffer) when Linux kernel execution to there addresses. After that, Linux kernel will keep execution.
KGTP supply some interfaces that GDB or other programe can take the data of tracepoint frame buffer out to parse.
About these interfaces, this doc have introduced "/sys/kernel/debug/gtp". And will introduce "/sys/kernel/debug/gtpframe" and "/sys/kernel/debug/gtpframe_pipe" later.

Doc of GDB tracepoint in http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html.

set tracepoint

The trace command is very similar to the break command. Its argument location can be a source line, a function name, or an address in the target program. The trace command defines a tracepoint, which is a address or some addresses that KGTP do some actions in it.

Here are some examples of using the trace command: 

(gdb) trace foo.c:121    // a source file and line number

(gdb) trace +2           // 2 lines forward

(gdb) trace my_function  // first source line of function

(gdb) trace *my_function // EXACT start address of function

(gdb) trace *0x2117c4    // an address

Howto handle the function is there but set tracepoint on it got fail

GCC will inline some static function to increase the performance. You cannot set tracepoint on the function name because object file doesn't have symbol of inline function.
You can use "trace filename:line" to set tracepoint on it.

actions [num]

This command will prompt for a list of actions to be taken when the tracepoint is hit. If the tracepoint number num is not specified, this command sets the actions for the one that was most recently defined (so that you can define a tracepoint and then say actions without bothering about its number). You specify the actions themselves on the following lines, one action at a time, and terminate the actions list with a line containing just end. So far, the only defined actions are collect, teval, and while-stepping.

collect expr1, expr2, ...

Collect values of the given expressions when the tracepoint is hit. This command accepts a comma-separated list of any valid expressions. In addition to global, static, or local variables, the following special arguments are supported: 

$regs	Collect all registers. 
$args	Collect all function arguments. 
$locals	Collect all local variables.

teval expr1, expr2, ...

Evaluate the given expressions when the tracepoint is hit. This command accepts a comma-separated list of expressions. The results are discarded, so this is mainly useful for assigning values to trace state variables (see HOWTO#Simple_trace_state_variables) without adding those values to the trace buffer, as would be the case if the collect action were used.

while-stepping n

Please goto HOWTO#If_the_debug_info_of_the_function_pointer_is_optimized_out see howto use it.

Start and stop the tracepoint

Tracepoint will exec actions only when it is starting use this GDB command: 

(gdb) tstart

It will stop by this GDB command: 

(gdb) tstop

Enable and disable the tracepoint

Like breakpoint, tracepoint can be control by GDB commands "enable" and "disable". But please note that it only useful when tracepoint stop.

Use "tfind" select the entry inside the trace frame info

GDB command "tfind" is used to select a entry of trace frame bufffer when tracepoint stop.
When GDB inside "tfind" mode, it will just show the values of this entry that the tracepoint action collect. So it will output some error when print some values that action doesn't collect for example the argument of function. That is not a bug, please don't worry about it.
Use "tfind" again will select next entry. "tfind id" will select entry id.
To go back to normal mode, please use GDB command "tfind -1". Please goto http://sourceware.org/gdb/current/onlinedocs/gdb/tfind.html get more info about it.

Show and save the tracepoint

You can use GDB command "info tracepoints" to show all the tracepoints.
You can use GDB command "save tracepoints filename" to save the commands that setup the tracepoints and actions into file filename. Then you use use GDB commands "source filename" to setup this tracepints again.

Delete tracepoint

GDB command "delete id" will delete tracepoint id. If "delete" without argument, it will delete all the tracepoint.

Use tracepoint get register info from a point of kernel

The following is an example that records the value of all registers when "vfs_readdir" is called. 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) trace vfs_readdir
Tracepoint 1 at 0xc01a1ac0: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
(gdb) actions
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $reg
>end
(gdb) tstart
(gdb) shell ls
(gdb) tstop
(gdb) tfind
Found trace frame 0, tracepoint 1
#0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
   buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
23      /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or directory.
       in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb) info reg
eax            0xc5528d00       -984445696
ecx            0xc0d09f90       -1060069488
edx            0xc01a1900       -1072031488
ebx            0xfffffff7       -9
esp            0xc0d09f8c       0xc0d09f8c
ebp            0x0      0x0
esi            0x8061480        134616192
edi            0xc5528d00       -984445696
eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
eflags         0x286    [ PF SF IF ]
cs             0x60     96
ss             0x8061480        134616192
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x0      0
(gdb) tfind
Found trace frame 1, tracepoint 1
0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb) info reg
eax            0xc5528d00       -984445696
ecx            0xc0d09f90       -1060069488
edx            0xc01a1900       -1072031488
ebx            0xfffffff7       -9
esp            0xc0d09f8c       0xc0d09f8c
ebp            0x0      0x0
esi            0x8061480        134616192
edi            0xc5528d00       -984445696
eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
eflags         0x286    [ PF SF IF ]
cs             0x60     96
ss             0x8061480        134616192
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x0      0

Use tracepoint get the value of variable from a point of kernel

The following is an example that records the value of "jiffies_64" when the function "vfs_readdir" is called: 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) trace vfs_readdir 
Tracepoint 1 at 0xc01ed740: file /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect jiffies_64
>collect file->f_path.dentry->d_iname
>end
(gdb) tstart 
(gdb) shell ls
arch	drivers   include  kernel    mm		      Module.symvers  security	System.map  virt
block	firmware  init	   lib	     modules.builtin  net	      sound	t	    vmlinux
crypto	fs	  ipc	   Makefile  modules.order    scripts	      source	usr	    vmlinux.o
(gdb) tstop 
(gdb) tfind 
Found trace frame 0, tracepoint 1
#0  0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580 <filldir64>, buf=0xd6dfdf90)
    at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
24	{
(gdb) p jiffies_64
$1 = 4297248706
(gdb) p file->f_path.dentry->d_iname
$1 = "b26", '\000' <repeats 28 times>

How to use use tracepoint condition

http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html
Like breakpoints, we can set conditions on tracepoints. The speed of tracepoints is faster than breakpoints because KGTP can do all the condition checks.
For example: 

(gdb) trace handle_irq
(gdb) condition 1 (irq == 47)

This action of tracepoint 1 will work only when irq number is 47.

How to handle "Unsupported operator (null) (52) in expression."

If you use condition about string, you will got this error when you call "tstart".
To handle it, you can convent the char to int to handle this issue, for example: 

(gdb) condition 1 (buf[0] == (int)'1')

Show all the traced data of current frame

After use "tfind" select an entry, you can use "tdump" to do it. 

(gdb) tdump 
Data collected at tracepoint 1, trace frame 0:
$cr = void
file->f_path.dentry->d_iname = "gtp\000.google.chrome.g05ZYO\000\235\337\000\000\000\000\200\067k\364\200\067", <incomplete sequence \364>
jiffies_64 = 4319751455

Get status of tracepoint

Please use GDB command "tstatus".

Set the trace buffer into a circular buffer

http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
The frame buffer is not a circular buffer by default. When the buffer is full, the tracepoint will stop.
Following command will set frame buffer to a circular buffer. When the buffer is full, it will auto discard traceframes (oldest first) and keep trace. 

(gdb) set circular-trace-buffer on

Do not stop tracepoint when the GDB disconnects

http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
KGTP will stop tracepoint and delete the trace frame when GDB disconnects with it by default.
Following command will open the KGTP disconnect-trace. After that, when GDB disconnects with KGTP, KGTP will not stop tracepoint. And after GDB reconnects to KGTP, it can keep control of KGTP like nothing happened. 

(gdb) set disconnected-tracing on

kprobes-optimization and the execution speed of tracepoint

The tracepoint is execution together with Linux kernel. So it speed will affect the speed the system.
The KGTP tracepoint base on Linux kernel kprobe. Because the normal kprobe base on breakpoint instruction, so it is not very fast.

But if arch of kernel is X86_64 or X86_32 and kernel config didn't open "Preemptible Kernel" (PREEMPT), the kprobe is speed up by kprobes-optimization (CONFIG_OPTPROBES) that make kprobe very fast.
To make sure about that, you can use following command in terminal: 

sysctl -A | grep kprobe
debug.kprobes-optimization = 1

That means that your kernel support kprobes-optimization.
Please note that some KGTP functions will make this tracepoint use simple kprobe even if this Kernel support kprobes-optimization. This doc will add note when introduce these functions. Please avoid using them when you really care about the tracepoint speed.

How to use trace state variables

http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html
Tracepoints have special variables. The variables can be traced directly, or used in tracepoint conditions.
Please note that just GDB 7.2.1 and later versions support use trace state variables directly, the old version of GDB can show the value of trace state variables through command "info tvariables".

Simple trace state variables

Define a trace state variable $c. 

(gdb) tvariable $c

Trace state variable $c is created with initial value 0. The following action uses $c to count how many irqs happened in the kernel. 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) trace handle_irq
(gdb) actions 
Enter actions for tracepoint 3, one per line.
End with a line saying just "end".
>collect $c     #Save current value of $c to the trace frame buffer.
>teval $c=$c+1  #Increase the $c.
>end

Also, you can set a value of variable to trace state variable, but don't forget covert variable to "uint64_t". 

>teval $c=(uint64_t)a

You can get the current value of $c while the trace is running or stopped. 

(gdb) tstart
(gdb) info tvariables
$c              0           31554
(gdb) p $c
$5 = 33652
(gdb) tstop
(gdb) p $c
$9 = 105559

When using tfind, you can parse the trace frame buffer. If the value of a trace state variable is collected, you can parse it out. 

(gdb) tstop 
(gdb) tfind 
(gdb) info tvariables
$c              0           0
(gdb) p $c
$6 = 0
(gdb) tfind 100
(gdb) p $c
$7 = 100

If need, the tracepoint action that access the simple trace state variables will auto lock a spin lock for trace state variables. So it can handle race condition issue about trace state variables.
The following example is OK even if it running a machine that have more than one CPU. 

>teval $c=$c+1

Per_cpu trace state variables

Per_cpu trace state variables are special simple trace state variables.
When tracepoint action access to it, it will access to this CPU special trace state variables.

It have 2 advantages:
1. The tracepoint actions that access to per_cpu trace state variables don't have the race conditon issue. So it don't need lock the spin lock for trace state variables. It is faster than simple trace state variables on multi-core machine. 2. Write the action that count some CPU special thing with it is easier than simple trace state variables.

To define per_cpu trace state variables, you need named it in format: 

"per_cpu_"+string+CPU_id

or 

"pc_"+string+CPU_id

Following example will define a series of per_cpu trace state variables in a 4 COREs CPU machine with string "count": 

(gdb) tvariable $pc_count0
(gdb) tvariable $pc_count1
(gdb) tvariable $pc_count2
(gdb) tvariable $pc_count3

You can use compatibility better way to do it: 

(gdb) set $tmp=0
(gdb) while $tmp<$cpu_number
 >eval "tvariable $pc_count%d",$tmp
 >set $tmp=$tmp+1
 >end

Tracepoint action can access anyone of a series of per_cpu trace state variables. KGTP will auto access the one of CPU that it running on.
For example: 

(gdb) trace vfs_read
(gdb) actions
>teval $pc_count0=$pc_count0+1
>end

These GDB commands define a tracepoint that count the times that call vfs_read of each CPU.

Special trace state variables $current_task, $current_task_pid, $current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format, $printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count

KGTP special trace state variables $current_task, $current_thread_info, $cpu_id and $clock can very easy to access to some special value. You can see them when GDB connects to the KGTP. You can use them in tracepoint conditions or actions.
Access $current_task in tracepoint condition and action will get that returns of get_current().
Access $current_task_pid in tracepoint condition and action will get that returns of get_current()->pid.
Access $current_thread_info in tracepoint condition and action will get that returns of current_thread_info().
Access $cpu_id in tracepoint condition and action will get that returns of smp_processor_id().
Access $clock in tracepoint condition and action will get that returns of local_clock() that return the timestamp in nanoseconds.
$rdtsc is only available on X86 and X86_64 architecture. Access it in anytime will get current value of TSC with instruction RDTSC.
Access $hardirq_count in tracepoint condition and action will get that returns of hardirq_count().
Access $softirq_count in tracepoint condition and action will get that returns of softirq_count().
Access $irq_count in tracepoint condition and action will get that returns of irq_count().

And KGTP has other special trace state variables $dump_stack, $printk_level, $printk_format and $printk_tmp. All of them output their values directly, as can be seen in HOWTO#Howto_let_tracepoint_output_value_directly.

The following example counts in $c how many vfs_read calls that process 16663 does and collects the struct thread_info of current task: 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid == 16663)
(gdb) tvariable $c
(gdb) actions
Enter actions for tracepoint 4, one per line.
End with a line saying just "end".
>teval $c=$c+1
>collect (*(struct thread_info *)$current_thread_info)
>end
(gdb) tstart 
(gdb) info tvariables 
Name            Initial     Current     
$c              0           184         
$current_task   0           <unknown>   
$current_thread_info 0           <unknown>   
$cpu_id         0           <unknown>   
(gdb) tstop
(gdb) tfind
(gdb) p *(struct thread_info *)$current_thread_info
$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status = 0, cpu = 1, preempt_count = 2, addr_limit = {
    seg = 4294967295}, restart_block = {fn = 0xc0159fb0 <do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11, 
        arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags = 1, bitset = 78, time = 977063750, 
        uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb, expires = 335007449089}, poll = {
        ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78, tv_nsec = 977063750}}}, 
  sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack = 0xef340044 "", uaccess_err = 0}

Another example shows how much sys_read() executes in each CPU. 

(gdb) tvariable $c0
(gdb) tvariable $c1
(gdb) trace sys_read 
(gdb) condition $bpnum ($cpu_id == 0)
(gdb) actions
>teval $c0=$c0+1
>end
(gdb) trace sys_read 
(gdb) condition $bpnum ($cpu_id == 1)
(gdb) actions
>teval $c1=$c1+1
>end
(gdb) info tvariables 
Name            Initial     Current     
$current_task   0           <unknown>   
$cpu_id         0           <unknown>   
$c0             0           3255        
$c1             0           1904

sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. Please note that this example just to howto use $cpu_id. Actially, this example use per_cpu trace state variables is better.

Special trace state variable $no_self_trace

$no_self_trace is different with the special trace state variables in the previous section. It is used to control the behavior of tracepoint.
If the action of a tracepoint include a command access to the $no_self_trace. The tracepoint will not trace anything if the current_task is the a KGTP self process (GDB, netcat, getframe or some others process that access to the interface of KGTP).
For example, if we want trace vfs_read or something that have process context, and we don't want trace the operation of KGTP self process. Add following command to the action: 

>collect $no_self_trace

Please note that the code that doesn't about process context (Irq handler, softirq) doesn't need set this variable.

Trace the function return with $kret

Sometime, set the tracepoint to the end of function is hard because the Kernel is compiled with optimization. At this time, you can get help from $kret.

$kret is a special trace state variable like $no_self_trace. When you set value of it inside the action of tracepoint, this tracepoint be set with kretprobe instead of kprobe. Then it can trace the end of this function.

Please note that this tracepoint must set in the first address of the function in format "function_name".

Following part is an example: 

#"*(function_name)" format can make certain that GDB send the first address of function to KGTP.
(gdb) trace *vfs_read
(gdb) actions
>teval $kret=0
#Following part you can set commands that you want.

Use $ignore_error and $last_errno to ignore the error of tstart

If KGTP got any error of tstart, this command will get fail.
But sometime we need ignore this error and let KGTP keep work. For example: If you set tracepoint on the inline function spin_lock. This tracepoint will be set to a lot of addresses that some of them cannot be set kprobe. It will make tstart get fail. You can use "$ignore_error" ignore this error.
And the last error number will available in "$last_errno". 

(gdb) tvariable $ignore_error=1

This command will open ignore. 

(gdb) tvariable $ignore_error=0

This command will close ignore.

Use $cooked_clock and $cooked_rdtsc the time without KGTP used

Access these two trace state variables can get the time without KGTP used. Then we can get more close to really time that a part of code used even if the actions of tracepoint is very complex. They will be introduce in Cookbook (coming soon).

Use $xtime_sec and $xtime_nsec get the timespec

Access these two trace state variables will return the time of day in a timespec that use getnstimeofday.
$xtime_sec will access to the second part of a timespec.
$xtime_nsec will access to the nanosecond part of a timespec.

Howto backtrace (stack dump)

Each time your program performs a function call, information about the call is generated. That information includes the location of the call in your program, the arguments of the call, and the local variables of the function being called. The information is saved in a block of data called a stack frame. The stack frames are allocated in a region of memory called the call stack.

Collect stack with $bt and use GDB command "backtrace"

Because this way is faster (just collect the stack when trace) and parse out most of info inside the call stack (it can show all the stack info that I introduce). So I suggest you use this way to do the stack dump.
First we need add the collect the stack command to the tracepoint action.
The general collect the stack command in GDB tracepoint is: In x86_32, following command will collect 512 bytes of stack.
>collect *(unsigned char *)$esp@512
In x86_64, following command will collect 512 bytes of stack.
>collect *(unsigned char *)$rsp@512
In MIPS or ARM, following command will collect 512 bytes of stack.
>collect *(unsigned char *)$sp@512
These commands is so hard to remember, and the different arch need different command.
 KGTP have an special tracepoint trace state variable $bt. If tracepoint action access it, KGTP will auto collect the $bt size (default value is 512) stack. For example, this command will collect 512 bytes stack memory: 

>collect $bt

If you want to change size of $bt, you can use following GDB command before "tstart": 

(gdb) tvariable $bt=1024

Following part is an example about howto collect stack and howto use GDB parse it: 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) trace vfs_readdir
Tracepoint 1 at 0xffffffff8118c300: file /home/teawater/kernel2/linux/fs/readdir.c, line 24.
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $bt
>end
(gdb) tstart 
(gdb) shell ls
1      crypto	 fs		include  kernel    mm		    Module.symvers  security  System.map  vmlinux
arch   drivers	 hotcode.html	init	 lib	   modules.builtin  net		    sound     usr	  vmlinux.o
block  firmware  hotcode.html~	ipc	 Makefile  modules.order    scripts	    source    virt
(gdb) tstop 
(gdb) tfind 
Found trace frame 0, tracepoint 1
#0  vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0 <filldir>, buf=0xffff880108709f40)
    at /home/teawater/kernel2/linux/fs/readdir.c:24
24	{
(gdb) bt
#0  vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0 <filldir>, buf=0xffff880108709f40)
    at /home/teawater/kernel2/linux/fs/readdir.c:24
#1  0xffffffff8118c689 in sys_getdents (fd=<optimized out>, dirent=0x1398c58, count=32768) at /home/teawater/kernel2/linux/fs/readdir.c:214
#2  <signal handler called>
#3  0x00007f00253848a5 in ?? ()
#4  0x00003efd32cddfc9 in ?? ()
#5  0x00002c15b7d04101 in ?? ()
#6  0x000019c0c5704bf1 in ?? ()
#7  0x0000000900000000 in ?? ()
#8  0x000009988cc8d269 in ?? ()
#9  0x000009988cc9b8d1 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb) up
#1  0xffffffff8118c689 in sys_getdents (fd=<optimized out>, dirent=0x1398c58, count=32768) at /home/teawater/kernel2/linux/fs/readdir.c:214
214		error = vfs_readdir(file, filldir, &buf);
(gdb) p buf
$1 = {current_dir = 0x1398c58, previous = 0x0, count = 32768, error = 0}
(gdb) p error
$3 = -9
(gdb) frame 0
#0  vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0 <filldir>, buf=0xffff880108709f40)
    at /home/teawater/kernel2/linux/fs/readdir.c:24
24	{

From this example, we can see some GDB commands that parse the the call stack:

  • bt is the alias of GDB commands backtrace that print a backtrace of the entire stack: one line per frame for all frames in the stack.
  • up n is move n frames up the stack. For positive numbers n, this advances toward the outermost frame, to higher frame numbers, to frames that have existed longer. n defaults to one.
  • down n is move n frames down the stack. For positive numbers n, this advances toward the innermost frame, to lower frame numbers, to frames that were created more recently. n defaults to one. You may abbreviate down as do.
  • frame n is select frame number n. Recall that frame zero is the innermost (currently executing) frame, frame one is the frame that called the innermost one, and so on. The highest-numbered frame is the one for main.
You can see that when you use up, down or frame to the different calll stack frame, you can output the value of the arguments and local variables of different call stack frame.
To get the more info about howto use GDB parse the call stack, please see http://sourceware.org/gdb/current/onlinedocs/gdb/Stack.html

Collect stack of current function's caller with $_ret

If you just want to collect stack of current function's caller, please use $_ret.
Please note that set the tracepoint that collect $_ret cannot in the first address of function.
For example: 

(gdb) list vfs_read
360	}
361	
362	EXPORT_SYMBOL(do_sync_read);
363	
364	ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)
365	{
366		ssize_t ret;
367	
368		if (!(file->f_mode & FMODE_READ))
369			return -EBADF;
(gdb) trace 368
Tracepoint 2 at 0xffffffff8117a244: file /home/teawater/kernel2/linux/fs/read_write.c, line 368.
(gdb) actions 
Enter actions for tracepoint 2, one per line.
End with a line saying just "end".
>collect $_ret
>end
(gdb) tstart 
(gdb) tstop 
(gdb) tfind 
Found trace frame 0, tracepoint 2
#0  vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0 out of bounds>, count=8192, pos=0xffff88012fa49f48)
    at /home/teawater/kernel2/linux/fs/read_write.c:368
368		if (!(file->f_mode & FMODE_READ))
(gdb) bt
#0  vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0 out of bounds>, count=8192, pos=0xffff88012fa49f48)
    at /home/teawater/kernel2/linux/fs/read_write.c:368
#1  0xffffffff8117a3ea in sys_read (fd=<optimized out>, buf=<unavailable>, count=<unavailable>)
    at /home/teawater/kernel2/linux/fs/read_write.c:469
Backtrace stopped: not enough registers or memory available to unwind further
(gdb) up
#1  0xffffffff8117a3ea in sys_read (fd=<optimized out>, buf=<unavailable>, count=<unavailable>)
    at /home/teawater/kernel2/linux/fs/read_write.c:469
469			ret = vfs_read(file, buf, count, &pos);
(gdb) p ret
$2 = -9

You see that the caller of function vfs_read is sys_read. And the local variable ret of sys_read is -9.

Use $dump_stack to output stack dump through printk

Because this way need parse the stack when tracing and call printk inside, so it will be slow, unsafe, unclear and cannot access a lot of info of call stack. So I suggest you use the prev way to do stack dump.
KGTP has special trace state variable $dump_stack, "collect" it will let Linux Kernel output stack dump through printk.
Following example lets Linux Kernel show the stack dump of vfs_readdir: 

target remote /sys/kernel/debug/gtp
trace vfs_readdir
  commands
    collect $dump_stack
  end

Then your kernel will printk like: 

[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
[22779.208068] Call Trace:
[22779.208072]  [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
[22779.208076]  [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
[22779.208080]  [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
[22779.208084]  [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
[22779.208088]  [<c023f152>] ? __find_get_block_slow+0xd2/0x160
[22779.208091]  [<c01a8c56>] ? delayacct_end+0x96/0xb0
[22779.208100]  [<c023f404>] ? __find_get_block+0x84/0x1d0
[22779.208103]  [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
[22779.208106]  [<c02e0838>] ? find_revoke_record+0xa8/0xc0
[22779.208109]  [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
[22779.208112]  [<c02db51f>] ? __jbd2_journal_temp_unlink_buffer+0x2f/0x110
[22779.208115]  [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
[22779.208118]  [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
[22779.208121]  [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
[22779.208124]  [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
[22779.208126]  [<c0227ac5>] ? vfs_readdir+0x15/0xb0
[22779.208128]  [<c0227ac4>] ? vfs_readdir+0x14/0xb0
[22779.208131]  [<c05b9743>] notifier_call_chain+0x43/0x60
[22779.208134]  [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
[22779.208137]  [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
[22779.208140]  [<c05b980d>] notify_die+0x2d/0x30
[22779.208142]  [<c05b71c5>] do_int3+0x35/0xa0

How to use performance counters

Performance counters are special hardware registers available on most modern CPUs. These registers count the number of certain types of hw events: such as instructions executed, cachemisses suffered, or branches mis-predicted - without slowing down the kernel or applications. These registers can also trigger interrupts when a threshold number of events have passed - and can thus be used to profile the code that runs on that CPU.

The Linux Performance Counter subsystem called perf event can get the value of performance counter. You can access it through KGTP perf event trace state variables.

Please goto read the file tools/perf/design.txt in Linux Kernel to get more info about perf event.

Define a perf event trace state variable

Access an performance counter need define following trace state variable: 

"pe_cpu_"+tv_name	Define the the CPU id of the performance counter.
"pe_type_"+tv_name	Define the the type of the performance counter.
"pe_config_"+tv_name	Define the the config of the performance counter.
"pe_en_"+tv_name	This the switch to enable or disable the performance counter.
			The performance counter is disable in default.
"pe_val_"+tv_name	Access this variable can get the value of the performance counter.

Define a per_cpu perf event trace state variable

Define a per_cpu perf event trace state variable is same with define HOWTO#Per_cpu_trace_state_variables. 

"pc_pe_"+perf_event type+string+CPU_id

Please note that if you define a per_cpu perf event trace state variable, you will not need define the cpu id("pe_cpu") because KGTP already get it.

The perf event type and config

The type of perf event can be: 

0	PERF_TYPE_HARDWARE
1	PERF_TYPE_SOFTWARE
2	PERF_TYPE_TRACEPOINT
3	PERF_TYPE_HW_CACHE
4	PERF_TYPE_RAW
5	PERF_TYPE_BREAKPOINT

If the type is 0(PERF_TYPE_HARDWARE), the config can be: 

0	PERF_COUNT_HW_CPU_CYCLES
1	PERF_COUNT_HW_INSTRUCTIONS
2	PERF_COUNT_HW_CACHE_REFERENCES
3	PERF_COUNT_HW_CACHE_MISSES
4	PERF_COUNT_HW_BRANCH_INSTRUCTIONS
5	PERF_COUNT_HW_BRANCH_MISSES
6	PERF_COUNT_HW_BUS_CYCLES
7	PERF_COUNT_HW_STALLED_CYCLES_FRONTEND
8	PERF_COUNT_HW_STALLED_CYCLES_BACKEND


If the type is 3(PERF_TYPE_HW_CACHE), the config need to divide to 3 parts: First one is cache id, it need be << 0 before set to config: 

0	PERF_COUNT_HW_CACHE_L1D
1	PERF_COUNT_HW_CACHE_L1I
2	PERF_COUNT_HW_CACHE_LL
3	PERF_COUNT_HW_CACHE_DTLB
4	PERF_COUNT_HW_CACHE_ITLB
5	PERF_COUNT_HW_CACHE_BPU

Second one is cache op id, it need be << 8 before set to config: 

0	PERF_COUNT_HW_CACHE_OP_READ
1	PERF_COUNT_HW_CACHE_OP_WRITE
2	PERF_COUNT_HW_CACHE_OP_PREFETCH

Last one is cache op result id, it need be << 16 before set to config: 

0	PERF_COUNT_HW_CACHE_RESULT_ACCESS
1	PERF_COUNT_HW_CACHE_RESULT_MISS

If you want get the perf count of PERF_COUNT_HW_CACHE_L1I(1), PERF_COUNT_HW_CACHE_OP_WRITE(1) and PERF_COUNT_HW_CACHE_RESULT_MISS(1), you can use: 

(gdb) tvariable $pe_config_cache=1 | (1 << 8) | (1 << 16)

tools/perf/design.txt in Linux Kernel have more info about type and config of perf event.

Enable and disable all the perf event in a CPU with $pc_pe_en

I think the best way that count a part of code with performance counters is enable all the count in the begin of the code and disable all of them in the end. You can do it with "pe_en". But if you have a lot of perf event trace state variables. That will make the tracepoint action very big. $pc_pe_en is for this issue. You can enable all the perf event trace state variables in current CPU with following action: 

>teval $pc_pe_en=1

Disable them with set $pc_pe_en to 0. 

>teval $pc_pe_en=0

GDB scripts to help with set and get the perf event trace state variables

Following is a GDB script define two commands dpe and spe to help define and show the perf event trace state variables.
You can put it to the ~/.gdbinit or your tracepoint script. Then you can use this two commands in GDB directly. 

define dpe
  if ($argc < 2)
    printf "Usage: dpe pe_type pe_config [enable]\n"
  end
  if ($argc >= 2)
    set $tmp=0
    while $tmp<$cpu_number
      eval "tvariable $pc_pe_type_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg0
      eval "tvariable $pc_pe_config_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg1
      eval "tvariable $pc_pe_val_%d%d_%d=0",$arg0, $arg1, $tmp
      if ($argc >= 3)
        eval "tvariable $pc_pe_en_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg2
      end
      set $tmp=$tmp+1
    end
  end
end

document dpe
Usage: dpe pe_type pe_config [enable]
end

define spe
  if ($argc != 2 && $argc != 3)
    printf "Usage: spe pe_type pe_config [cpu_id]\n"
  end
  if ($argc == 2)
    set $tmp=0
    while $tmp<$cpu_number
      eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $tmp
      set $tmp=$tmp+1
    end
  end
  if ($argc == 3)
    eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $arg2
  end
end

document spe
Usage: spe pe_type pe_config [cpu_id]
end

Following is an example to use it get the performance counters of function tcp_v4_rcv: 

#Connect to KGTP
(gdb) target remote /sys/kernel/debug/gtp
#Define 3 pe tvs for PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_CACHE_MISSES and PERF_COUNT_HW_BRANCH_MISSES.
(gdb) dpe 0 0
(gdb) dpe 0 3
(gdb) dpe 0 5
#enable the performance counters of this CPU in the begin of this function.
(gdb) trace tcp_v4_rcv
(gdb) action
>teval $pc_pe_en=1
>end
#$kret make this hanler the end of function tcp_v4_rcv.
(gdb) trace *(tcp_v4_rcv)
(gdb) action
>teval $kret=0
#disable all performance counters of this CPU
>teval $pc_pe_en=0
#Access the per cpu perf event tv will access to the current cpu pe tv.
>collect $pc_pe_val_00_0
>collect $pc_pe_val_03_0
>collect $pc_pe_val_05_0
#Set all the pe tv to 0
>teval $pc_pe_val_00_0=0
>teval $pc_pe_val_03_0=0
>teval $pc_pe_val_05_0=0
>end
tstart
#Wait some time that current pc receive some tcp package.
(gdb) tstop
(gdb) tfind
(gdb) spe 0 0 $cpu_id
$pc_pe_val_00_2=12676
(gdb) spe 0 3 $cpu_id
$pc_pe_val_03_2=7
(gdb) spe 0 5 $cpu_id
$pc_pe_val_05_2=97

Howto let tracepoint output value directly

In the previous parts, you may understand that to get a value from Linux kernel, you need to use a tracepoint "collect" action to save the value to the tracepoint frame and use the GDB command "tfind" to parse the value from the frame data.
But we want get the value directly sometimes, so KGTP supports two ways to output values directly.

Switch collect to output the value directly

KGTP has special trace state variables $printk_level, $printk_format and $printk_tmp to support this function.
$printk_level: if its value is 8 (this is the default value), "collect" action will save value to the tracepoint frame in the simple behavior.
If its value is 0-7, "collect" will output the value through "printk" directly, and value will be the level of printk. The level is: 

0	KERN_EMERG	system is unusable
1	KERN_ALERT	action must be taken immediately
2	KERN_CRIT	critical conditions
3	KERN_ERR	error conditions
4	KERN_WARNING	warning conditions
5	KERN_NOTICE	normal but significant condition
6	KERN_INFO	informational
7	KERN_DEBUG	debug-level messages

$printk_format, collect printk will output value in the format that is set by it. The format is: 

0	This is the default value.
	If the size of collect value is 1, 2, 4 or 8, it will be output as an unsigned decimal.
	If not, it will be output as a hexadecimal string.
1	Output value in signed decimal.
2	Output value in unsigned decimal.
3	Output value in unsigned hexadecimal.
4	Output value as a string.
5	Output value as a hexadecimal string.

$printk_tmp, to output the value of global variable need set to it first.

Following example shows a count number, pid, jiffies_64 and the file name that call vfs_readdir: 

(gdb) target remote /sys/kernel/debug/gtp
(gdb) tvariable $c
(gdb) trace vfs_readdir
(gdb) actions
>teval $printk_level=0
>collect $c=$c+1
>collect ((struct task_struct *)$current_task)->pid
>collect $printk_tmp=jiffies_64
>teval $printk_format=4
>collect file->f_path.dentry->d_iname
>end

Then your kernel will printk like: 

gtp 1:$c=$c+1=41
gtp 1:((struct task_struct *)$current_task)->pid=12085
gtp 1:$printk_tmp=jiffies_64=4322021438
gtp 1:file->f_path.dentry->d_iname=b26
gtp 1:$c=$c+1=42
gtp 1:((struct task_struct *)$current_task)->pid=12085
gtp 1:$printk_tmp=jiffies_64=4322021438
gtp 1:file->f_path.dentry->d_iname=b26

"gtp 1" means that it was output by tracepoint 1.

Howto show a variable whose value has been optimized away

Sometimes, GDB will output some value like: 

inode has been optimized out of existence.
res has been optimized out of existence.

That is because value of inode and res is optimized. Linux Kernel is built with -O2 so you will get this trouble sometimes.
There are 2 ways to handle it:

Update your GCC

The VTA branch http://gcc.gnu.org/wiki/Var_Tracking_Assignments was merged for GCC 4.5. This helps a lot with generating dwarf for previously "optimized out" values.

How to get the function pointer point to

If the debug info of the function pointer is not optimized out

You can collect it directly and print what it point to. For example: 

377			count = ret;
378			if (file->f_op->read)
379				ret = file->f_op->read(file, buf, count, pos);
(gdb) 
(gdb) trace 379
Tracepoint 1 at 0xffffffff81173ba5: file /home/teawater/kernel/linux/fs/read_write.c, line 379.
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect file->f_op->read
>end
(gdb) tstart
(gdb) tstop
(gdb) tfind
(gdb) p file->f_op->read
$5 = (ssize_t (*)(struct file *, char *, size_t, loff_t *)) 0xffffffff81173190 <do_sync_read>
#Then you know file->f_op->read point to do_sync_read.

If the debug info of the function pointer is optimized out

You can use tracepoint step to handle it. For example: 

#Find out which instrunction that it is called.
(gdb) disassemble /rm vfs_read
379				ret = file->f_op->read(file, buf, count, pos);
   0xffffffff81173ba5 <+181>:	48 89 da	mov    %rbx,%rdx
   0xffffffff81173ba8 <+184>:	4c 89 e9	mov    %r13,%rcx
   0xffffffff81173bab <+187>:	4c 89 e6	mov    %r12,%rsi
   0xffffffff81173bae <+190>:	4c 89 f7	mov    %r14,%rdi
   0xffffffff81173bb1 <+193>:	ff d0	callq  *%rax
   0xffffffff81173bb3 <+195>:	48 89 c3	mov    %rax,%rbx
(gdb) trace *0xffffffff81173bb1
Tracepoint 1 at 0xffffffff81173bb1: file /home/teawater/kernel/linux/fs/read_write.c, line 379.
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>while-stepping 1
 >collect $reg
 >end
>end
(gdb) tstart 
(gdb) tstop 
(gdb) tfind 
#0  tty_read (file=0xffff88006ca74900, buf=0xb6b7dc <Address 0xb6b7dc out of bounds>, count=8176, 
    ppos=0xffff88006e197f48) at /home/teawater/kernel/linux/drivers/tty/tty_io.c:960
960	{
#Then you know file->f_op->read point to tty_read.

Please note that while-stepping will make tracepoint cannot use kprobes-optimization.

/sys/kernel/debug/gtpframe and offline debug

/sys/kernel/debug/gtpframe supplies trace frame in tfile format (GDB can parse it) when KGTP is stop.

In the PC that can run the GDB:
Change the "target remote XXXX" to 

(gdb) target remote | perl ./getgtprsp.pl

After that, set tracepoint and start it as usual: 

(gdb) trace vfs_readdir
Tracepoint 1 at 0xffffffff8114f3c0: file /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
#If your GDB support tracepoint "printf" (see "Howto use tracepoint printf"), use it to show the value directly is better.
>collect $reg
>end
(gdb) tstart 
(gdb) stop 
(gdb) quit

Then you can find files gtpstart and gtpstop in current directory. Copy it to the machine that you want to debug.

In the debugged machine after insmod the gtp.ko:
Start the tracepoint: 

cat gtpstart > /sys/kernel/debug/gtp

Stop the tracepoint: 

cat gtpstop > /sys/kernel/debug/gtp

You can let Linux Kernel show the value directly, please see HOWTO#Howto_let_tracepoint_output_value_directly.

If you want to save the value to the trace frame and parse later, you can use file "/sys/kernel/debug/gtpframe" that has the trace frame. Copy it to the PC that has GDB.
Please note that some "cp" cannot handle it very well, please use "cat /sys/kernel/debug/gtpframe > ./gtpframe" to copy it.
In the PC that can run the GDB:

(gdb) target tfile ./gtpframe
Tracepoint 1 at 0xffffffff8114f3dc: file /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
Created tracepoint 1 for target's tracepoint 1 at 0xffffffff8114f3c0.
(gdb) tfind 
Found trace frame 0, tracepoint 1
#0  vfs_readdir (file=0xffff880036e8f300, filler=0xffffffff8114f240 <filldir>, buf=0xffff880001e5bf38)
    at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
24	{

How to use /sys/kernel/debug/gtpframe_pipe

This interface supplies same format trace frame with "gtpframe". But it can work when KGTP is running. After data is read, it will auto deleted from trace frame like "trace_pipe" of ftrace.

Get the frame info with GDB

#connect to the interface
(gdb) target tfile /sys/kernel/debug/gtpframe_pipe
#Get one trace frame entry
(gdb) tfind 0
Found trace frame 0, tracepoint 1
#Get the next one
(gdb) tfind 
Target failed to find requested trace frame.
(gdb) tfind 0
Found trace frame 0, tracepoint 1

This way is better to work with python to parse Kernel. add-ons/hotcode.py is an example of python script.

Get the frame info with cat

sudo cat /sys/kernel/debug/gtpframe_pipe > g

Then all the trace frame will be saved in file "g".

Get the frame info with getframe

KGTP package include a program "getframe" can help you save the trace frame to files.
Following part is the help of it: 

getframe -h
Get the trace frame of KGTP and save them in current 
directory with tfile format.
Usage: ./getframe [option]

  -g n    Set the minimum free size limit to n G.
          When free size of current disk is smaller than n G,
          ./getframe will exit (-q) or wait some seconds (-w).
          The default value of it is 2 G.

  -q      Quit when current disk is smaller than
          minimum free size limit (-g).

  -w n    Wait n seconds when current disk is smaller
          than minimum free size limit (-g).

  -e n    Set the entry number of each tfile to n.
          The default value of it is 1000.

  -h      Display this information.

Use $pipe_trace

For the lock safe, KGTP will ignore the task that read the /sys/kernel/debug/gtpframe_pipe in default.
If you really need trace this task, and be sure that is safe. You can use following command before call "tstart": 

(gdb) tvariable $pipe_trace=1

Then KGTP will not ignore the task that read /sys/kernel/debug/gtpframe_pipe.

How to use add-ons/hotcode.py

This script can show the hotest code line in the Linux kernel or user space program through parse and record the pc address in the interrupt handler.
Please goto http://code.google.com/p/kgtp/wiki/hotcode see howto use it.


► Sign in to add a comment
Powered by Google Project Hosting