Debugger Based Debugging

If traces alone are not sufficient, you'll need to inspect what gem5 is doing in detail using a debugger (e.g., gdb). You definitely want to use the gem5.debug binary if you reach this point. Ideally, looking at traces should at least allow you to narrow down the range of cycles in which you think something is going wrong. The fastest way to reach that point is to use a DebugEvent, which goes on gem5's event queue and forces entry into the debugger when the specified cycle is reached by sending the process a SIGTRAP signal. You'll need to to start gem5 under the debugger or have the debugger attached to the gem5 process for this to work.

You can create one or more DebugEvents when you invoke gem5 using the "--debug-break=100" parameter. You can also create new DebugEvents from the debugger prompt using the schedBreak() function. The following example session illustrates both of these approaches:

% gdb m5/build/ALPHA/gem5.debug
GNU gdb 6.1
Copyright 2002 Free Software Foundation, Inc.
[...]
(gdb) run --debug-break=2000 configs/run.py 
Starting program: /z/stever/bk/m5/build/ALPHA/gem5.debug --debug-break=2000 configs/run.py 
M5 Simulator System
[...]
warn: Entering event queue @ 0.  Starting simulation...

Program received signal SIGTRAP, Trace/breakpoint trap.
0xffffe002 in ?? ()
(gdb) p curTick
$1 = 2000
(gdb) c
Continuing.

(gdb) call schedBreak(3000)
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0xffffe002 in ?? ()
(gdb) p _curTick
$3 = 3000
(gdb)

gem5 includes a number of functions specifically intended to be called from the debugger (e.g., using the gdb 'call' command, as in the schedBreak() example above). Many of these are "dump" functions which display internal simulator data structures. For example, eventq_dump() displays the events scheduled on the main event queue. Most of the other dump functions are associated with particular objects, such as the instruction queue and the ROB in the detailed CPU model. These include:

function	Effect
schedBreak(<tick>)	Schedule a SIGTRAP to occur at <tick>
setDebugFlag("<flag>")	Enable a debug flag from the debugger
clearDebugFlag("<flag>")	Disable a debug flags from the debugger
eventqDump()	Print out all events on the event queue
takeCheckpoint(<tick>)	Create a checkpoint at cycle <tick>

Debugging simulated code with GDB

gem5 has built-in support for gdb's remote debugger interface. If you are interested in monitoring what the kernel on the simulated machine is doing, you can fire up kgdb on the host platform and have it talk to the simulated gem5 system as if it were a real machine (only better, since gem5 executions are deterministic and gem5's remote debugger interface is guaranteed not to perturb execution on the simulated system). To use a remote debugger with gem5, the most important part is that you have gdb compiled to work with the target system you're simulating (e.g. alpha-linux if simulating an Alpha target, arm-linux if simulating an ARM target, etc). It is possible to compile an non-native architecture gdb on an x86 machine for example. All that must be done is add the --target= option to configure when you compile gdb. You may also get pre-compiled debuggers with cross compilers. See Download for links to some cross compilers that include debuggers.

% wget http://ftp.gnu.org/gnu/gdb/gdb-6.3.tar.gz
--08:05:33--  http://ftp.gnu.org/gnu/gdb/gdb-6.3.tar.gz
           => `gdb-6.3.tar.gz'
Resolving ftp.gnu.org... done.
Connecting to ftp.gnu.org[199.232.41.7]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17,374,476 [application/x-tar]

100%[====================================>] 17,374,476   216.57K/s    ETA 00:00

08:06:52 (216.57 KB/s) - `gdb-6.3.tar.gz' saved [17374476/17374476]

% tar xfz gdb-6.3.tar.gz
% cd gdb-6.3
% ./configure --target=alpha-linux
<configure output....>
% make
<make output...this may take a while>

The end result is gdb/gdb which will work for remote debugging.

When gem5 is run, each CPU listens for a remote debugging connection on a TCP port. The first port allocated is generally 7000, though if a port is in use, the next port will be tried.

To attach the remote debugger, it's necessary to have a copy of the kernel and of the source. Also to view the kernel's call stack, you must make sure Linux was built with the necessary debug configuration parameters enabled. To run the remote debugger, do the following:

ziff% gdb-linux-alpha arch/alpha/boot/vmlinux
GNU gdb 
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=alpha-linux"...
(no debugging symbols found)...
(gdb) set remote Z-packet on                [ This can be put in .gdbinit ]
(gdb) target remote ziff:7000
Remote debugging using ziff:7000
0xfffffc0000496844 in strcasecmp (a=0xfffffc0000b13a80 "", b=0x0)
    at arch/alpha/lib/strcasecmp.c:23
23              } while (ca == cb && ca != '\0');
(gdb) 

The gem5 simulator is already running and the target remote command connects to the already running simulator and stops it in the middle of execution. You can set breakpoints and use the debugger to debug the kernel. It is also possible to use the remote debugger to debug console code and palcode. Setting that up is similar, but a how to will be left for future work.

If you're using both the remote debugger and the debugger on the simulator, it is possible to trigger the remote debugger from the main debugger by doing a 'call debugger()'. Before you do this you'll need to figure out what CPU (the cpu id) you want to debug and set current_debugger to that cpuid. If you only have one cpu, then it will be cpuid 0, however if there are multiple cpus you will need to match the cpu id with the corresponding port number for the remote gdb session. For example, using the following sample output from M5, calling the kernel debugger for cpu 3 requires the kernel debugger to be listening on port 7001.

%./build/ALPHA/gem5.debug configs/example/fs.py
...
making dual system
Global frequency set at 1000000000000 ticks per second
Listening for testsys connection on port 3456
Listening for drivesys connection on port 3457
0: testsys.remote_gdb.listener: listening for remote gdb #0 on port 7002
0: testsys.remote_gdb.listener: listening for remote gdb #1 on port 7003
0: testsys.remote_gdb.listener: listening for remote gdb #2 on port 7000
0: testsys.remote_gdb.listener: listening for remote gdb #3 on port 7001
0: drivesys.remote_gdb.listener: listening for remote gdb #4 on port 7004
0: drivesys.remote_gdb.listener: listening for remote gdb #5 on port 7005
0: drivesys.remote_gdb.listener: listening for remote gdb #6 on port 7006
0: drivesys.remote_gdb.listener: listening for remote gdb #7 on port 7007

ARM Target

If you're planning to debug an ARM kernel you'll need a reasonably new version of gdb (7.1 or greater). Additionally, you'll have to manually specify the tspecs like this (port number may be different). The tspec file is available in the gdb source code:

set remote Z-packet on
set tdesc filename path/to/features/arm-with-neon.xml
symbol-file <path to vmlinux used for gem5>
target remote <ip addr of host running gem5 or if local host 127.0.0.1>:7000

Debugging Python with PDB

You can debug configuration scripts with the Python debugger (PDB) just as you would other Python scripts. You can enter PDB before your configuration script is executed by giving the --pdb argument to the gem5 binary. Another approach is to put the following line in your configuration script (e.g., fs.py or se.py) wherever you would like to enter the debugger:

import pdb; pdb.set_trace()

Note that the Python files under src are compiled in to the gem5 binary, so you must rebuild the binary if you add this line (or make other changes) in these files. Alternatively, you can set the M5_OVERRIDE_PY_SOURCE environment variable to "true" (see src/python/importer.py).

See the official PDB documentation for more details on using PDB.