How to Use `kgdb` for Kernel Debugging on FreeBSD Operating System

How to Use kgdb for Kernel Debugging on FreeBSD Operating System

Introduction

Kernel debugging is a crucial skill for FreeBSD developers and system administrators dealing with kernel crashes, panics, or unexpected behavior. The kgdb (Kernel GNU Debugger) tool provides an efficient way to inspect and analyze kernel core dumps. It allows developers to step through kernel code, inspect variables, and trace stack frames to diagnose kernel issues effectively.

This guide will walk you through setting up kgdb, capturing crash dumps, and using kgdb to debug kernel issues on FreeBSD.

Prerequisites

Before using kgdb for debugging FreeBSD’s kernel, ensure you have:

• A FreeBSD system (either physical or virtual machine) with kernel debugging enabled.
• Kernel debugging symbols installed (kernel.debug).
• A core dump (vmcore) from a kernel panic.
• The kgdb utility installed.

1. Enabling Kernel Debugging and Crash Dumps

To debug the FreeBSD kernel, you need debugging symbols and core dumps enabled.

Enable Kernel Debugging Symbols

By default, the FreeBSD kernel does not include debugging symbols. To enable them:

1. Edit the kernel configuration file (e.g., /usr/src/sys/amd64/conf/MYKERNEL). Add:

   makeoptions DEBUG=-g
   options     DDB
   options     GDB
   options     KDB
   options     KDB_TRACE
   

2. Compile and install the kernel:

   cd /usr/src
   make buildkernel KERNCONF=MYKERNEL
   make installkernel KERNCONF=MYKERNEL
   

3. Reboot the system to load the new kernel:

   reboot
   

Enable Crash Dumps

To capture kernel crash dumps, configure the system to save core dumps:

1. Edit /etc/rc.conf and add:

   dumpdev="AUTO"
   dumpdir="/var/crash"
   

2. Verify the dump device:

   dumpon -l
   

3. Ensure enough space is available in /var/crash.

2. Capturing a Kernel Crash Dump

When a kernel panic occurs, FreeBSD writes the crash dump to the configured dump device. Upon reboot, the system moves the dump to /var/crash and generates a text file with debugging information.

To manually trigger a crash for testing:

sysctl debug.kdb.panic=1

After rebooting, check the crash dump:

ls /var/crash

You should see files like vmcore.0 and info.0.

3. Using kgdb to Analyze Kernel Dumps

Once you have a kernel dump, you can analyze it using kgdb.

Load the Core Dump

Run kgdb with the debugging kernel (kernel.debug) and the core dump (vmcore):

kgdb /boot/kernel/kernel.debug /var/crash/vmcore.0

You should see output like:

[GDB will be used to debug FreeBSD Kernel]
Reading symbols from /boot/kernel/kernel.debug... done.
Loaded core dump from vmcore.0

Inspect the Crash

1. Check the Backtrace

   bt
   

   This command shows the function calls leading to the crash.

2. Examine Registers

   info registers
   

   This displays the CPU registers at the time of the crash.

3. Inspect Memory and Variables

   x/10x $rsp
   

   This examines memory around the stack pointer.

4. Find the Faulting Instruction

   list *$rip
   

   This prints the source code around the instruction pointer.

5. Inspect Kernel Threads

   info threads
   thread apply all bt
   

   These commands list kernel threads and their backtraces.

4. Remote Kernel Debugging with kgdb

For live kernel debugging, you can connect to a remote system using kgdb over a serial connection.

Configure the Target System (Debuggee)

1. Add the following to /boot/loader.conf:

   gdb_enable="YES"
   ``
   - Set up a serial console for debugging:
   ```sh
   echo '-D -S115200' > /boot.config
   

   • Reboot the system.

2. Manually trigger kgdb on the target:

   sysctl debug.kdb.enter=1
   

Configure the Host System (Debugger)

On a second FreeBSD system, connect via serial cable and run:

kgdb -n /dev/cuau0

This starts a remote debugging session, allowing you to inspect the live kernel.

5. Common kgdb Commands

Conclusion

Using kgdb for kernel debugging on FreeBSD provides deep insights into kernel crashes and panics. By enabling kernel debugging symbols, configuring crash dumps, and using kgdb effectively, developers can diagnose and fix kernel issues efficiently. Whether analyzing core dumps or debugging remotely, kgdb is a powerful tool for FreeBSD kernel debugging.