Linux Kernel Debugging Overview

This page gives an overview of Linux debugging process used by VisualKernel and explains which components are involved in it.

Debugging a Linux kernel always involves two machines: one machine running the kernel itself and another machine running the debugger (GDB). If the debugged machine is stopped on a kernel breakpoint, it cannot access any files, so the machine with the debugger needs to have a local copy of the kernel image and the relevant source files:

Connecting a debugger to the kernel

There are 2 different ways to connect a debugger to the Linux kernel:

  • Using KGDB, the built-in debugger included in the Linux kernel.
  • Using external debugging tool that can access the memory and registers of the debugged computer. E.g. VMWare provides a GDB stub that can be used to debug a Linux system running on a VMWare machine. An embedded box like Raspberry PI can be debugged using an external JTAG debugger.

Both methods have advantages and disadvantages. KGDB is relatively slow, but will work on any setup that provides a COM port on the debugged machine (including a virtual COM port supported by VMWare/VirtualBox). As KGDB is a part of the kernel, if a massive memory corruption occurs in the kernel, it can also damage KGDB and stop you from debugging.

Using an external debug interface can be more reliable and faster, however it highly depends on the interface. E.g. the VMWare GDB stub is very fast compared to KGDB, while most of JTAG debuggers are fairly slow. The advantage, however, is that no matter what happens to the OS, you should be able to debug it as the debugger runs on top of the kernel and not inside it.

VisualKernel supports both options and allows choosing between them on the last page of the project wizard and in the VisualKernel Project Properties dialog:

Symbols and sources

In order to debug your kernel, VisualKernel needs to store a copy of the kernel image with symbols and the related sources on the machine where it runs GDB. It can be either your Windows machine or a second Linux machine (you cannot store it on the same machine that is being debugged).

The sources and symbols are normally not present on the Linux machine. Instead they are available as separate packages for most of popular distros. VisualKernel can fetch them automatically in most of the cases. Note that sources/symbols for each kernel version occupy 2-5GB of disk space. Use the Tools->Manage Linux Kernel Symbols command in Visual Studio to manage the symbol packages that have been cached on your machine:

If you are using a 2-machine setup (build your code on one Linux machine and deploy it on another one), the files will be stored on the build machine and not on the Windows machine.

Each kernel symbol directory has the following contents:

  • The kernel image with symbols (typically called vmlinux). It should 100% match the kernel image on the debugged machine (i.e. the image on the debugged machine should be a stripped version of the vmlinux file inside the symbol directory).
  • A source folder containing kernel source files.
  • A folder containing symbols for in-tree modules.
  • A description file (KernelRelease.xml). It contains the version of the kernel, lists in-tree modules and other properties.

Building and deploying

When you use one-machine configuration, VisualKernel will do the following each time you build your module:

  • Transfer the updated source files to the Linux machine
  • Build the kernel module on the Linux machine
  • Transfer the module with symbols from the Linux machine back to the Windows machine

When you start debugging your module, VisualKernel will do the following:

  • Find a kernel symbol package on your Windows machine that matches the remembered kernel version
  • Start a GDB session using symbols and sources from that package
  • Connect to the debugged machine using SSH
  • Launch your kernel module
  • Load the symbols transferred to the Windows machine during build

The module symbol file is stored in the VisualGDBCache subdirectory inside your project directory.