WinDbg debugger allows you to debug all modern versions of Windows using a built-in kernel debugger and either COM or IEEE1394 port. Let's see how is it implemented. To start windows in Kernel Debugging mode, you specify additional parameters in boot.ini file that look like this:
Let's now see what actually happens when NTOSKRNL detects that it was started with /DEBUG parameter. First of all, it analyzes the /DEBUGPORT parameter from boot.ini and determines what packet-level plugin (KD extension DLL in Microsoft terminology) to load. For COM-based debugging the plugin DLL is called KDCOM.DLL, for IEEE1394-based debugging it is called KD1394.DLL. Fortunately, when you specify something like /DEBUGPORT=FOO, NTOSKRNL will try to load KDFOO.DLL and use it as a KD extension DLL. As we are providing our own DLL making a fast interface to WinDbg, that expects a named pipe from a virtual COM port, we need to solve two problems here:
Note that in Windows Vista the kernel debugging flags are specified using bcdedit.exe utility and cannot specify a non-standard KD extension DLL. The only way to load KDVMWare DLL to kernel is to replace a standard one, for example, KD1394.DLL.
Let's explore the structure of a typical KD extension DLL. A quick analysis of KDCOM.DLL shows that it exports the following functions:
Let's analyze how this functions work.
As it is evident from their names, two functions are used to initialize a KD extension DLL: KdDebuggerInitialize0() and KdDebuggerInitialize1(). Fortunately, Microsoft provides PDB file for WinXP version of KDCOM.DLL. Additionally, some of the functions are described by Ken Johnson (http://www.nynaeve.net/?p=169). Let's use the PDB to recover declarations for the initialization functions:
The first function performs initial initialization of a KD extension DLL. For example, it can read the parameters specified in BOOT.INI using the LOADER_PARAMETER_BLOCK::LoadOptions field. For example, KDCOM.DLL can get determine the COM port number to use and its baud rate. Both initialization functions return a NTSTATUS value with STATUS_SUCCESS corresponding to successful completion. Note that if an initialization function such as KdDebuggerInitialize0() returns an unsuccessful status, kernel is started without debugging support and the DLL is not actually used any more.
All communication between kernel and a kernel debugger is packet-based. The following rules describe the packet behavior:
Each buffer is represented by a STRING structure defined in NTDEF.H (on x64 systems the data pointer is aligned at 8-byte boundary):
In KDVMWare this structure is redefined as KD_BUFFER.
A special structure called KD_CONTEXT maintains the global state for KD packet layer:
The RetryCount member is set before a call to KdSendPacket() and specifies the number of retries for a droppable packet to set. A droppable packet is a packet that can be simply dropped if no acknowledgment comes from WinDbg after some number of retries (KdSendPacket() will just return). The BreakInRequested is set to TRUE by KdReceivePacket() if WinDbg has requested a kernel breakpoint (ctrl+break was pressed, or WinDbg was just started). The break-in request is not a part of a packet and is transferred separately (see KDCOM protocol description below).
Here are the definitions for packet sending and receiving functions:
KD_RECV_CODE
NTAPI KdReceivePacket(__in ULONG PacketType,
__inout_opt PKD_BUFFER FirstBuffer,
__inout_opt PKD_BUFFER SecondBuffer,
__out_opt PULONG PayloadBytes,
__inout_opt PKD_CONTEXT KdContext);
The KdReceivePacket() return value can be defined as a following enumeration:
The PacketType parameter specifies the type of the packet being sent or being received (all packets with other types should be ignored), however there is one exception. When PacketType is set to 8 in a KdReceivePacket() call, the function checks whether there is any data available (for example, whether the COM port buffer is non-empty), and returns immediately either KD_RECV_CODE_OK or KD_RECV_CODE_TIMEOUT.
A KD extension DLL exports some additional functions that are not directly involved in packet sending/receiving and can simply return STATUS_SUCCESS in most of implementations:
The information about these functions was taken from RectOS documentation pages. Although Microsoft implementation can be different from ReactOS one, just returning STATUS_SUCCESS from these functions should work.
Another problem to be solved in order to connect kernel and WinDbg using a custom KD extension DLL is the protocol that KDCOM.DLL uses to transfer packets over a COM port. As WinDbg receives and sends KDCOM packets when connected to a kernel using a named pipe, our tool should be able to produce and to parse such packets. In KDVMWare these packets are processed in KDCLIENT.DLL on host side, however, in Microsoft implementation, all packet processing logic is implemented inside KDCOM.DLL. Let's see, how it works.
First of all, there are two kinds of packets: control packets and data packets. Data packets directly transfer KdSendPacket()/KdReceivePacket() data, while control packets signalize receive acknowledgment, retry requests, resync requests, etc. Each control packet consists of a packet header, a data block, and a terminating byte (0xAA). A data block contains contents of two buffers, one after another, with no indication of where one ends and another starts. Moreover, sender and receiver can use split the packet data in different ways:
Let's define a C structure describing the packet header:
Packet signature is either 0x30303030 ('0000') for data packets, or 0x69696969 for control packets ('iiii'). Packet type specifies the exact type of the packet. Types for control and data packets are members of the same enumeration:
As it was described before, packet type 8 is not used as a packet type. Instead, when KdReceivePacket() is called with that value, it checks whether any data can be received from WinDbg and returns immediately.
Packet ID is used to detect if a single packet was missed, as the least significant bit of a packet ID toggles with every new packet sent. The initial packet ID is 0x80800800, however, resync command sets it to 0x80800000. Checksum is just an arithmetic sum of all bytes from the data section of the packet.
As I have discovered after development of KDVMWare, a file named windbgkd.h was included in Windows 2000 DDK and contained information about KDCOM protocol internals. The ReactOS version containing most of the information from it can be found here. According to that file, the following packet types are actually used:
Let's discuss the types of control packets and their roles in KDCOM protocol:
To illustrate, how KDCOM packet layer works, let's check out some examples:
Some packets are "droppable". It means that KdSendPacket() may return control when such a packet was not acknowledged by WinDbg after some number of retries. KDCOM.DLL treats the following packets as droppable:
Packet subtype (ApiNumber) is the first DWORD in the packet data block. The types referenced here is defined in the following way in windbgkd.h:
There is one significant detail in original KDCOM implementation. When the KDCOM.DLL initializes, it reinitializes the COM port and resets its buffer. A named pipe implementation should do the same. In other case, the following scenario is possible:
To avoid this problem, KDVMWare simply clears the named pipe receive buffer when it receives a resync packet.
All KDCOM-related functionality is implemented in the KdComDispatcher class. Feel free to explore its documentation using the link above.