Principle of operation and main components
Normal kernel debugging
Normally, when the kernel of the OS being debugged wants to send a packet to the debugger, it calls the KdSendPacket() function inside a corrseponding extension DLL (KDCOM.DLL for COM port-based debugging; KD1394.DLL for 1394-based). The code in DLL sends data over the physical port by directly accessing the hardware (avoiding the use of any other kernel functions that can be debugged themselves). VMWare emulates a COM port, so all data sent to it using conventional UART controller interface is directed to a named pipe available on host computer. The debugger (WinDBG/KD) reads data from the pipe and interprets it. All communication between debugger and OS being debugged is framed in packets, so the lowest level is absolutely transparent to the kernel calling KdSendPacket()/KdReceivePacket(). Most debugger functionality exposed by KD is contained inside the DBGENG.DLL library, including the code parsing reading stream data from the pipe and restoring packets from it.Boosted kernel debugging
As it was described before, normally kernel uses the conventional UART controller interface to send data to the debugger. VirtualKD project overrides this behavior.First of all, the KDVM.DLL library implements all the functions exported by KDCOM.DLL and KD1394.DLL. As it is described in readme.txt, there are two ways of making Windows use our DLL: either to specify /DEBUGPORT=VM in BOOT.INI , or using the KDPATCH driver.
When KDSendPacket() or KdReceivePacket() inside the KDVM.DLL is called, it serializes all its parameters in a single data block and sends it to VMWare using the GuestRPC mechanism. On the host side, KDCLIENT.DLL receives the data block and calls the IKdComDispatcher::KdSendPacket() or IKdComDispatcher::KdReceivePacket() methods. This methods are implemented inside the KdComDispatcher class. Basically, the KDVM.DLL marshalls the calls, while KDCLIENT unmarshalls and executes them. This behaviour can be slightly modified using a Named Pipe Proxy PIPEPROXY module to simplify debugging.
The KdSendPacket()/KdReceivePacket() contained in KdComDispatcher class perform the same functionality as the corresponding methods in KDCOM.DLL with one important difference: instead of sending/receiving the packets using a slow UART interface, they operate with a named pipe directly (they already run on the host side, as KDVM.DLL/KDCLIENT.DLL perform call marshalling), transferring the debugging data very rapidly. From all other points of view, the functionality is identical to KDCOM.DLL one: packet building, checksum computation, packet acknowledging, resynchronizing, etc.
Schematically, a single debug packet transfer can be represented by the following diagram:
KDPATCH driver
The KDPATCH driver locates KDCOM.DLL loaded in memory by NTOSKRNL, finds the KdSendPacket() and KdReceivePacket() functions, and inserts jump instructions in the beginning of them to divert control to the corresponding functions in KDVM.DLL. The driver supports unloading, in that case it simply restores the original contents of KdSendPacket()/KdReceivePacket() allowing the COM port-based debugging to used again.GuestRPC mechanism
VMWare supports the so-called backdoor mechanism allowing to exchange various data between virtual machine and a host. The backdoor interface is described here and is used in the open-vm-tools project. Aside some simple functions, such as querying/setting mouse cursor position, the backdoor interface allows executing commands on host. This mechanism is called GuestRPC. A simple GuestRPC client is called vmwareservice.exe and is located in VMWare Tools folder on the guest machine. It allows executing several GuestRPC commands, for example, the following command returns the IP address of the virtual machine:VMWareService.exe --cmd info-get guestinfo.ipThe info-get here is the command name, while the "guestinfo.ip" is the command parameter. VMWare process (vmware-vmx.exe) holds an internal table of handlers, that associates each command name with a corresponding handler function that receives the command parameters and can also return some data back to VM. Both command name and arguments can be either printable strings, or binary data sequences, as the handler table contains both pointer to the command name and its length. However, according to comments in open-vm-tools, in older versions of VMWare the command names were required to be printable and to contain no spaces.
Patching VMWare
VMWare patching is implemented using the KDCLIENT.DLL library. When it is loaded to VMWARE-VMX.EXE process, it locates the GuestRPC handler table, finds an unused entry and initializes it with a pointer to VirtualKD command handler inside the DLL. As the handler table is an internal part of VMWARE-VMX process and is not exported, its address is unknown. KDCLIENT.DLL uses a kind of a heuristic algorithm to detect the table location. The algorithm is implemented in the RPCTableManager class.Once a table is found, the information about it is saved in a patch database. The information about the VMWARE-VMX.EXE size and timestamp is used as a key, and the GuestRPC handler table address and size are saved. This mechanism speeds up patching the table when a previously used version of VMWARE-VMX.EXE is detected, as the information about the table is directly read from the corresponding .VMPATCH file instead of performing a long scanning operation.
Named Pipe Proxy
As it was described above, the KDCOM packet generation/processing logic is implemented inside KDCLIENT.DLL library. This fact makes it not that easy to debug this logic. After changing the source code, the original DLL needs to be unloaded from VMWARE-VMX.EXE, new one should be compiled and reloaded. Plus, Edit-And-Continue does not always work correctly in this configuration. To simplify debugging, the so-called Pipe Proxy is provided. In proxy configuration, KDCLIENT.DLL simply resends all data blocks containing marshalled call parameters to a named pipe ("kdvmware_proxypipe") using CallNamedPipe() function. All further processing is performed inside the PIPEPROXY.EXE process (KdRpcDispatcher and KdComDispatcher classes). As the CallNamedPipe() function is used, nothing bad happens if the PIPEPROXY.EXE is terminated. CallNamedPipe() returns an error, KdSendPacket() retries the call from the VM until a new instance of PIPEPROXY.EXE ready to serve requests is started. PIPEPROXY.EXE creates another named pipe compatible with WinDBG and translates low-level requests containing parameters for KdSendPacket()/KdReceivePacket() into KDCOM packets transferred using the second pipe.A typical debugging session with PIPEPROXY.EXE consists of building VMMON.EXE and KDCLIENT.DLL in the "Debug (Proxy mode)" configuration, starting VMMON and then loading PIPEPROXY, debugging it, changing, unloading, recompiling and loading again. No other changes are required. The proxy functionality is implemented in RpcProxy class.