Setting up kernel mode debugger in windows

When ever there is a bug in your program you usually open a debugger(Turbo C++, GDB,visual studio debugger etc) to fix it, but how do you debug a bug in the operating system? Do you load the running OS in to debugger? Is it possible? The simple answer is no. It is not possible because in order for any debugger to work it requires help from the Os on which the program is being debugged. So we cannot debug an OS with the help of itself. So in such scenarios we require 2 machines one is your defective Os(slave) and other machine contains the debugger software(master). Earlier people used to connect slave and master machines using a high speed cable and then once the slave machine is started they used to pause its execution by connecting the debugger to it from the master machine. It used to be the only solution with many drawbacks. 

  1. The connection speed between the machines is too slow, because the data and commands should be passed to and fro between master and slave.
  2. Require extra hardware like cable and two separate machines

Fortunately we now have much better options for beginners who want to study the internals of the OS by debugging. With the help of virtual machines we now donot require two separate machines. The slave machine can be thought of as a one of the guest VMs and master machine can be thought of as a host computer(your real physical machine). The connection between these host and guest have been made even simpler with the help of a software called VirtualKD(Virtual Kernel Debugger)[with out this tool we have to manually set up a named pipe in the guest and modify boot.ini to enable some special options. Its little time consuming]. So in this tutorial I will help you set up kernel mode debugger.


I will be using following tools.

  1. WinDbg (Windows Kernel Debugger)
  2. Virtual Box (Virtual Machine Manager)
  3. VirtualKD (Tool to enable very high speed kernel debugging between host and just machines)

here after when ever I refer to OS it will be one version of windows


First thing we have to do is install virtual box and then install a guest OS of your choice Here I would like to demo you using Windows XP as my guest 
Second extract the virtualKD to some folder 
Rename the VBoxDD.dll file in your VirtualBox program files folder to VBoxDD0.dll.
Copy modified VBoxDD.dll from VirtualKD archive to VirtualBox directory(Ensure that you have selected correct version (x86 or x64) of VBoxDD.dll)

1

Third install WinDbg (simple next next install)
for any debugger to properly work we should have symbol files of the program being debugged, think of these files as extra information about your program which helps debugger in displaying meaningful information to the user. If we have correct symbol files we will have the extract function names and the line numbers etc getting showed in the debugger, without that we will see some hard to understand Hex address. So following few lines will help you configure symbol files for you operating system

If you have internet connection all the time follow this
o   Create an environment variable named _NT_SYMBOL_PATH and set its value to say srv*c:symbols*http://msdl.microsoft.com/download/symbols       
o   Symbols will be downloaded from the microsoft symbol server on demand to c:symbols
Else follow this
o   Create an environment variable named _NT_SYMBOL_PATH and set its value to say c:symbols
o   C:program filessymchk.exe  /r  c:windowssystem32*  /s SRV*C:symbols*http://msdl.microsoft.com/download/symbols
o   The above command will fetch all the symbols for the files in system32 at once (takes time and space)
o   Its a one time task 

when WinDbg is launched it will check this variable to know the path of symbol files.
2

Fourth run VirtualKD in the guest machine and then reboot the guest as instructed

once you are at the boot prompt open 

3



choose the highlighted one and then open vmmon.exe from VirtualKD which will automatically launch WinDBG and connect to the currently running VM and pauses its execution

4_small
5
6

done. the rest is left to your imagination.