Advanced Windows Kernel Debugging Techniques

Kernel debugging is a different animal from the user-mode debugging most developers grow up with. You can’t just attach a debugger to a live kernel the way you’d attach to a misbehaving process, because the kernel is the thing that’s supposed to be managing every process on the box, including the debugger’s own. If it halts, everything halts. So Windows solves this the old-fashioned way: two machines. One is the “target,” the system whose kernel you actually want to inspect. The other is the “host,” running WinDbg, connected over a transport that survives the target being frozen mid-instruction: a network link, a serial cable, or USB. When you set a breakpoint and it hits, the entire target machine stops, not just one process, and control passes to your host over that wire. That’s the mental model to keep in the back of your head for everything that follows: you’re not debugging a program, you’re debugging an operating system from the outside, one machine talking to another.

Setting Up a Kernel Debugging Environment

Configuring the Target Machine

Before any of this works, the target has to be told, at boot time, that it should load a debug-aware boot path and start listening for a debugger. This is stored in the Boot Configuration Data (BCD) store, which is why you configure it with bcdedit rather than some in-OS setting you can flip while Windows is running. The changes only take effect on the next boot, since the debug stub has to be wired in before the kernel itself initializes.

You’ve got three realistic transport choices, and which one you pick is mostly a tradeoff between speed and how much you trust the environment to stay up.

Network debugging (KDNET) is the one you want if your hardware supports it, which most modern NICs do. It’s fast, doesn’t need a special cable, and works great for debugging over Ethernet even between physical machines sitting in different rooms:

bcdedit /debug on
bcdedit /dbgsettings net hostip:192.168.1.100 port:50000 key:1.2.3.4

The key here isn’t a network password in the usual sense, it’s a shared secret WinDbg and the target use to derive a session key, so a random device on the same subnet can’t hijack or snoop your debug session just by guessing the port. Treat it like you would any other credential.

Serial debugging is the grandparent of all this, and it’s still worth knowing because it’s dead simple and doesn’t depend on a functioning network stack, which matters if the very thing you’re debugging is the network stack:

bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200

It’s slow (115200 baud is about as fast as a null-modem cable reliably goes) but it’s rock solid, and it’s a common fallback for debugging inside VMs where the hypervisor exposes a virtual COM port.

USB debugging sits in between: faster than serial, and on hardware with a debug-capable USB controller it doesn’t need a network connection at all:

bcdedit /debug on
bcdedit /dbgsettings usb targetname:WindowsTarget

The targetname is just a friendly label WinDbg uses to find the right target when it enumerates USB debug devices, useful when you’ve got more than one target plugged in.

One thing that trips people up the first time they try to debug an actual driver: if you’re going to load an unsigned or test-signed driver on the target (which you almost always are when developing or reverse engineering one), you’ll also need to enable test signing mode with bcdedit /set testsigning on. Otherwise Windows’ driver signature enforcement will refuse to load it regardless of what the debugger is doing. Once you’ve made any of these BCD changes, restart the target machine so the boot manager picks up the new settings.

Setting Up WinDbg

WinDbg itself ships two ways these days: as part of the Windows SDK (the classic version), or as WinDbg Preview from the Microsoft Store, which is a modern rewrite with better UI, DML (Debugger Markup Language) output, and a JavaScript scripting engine we’ll get to later. Either works for kernel debugging; Preview is generally the nicer experience.

Before you connect to anything, it’s worth setting up a symbol path (.sympath srv*C:\symbols*https://msdl.microsoft.com/download/symbols). Kernel debugging without correct symbols is miserable: you’ll be staring at raw hex addresses instead of function names, and half of what makes !analyze and friends useful is their ability to resolve those addresses against Microsoft’s public symbol server. Get this wrong and every subsequent command in this post produces noise instead of answers.

To actually connect to the target machine:

1. Launch WinDbg as administrator (kernel debugging needs elevated privileges on the host).
2. Navigate to File > Kernel Debug.
3. Select the connection type (NET, COM, or USB) matching whatever you configured with bcdedit.
4. Enter the connection parameters (the same IP/port/key, COM port/baud rate, or USB target name from the previous step).

If the target was configured correctly and is running, you should see the debugger break in and the target machine freeze the instant the connection establishes.

Essential Kernel Debugging Commands

WinDbg’s command set is really two languages layered on top of each other: built-in “dot” commands (.bugcheck, .reload, and so on) that are part of the debugger engine itself, and “bang” commands (!analyze, !pool, !process) that are extension commands implemented in DLLs like kdexts.dll. The extension commands are where most of the kernel-specific power lives, because they understand the layout of internal kernel structures well enough to walk and format them for you instead of making you decode raw memory by hand.

Analyzing Crashes

!analyze -v       # Verbose crash analysis
!analyze -hang    # Analyze a system hang
.bugcheck         # Display bugcheck information
!thread           # Show current thread information
!process 0 0      # List all processes

!analyze -v is usually the first command anyone runs against a crash dump, and it earns that reputation. It reads the bugcheck code and its four parameters, walks the stack at the point of failure, tries to identify the module that owns the faulting code, and cross-references the pattern against a triage database of known failure signatures. It also computes a “bucket ID,” the same kind of fingerprint Windows Error Reporting uses to group crash reports from millions of machines into buckets of “probably the same root cause.” That’s genuinely useful even on a single crash, because a recognizable bucket ID tells you this is a known class of failure rather than something novel.

.bugcheck shows you the raw stop code and its four parameters without any of that analysis. Those four parameters mean different things depending on the bugcheck code itself: for a 0xD1 (DRIVER_IRQL_NOT_LESS_OR_EQUAL), for instance, they’re the memory address referenced, the IRQL at the time of the fault, whether it was a read or write, and (if known) the address of the instruction that made the reference. You end up memorizing the parameter layouts for the bugcheck codes you see most often, because !analyze -v doesn’t always get the culprit right on its own, especially with memory corruption where the crashing stack frame isn’t actually where the bug lives.

!thread dumps the current ETHREAD structure in human-readable form: its state, priority, owning process, and stack. !process 0 0 lists every process on the system; the first 0 means “all processes” (you’d pass a specific EPROCESS address to target one), and the second 0 is a flags bitmask that controls how much detail gets printed per process. Crank that second argument up and you get thread lists, handle counts, and more, at the cost of a much noisier screen.

Examining Memory

!pool             # Examine pool usage
!vm               # Virtual memory statistics
!address <addr>   # Info about a virtual address
db, dw, dd, dq    # Memory display in byte/word/dword/qword

!pool parses the pool header at a given address, the small metadata block the kernel memory manager prepends to every dynamically allocated block, and tells you its size, its allocation type (paged vs. nonpaged), and its four-byte tag. That tag matters a lot in practice, because driver developers are supposed to tag every pool allocation with something identifiable (IoAllocateMdl might use 'lMdI', for example), which turns an otherwise anonymous blob of memory into something you can attribute to a specific subsystem or even a specific driver.

!vm gives you a systemwide memory picture: commit charge against the commit limit, how much of the paged and nonpaged pools are in use, PTE consumption, and so on. It’s the kernel-level equivalent of Task Manager’s performance tab, and it’s usually where you start when a bugcheck says something vague like “the system ran out of pool” (0x19, BAD_POOL_HEADER, or 0x2A, INCONSISTENT_IRP can sometimes trace back here).

!address is the one you reach for during exploit analysis specifically, because it tells you the protection flags on a virtual address range: is it executable, writable, both (which is a red flag by itself, since a writable-and-executable page is exactly what shellcode wants), whether it’s backed by a mapped file, and what VAD (Virtual Address Descriptor) owns it. And then there’s the bread-and-butter memory display family, db/dw/dd/dq for byte, word, dword, and qword granularity respectively. Windows is little-endian, so when you’re reading a dq output and trying to reconstruct a pointer value by eye, remember the bytes are stored least-significant first; it’s a classic source of “why doesn’t this address look right” confusion for people new to raw memory dumps.

Debugging Drivers

!drvobj <driver>  # Driver object information
!devobj <device>  # Device object information
!devstack <addr>  # Device stack
!irp <addr>       # IRP information

To make sense of these four, you need the underlying model: a driver (DRIVER_OBJECT) can create one or more device objects (DEVICE_OBJECT), and those device objects can be layered on top of each other into a “device stack,” where filter drivers sit above or below the driver that actually owns the hardware. A file system minifilter, an antivirus driver hooking disk I/O, and the actual disk driver might all be stacked on the same device. Requests travel down that stack as I/O Request Packets (IRPs), passed from one driver to the next via IoCallDriver, with each driver doing its part and either completing the IRP or passing it further down.

!drvobj and !devobj show you the driver object and device object structures respectively, !devstack shows you the whole layered stack for a given device so you can see exactly which filter drivers are involved, and !irp dumps a specific IRP’s current stack location, showing you which driver in that stack currently owns it. That last one is invaluable for diagnosing hangs: if a system call never returns, walking the IRP with !irp will often show you it’s sitting, uncompleted, in some filter driver three layers down that never called IoCompleteRequest. That’s a completely different failure mode than a crash, and !analyze won’t help you find it, because nothing actually faulted; the system is just waiting forever.

Advanced Debugging Techniques

Kernel Mode Breakpoints

bp nt!PspCreateProcess    # Break on process creation
ba r4 <addr>              # Break on memory read (4 bytes)
bu <module>!<function>    # Deferred breakpoint

bp sets a classic software breakpoint by patching in an int3 instruction at the target address, which means it only works on code that’s actually resident and writable in memory at the moment you set it. nt!PspCreateProcess is a good example of why symbols matter here: it’s an internal, undocumented kernel function, and without private symbols (which most people don’t have) you’re relying on public symbol exports that happen to still expose its name, or on pattern-matching its prologue by hand, which is riskier and breaks across Windows builds.

ba sets a hardware breakpoint instead, backed by one of the CPU’s debug registers (there are four on x86/x64, so you only get four hardware breakpoints at a time, ever). The r4 in ba r4 <addr> means “break on read or write access, 4 bytes wide” (the access type can also be e for execute-only or w for write-only, depending on what you’re hunting). Hardware breakpoints are what you want when you’re chasing memory corruption, because they fire at the exact instruction that touches the address, rather than relying on a software breakpoint at some function entry point that runs long after the corrupting write already happened. If you already know an address is getting clobbered but don’t know by what, a hardware watchpoint is how you catch the culprit red-handed instead of just observing the aftermath.

bu is a deferred, or “unresolved,” breakpoint: you set it on a module!function pair before that module is even loaded, and the debugger quietly resolves it the moment the module shows up. This is exactly what you want when debugging a driver’s initialization: set a bu on its DriverEntry before you ever load it, then trigger the load, and the breakpoint fires the instant the module maps in.

Conditional Breakpoints

bp nt!NtCreateFile "j (poi(@esp+8) & 0x40000000) 'kb'; 'g'"

This one only stops execution when NtCreateFile is called with the FILE_DIRECTORY_FILE flag (0x40000000) set, which is a much more surgical approach than breaking on every single file open on the system and manually filtering by hand. The syntax is worth unpacking because it’s the backbone of a lot of serious kernel debugging: j evaluates a MASM-style expression, and if it’s non-zero, runs the first quoted command string; if it’s zero, runs the second. Here, if the flag bit is set, WinDbg runs kb (print a stack trace with parameters) and stops; otherwise it just runs g (go) and keeps executing without ever bothering you.

One thing worth flagging explicitly: poi(@esp+8) reads an argument off the stack, which is how arguments are passed under the x86 calling convention this example targets. On x64 Windows, the first four integer arguments are passed in registers (@rcx, @rdx, @r8, @r9) rather than on the stack, so the equivalent x64 conditional would inspect one of those registers instead of @esp. It’s an easy mistake to carry an x86-era breakpoint expression onto a 64-bit target and wonder why it never triggers.

Tracing and Logging

!logexts.logopen trace.txt    # Start logging
wt -l3 nt!PspCreateProcess    # Trace function and children up to 3 levels
!logexts.logclose             # Stop logging

!logexts.logopen redirects debugger output to a file so you’ve got a durable record of everything the debugger prints, which matters a lot in kernel debugging sessions that can run for hours. wt (trace and watch) single-steps through a function and everything it calls, building a call tree annotated with instruction counts, and the -l3 flag caps how many levels deep it follows nested calls so the output doesn’t explode into an unreadable wall of text.

Worth knowing before you reach for wt in kernel mode: it’s slow, and it’s slow in a way that can genuinely disrupt what you’re observing, because single-stepping the processor doesn’t pause interrupts. Timer interrupts, DPCs, and other asynchronous kernel activity can fire in the middle of your trace and show up interleaved with the function you actually care about, muddying the call tree with noise that has nothing to do with the code path you’re studying. It’s a great tool for a short, targeted function, and a painful one if you point it at something that runs for any real length of time.

Kernel Pool Exploitation Detection

Detecting Pool Overflows

!poolused 2
!poolfind "EVIL"
!pool <addr>

Every pool allocation in the kernel carries a small header, and part of that header is the four-byte tag mentioned earlier. !poolused 2 walks the entire pool and sums up allocations by tag, sorted by total size, which is exactly what you want when hunting a memory leak or a runaway allocation pattern: a tag you don’t recognize consuming gigabytes of nonpaged pool is a strong lead. !poolfind does something more like a targeted memory search, scanning pool memory for a specific tag pattern ("EVIL" here is a stand-in for whatever tag a suspicious or malicious driver happens to be using) so you can locate every instance of it rather than waiting to trip over one by accident. !pool <addr> is the single-allocation version: point it at an address and it parses that block’s header to tell you its size, tag, whether it’s currently allocated or has already been freed, and what kind of pool it lives in.

All of this is fundamentally reactive: you’re inspecting pool state after something has already gone wrong, or while actively hunting for evidence of tampering. For proactive detection, Windows has Driver Verifier’s special pool option (enabled via gflags or verifier.exe), which allocates each tracked allocation on its own page, bracketed by unmapped guard pages. An overflow that would normally just quietly corrupt an adjacent allocation instead trips a hardware access violation the instant it happens, right at the guilty instruction, rather than corrupting silent state that only manifests as a mysterious crash somewhere else entirely later. It’s slower and uses far more memory, which is exactly why it’s a debugging tool and not something you’d run in production, but it turns “find the bug days later from a corrupted structure” into “catch the bug the moment it happens.”

Detecting Use-After-Free (UAF)

DEVICE_OBJECT *g_DeviceObject = NULL;

VOID DriverUnload(DRIVER_OBJECT *DriverObject) {
    IoDeleteDevice(g_DeviceObject);
    g_DeviceObject = NULL;
}

NTSTATUS DeviceControl(DEVICE_OBJECT *DeviceObject, IRP *Irp) {
    if (g_DeviceObject) {  // Potential UAF
        ULONG deviceType = g_DeviceObject->DeviceType;
    }
    return STATUS_SUCCESS;
}

At first glance this driver looks like it’s actually being careful: it nulls g_DeviceObject right after freeing it, and DeviceControl checks for NULL before touching it. If everything ran on a single thread in strict sequence, that check would be sufficient. The bug is that kernel drivers are inherently multithreaded, and DriverUnload isn’t guaranteed to run in isolation from DeviceControl. Picture the actual race: DriverUnload calls IoDeleteDevice(g_DeviceObject), which frees the memory backing that device object, but hasn’t executed the very next line yet, g_DeviceObject = NULL. On another CPU, a request that’s already in flight reaches DeviceControl, evaluates if (g_DeviceObject), sees a non-NULL pointer (because the assignment to NULL hasn’t happened yet), and dereferences memory that’s already been freed. This is a textbook time-of-check-to-time-of-use (TOCTOU) bug, and it’s a genuinely common pattern in real-world vulnerable drivers, not a contrived example: the fix requires a lock (or an interlocked compare-and-swap) around both the free and the null, not just careful ordering of the two statements.

bp IoDeleteDevice "r $t0 = poi(@esp+4); !devobj @$t0; gc"
bp DriverName!DeviceControl+0x42

The first breakpoint catches every call to IoDeleteDevice system-wide, stashes the device object pointer being freed into the pseudo-register $t0, dumps its state with !devobj, and then continues automatically with gc (go, continuing past the conditional breakpoint without stopping interactively). That gives you a running log of every device deletion on the system, letting you watch for the specific object you care about getting torn down, without having to babysit the debugger by hand at every single deletion. The second breakpoint targets a raw offset, DriverName!DeviceControl+0x42, which is the technique you fall back on when you don’t have source-level symbols for the exact line you want (say, the vulnerable dereference itself) but you’ve disassembled the function and identified the instruction offset by hand.

Manually chasing a race like this with breakpoints works, but it’s genuinely tedious and timing-sensitive, since the whole bug depends on a narrow window between two instructions. This is exactly the kind of thing Driver Verifier’s pool tracking and I/O verification checks are built to catch more reliably: it deliberately fills freed pool with a poison pattern and tags recently-freed allocations, so a subsequent use-after-free reads back an obviously-corrupt value (or trips a guard page under special pool) instead of silently succeeding by reading stale-but-plausible-looking memory that happens not to crash on that particular run.

Kernel Debugging for Security Research

Analyzing Exploit Attempts

!process 0 0      # List all processes
!handle 0 0       # List system handles
!dml_proc         # Enhanced process view (WinDbg Preview)

For exploit analysis, !process 0 0 and !handle 0 0 are your enumeration workhorses: the former lists every process, the latter every open handle system-wide, complete with the object type and access rights each handle grants. Handle enumeration in particular is useful for spotting privilege escalation attempts, since an exploit that’s successfully elevated will often be holding a handle to something it has no business touching, like a process handle to a SYSTEM process with PROCESS_ALL_ACCESS, or a handle to a token object it’s about to duplicate and swap in. !dml_proc is a WinDbg Preview extension that renders the process list with DML (Debugger Markup Language), meaning each entry becomes a clickable link that drills into that process’s detail rather than making you copy an address and re-run !process manually.

Monitoring System Calls

bp nt!NtCreateFile "r $t0 = poi(@esp+8); !pool @$t0; gc"

This is the same conditional-breakpoint pattern from earlier, repurposed as ad hoc telemetry: every time NtCreateFile runs, log some pool state about one of its arguments and keep going. It’s a genuinely handy technique for building a quick, throwaway trace of a specific syscall’s behavior without setting up a full ETW session or reaching for Process Monitor. The catch is that kernel breakpoints stop every CPU on the system each time they fire, even the “non-stopping” conditional ones still pay that cost internally before evaluating the condition, so wiring this up on a syscall as hot as NtCreateFile on a busy production-like system will make the machine feel like it’s wading through molasses. Treat breakpoint-based monitoring as a scalpel for narrow, specific investigations; for broad behavioral monitoring across a live, responsive system, ETW-based tracing or Process Monitor is the better-suited tool, precisely because they’re designed to have low enough overhead to run without freezing the whole box on every hit.

Scripting with WinDbg

"use strict";

function initializeScript() {
    return [new host.apiVersionSupport(1, 3)];
}

function invokeScript() {
    host.namespace.Debugger.Utility.Control.ExecuteCommand(
        "bp nt!PspInsertProcess \"!process @$proc; g\""
    );
    host.diagnostics.debugLog("Process creation monitoring enabled.\n");
}

WinDbg Preview’s JavaScript engine, built on the Debugger Data Model, is the modern answer to the problem of conditional breakpoint expressions getting unreadable once you’re chaining more than one or two commands together. initializeScript is a required entry point that declares which version of the scripting API your script expects, via host.apiVersionSupport, so the debugger can refuse to run it gracefully if it’s too old or too new for what the script needs. invokeScript is where your actual logic lives, and it runs when you load and invoke the script from the debugger.

The interesting bridge here is host.namespace.Debugger.Utility.Control.ExecuteCommand, which lets JavaScript reach back into the classic command-line world and run an ordinary debugger command as a string, in this case setting a breakpoint on nt!PspInsertProcess that prints process info on every hit and continues automatically. That’s the real value of scripting: instead of a one-off command you type once and lose, you get a reusable, version-controllable, shareable tool. Once you’re past the “single conditional breakpoint” stage of a kernel debugging investigation and into “I want to build a repeatable process creation monitor I can hand to a teammate,” JavaScript scripting is where you graduate to.

Kernel debugging rewards patience more than cleverness. Most of what looks like debugger magic in this post, !analyze finding the right bucket, a hardware breakpoint catching a corrupting write at the exact instruction, a conditional breakpoint filtering out noise, is really just the debugger giving you a very precise window into kernel state, and your job is knowing which window to open. Once you’re comfortable with these fundamentals, the natural next step is Time Travel Debugging (TTD), which records an entire execution trace so you can step backward through a crash instead of only forward from a breakpoint, but that’s a big enough topic to deserve its own post.