Using the Windows Debugger:
Exceptions, Bugchecks, and Register Context
September
15, 2003
Jamie E. Hanrahan
Copyright © 2003 by Jamie E. Hanrahan. All rights reserved
This is the first in a series of articles in which I'll provide tutorials and guides for various aspects of the Windows Debugging Tools.
The first thing we're going to talk about is something called register context. You've probably noticed that if you have the debugger connected to your target system when a crash occurs, the debugger often points you directly to the line of code in your driver that caused the problem. But when you open up the resulting memory dump file, the debugger unhelpfully opens a disassembly window showing a portion of something like KiTrap0E or PspUnhandledExceptionInSystemThread.
Briefly, the reason for this is that the debugger is not in the right register context. The debugger has five different contexts you should be aware of, but the register context is probably the most important for kernel mode debugging. When you open a memory dump file the debugger’s register context usually reflects the state of the CPU at the moment when KeBugCheckEx was called. As it turns out, this context is usually not what you want to look at if your goal is to understand the reason for the crash that produced the dump.
To understand why this is so and what you can do about it, we need to learn some vital facts about exceptions, which are the usual causes of crashes, and about how Windows reports these exceptions by forcing the system to crash.
Bugchecks and KeBugCheckEx
Every time you see a “blue screen of death” – or, if you have the system enabled to reboot automatically after a crash (and if so, what kind of kernel developer are you, anyway?), every time you see the system reboot itself spontaneously and then tell you “the system has recovered from a serious error” – someone has called KeBugCheckEx. Calling this routine is often referred to as “declaring a bugcheck.”
The term “bugcheck” has been used by some of the original Windows NT developers since at least their VAX/VMS days at Digital. It’s the same concept as what Unix calls a “kernel panic:” Something in the operating system has decided that an unrecoverable error has occurred and that the best thing to do is to force the system to an immediate halt and subsequent reboot. To put it another way, the system declares a bugcheck when the consequences of the abrupt shutdown, which may include loss of data in open applications, corruption of file system metadata on non-NTFS volumes, etc., are less drastic than those of ignoring the problem.
Bugchecks are declared as the result of unrecoverable errors detected in kernel mode. The Windows NT-family operating systems check for such errors at a perhaps surprisingly small number of points. The interfaces exposed to user mode do perform extensive validity, security, etc., tests on the way to kernel mode. The OS considers everything in kernel mode to be trusted, so the routines exposed only to kernel mode perform relatively little validity checking.
A few kernel mode routines do, however, perform explicit argument validity checks, consistency tests within data structures, and so on. Most driver writers have encountered an example in IoCompleteRequest: If the same instance of the same IRP is passed to IoCompleteRequest more than once (without an IoCompletion routine having returned STATUS_MORE_PROCESSING_REQUIRED in between), the system will crash, and the blue screen will display the moderately informative message MULTIPLE_IRP_COMPLETE_REQUESTS. This happens because, under these circumstances, IoCompleteRequest calls KeBugCheckEx.
There are several factors which, taken together, make this the right thing for IoCompleteRequest to do:
IoCompleteRequest's implementation makes the test practically free. Compared to everything else that happens in IoCompleteRequest, let alone the total life of an IRP, the additional work to perform the test is completely negligible.
IoCompleteRequest has no way of knowing which driver “should have” completed the IRP, and therefore has no way to correct the situation.
If anyone does mistakenly try to complete the same IRP more than once, and IoCompleteRequest did not detect the problem and force the system to crash at that point, several assumptions made by the I/O manager would be broken. And because of that...
...if IoCompleteRequest allowed processing to continue even though the IRP had been completed more than once, a fatal error of some sort would still occur – but at a point very far removed from the code that caused the problem. By declaring the bugcheck as soon as the problem is detected, IoCompleteRequest helps make the problem easier to find. At least that's the theory...
If you aren't convinced yet that forcing the system to crash is the right thing for IoCompleteRequest to do at such a point, we'll have to leave it at that. Forcing a crash is what it does, and we all have to cope.
KeBugCheckEx Arguments
KeBugCheckEx takes five arguments. The BSOD shows these in this form:
*** STOP: 0x00000044 (0xffac9d80, 0x00000d60, 0x00000000, 0x00000000)
The debugger displays them slightly differently:
BugCheck 44, {ffac9d80, d60, 0, 0}
The first argument to KeBugCheckEx – 0x44 in the cases shown above – is always the bugcheck code. (At least it's supposed to be. I've noticed some third party drivers passing an address, apparently a procedure entry point address, as the bugcheck code. Bad driver! Bad!) This is like a status code, a numeric value that indicates what sort of error condition has been detected. The human-readable error message you see on the blue screen, such as MULTIPLE_IRP_COMPLETE_REQUESTS, is simply the name of a #define’d macro that equates to this numeric code.
Most of the #define’s for the bugcheck codes declared by the operating system can be found in the file bugcodes.h, included with the DDK. If you look at this file you’ll find that MULTIPLE_IRP_COMPLETE_REQUESTS has the value 0x44. Therefore, whenever you see a blue screen with the message MULTIPLE_IRP_COMPLETE_REQUESTS, someone has called KeBugCheckEx with a first argument of 0x44.
Microsoft refers to the latter four arguments to KeBugCheckEx as the bugcheck parameters. (Somewhat confusingly, Microsoft refers to KeBugCheckEx's second through fifth arguments as bugcheck parameter 1 through bugcheck parameter 4. Just think of the bugcheck code itself as "bugcheck parameter 0" and you'll be fine.) These are the four values shown in parentheses or braces in the displays above.
The semantics of these parameters vary widely from one bugcheck code to another. Your best and most up to date reference for these is the debugger documentation, under Bug Checks (Blue Screens) | Bug Check Code Reference. For example, if you look up MULTIPLE_IRP_COMPLETE_REQUESTS in the debugger documentation, you’ll find that the first bugcheck parameter (the second argument to KeBugCheckEx) for this bugcheck code is the address of the IRP that was apparently completed twice. In the cases shown above, the IRP address is 0xffac9d80. The other bugcheck parameters are not used for this particular bugcheck code. (The value 0xd60 for the second parameter seems to be a constant in all instances of bugcheck 0x44.)
Obviously, whenever you look at a memory dump file, you should look at this portion of the debugger documentation to help you interpret the bugcheck code and parameters. The debugger will show you these values whenever you open a crash dump file, and you can recall them at any time with the command .bugcheck. The !analyze and !analyze –v commands will also display information about the bugcheck code and arguments; more on this shortly.
An Easy Memory Dump
Let’s take a look at a memory dump file that displayed this particular bugcheck code and see what the debugger can tell us:
Microsoft (R) Windows Debugger Version 6.2.0007.4 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [S:\share\dumps2\multirp.dmp] [...] Use !analyze -v to get detailed debugging information.
BugCheck 44, {ffac9d80, d60, 0, 0}
*** ERROR: Module load completed but symbols could not be loaded for fishbowl.sys Probably caused by : fishbowl.sys ( fishbowl+b725 )
Followup: MachineOwner --------- kd>
(We should note before proceeding that the driver mentioned here, "fishbowl.sys," was deliberately "bugged" to show this problem; we aren't showing a real bug in anybody's product here. We will do that later, however!)
The line that we colored in blue above shows the bugcheck code and parameters. We know from the debugger help that bugcheck code 0x44 is the error we're discussing here – someone tried to complete the same IRP twice, and the address of that IRP must be 0xffac9d80.
One of the first things we often do when looking at a crash dump is to inspect the stack. The debugger has many variations of the k (display stack) command; kv (display stack verbose) is my personal default. kv at this time produces the following:
kd> kv ChildEBP RetAddr Args to Child f897596c 805135f6 00000044 ffac9d80 00000d60 nt!KeBugCheckEx+0x19 (FPO: [Non-Fpo]) f89759a4 f1b29725 ffac9e80 81b94030 ffac9e38 nt!IopfCompleteRequest+0x2c4 (FPO: [Non-Fpo]) WARNING: Stack unwind information not available. Following frames may be wrong. f89759e4 f1b207ef ffa70838 ffac9d80 00000000 fishbowl+0xb725 f8975a1c f1b216e1 ffa70838 ffac9d80 ffa708f0 fishbowl+0x27ef f8975a3c 804ec04f ffa70838 ffac9d80 f8975ab8 fishbowl+0x36e1 f8975a4c 8055b50d f8975ab8 ffafdc88 00000000 nt!IopfCallDriver+0x31 (FPO: [0,0,1]) f8975a78 805ae37f ffa70838 f8975a94 00000000 nt!IopSynchronousCall+0xb8 (FPO: [Non-Fpo]) f8975ab8 805054d6 ffafdc88 00000001 c00002ce nt!IopStartDevice+0x4a (FPO: [Non-Fpo]) f8975ad4 805ae1e6 ffafdc88 ffafdc01 ffab4da8 nt!PipProcessStartPhase1+0x4c (FPO: [Non-Fpo]) f8975d1c 805f9be6 81b49de8 00000001 00000000 nt!PipProcessDevNodeTree+0x171 (FPO: [Non-Fpo]) f8975d4c 805142c6 00000003 80543d40 80548abc nt!PiRestartDevice+0x7e (FPO: [Non-Fpo]) f8975d74 804ebd08 00000000 00000000 81fccda8 nt!PipDeviceActionWorker+0x150 (FPO: [Non-Fpo]) f8975dac 80559026 00000000 00000000 00000000 nt!ExpWorkerThread+0xfe (FPO: [Non-Fpo]) f8975ddc 8050f513 804ebc35 00000001 00000000 nt!PspSystemThreadStartup+0x34 (FPO: [Non-Fpo]) 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16 kd>
The debugger also suggested that we should use the command !analyze -v for more information. If we try that, the resulting output will look like this:
kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* MULTIPLE_IRP_COMPLETE_REQUESTS (44) A driver has requested that an IRP be completed (IoCompleteRequest()), but (1) the packet has already been completed. This is a tough bug to find because the easiest case, a driver actually attempted to complete its own packet twice, is generally not what happened. Rather, two separate drivers each believe that they own the packet, and each attempts to complete it. The first actually works, and the second fails. Tracking down which drivers in the system actually did this is difficult, generally because the trails of the first driver have been covered by the second. However, the driver stack for the current request can be found by examining the DeviceObject fields in each of the stack locations. Arguments: Arg1: ffac9d80, Address of the IRP (2) Arg2: 00000d60 Arg3: 00000000 Arg4: 00000000 Debugging Details: ------------------ Database SolnDb not connected IRP_ADDRESS: ffac9d80 DEFAULT_BUCKET_ID: DRIVER_FAULT BUGCHECK_STR: 0x44 LAST_CONTROL_TRANSFER: from 805135f6 to 804fc1bb STACK_TEXT: f897596c 805135f6 00000044 ffac9d80 00000d60 nt!KeBugCheckEx+0x19 (3) f89759a4 f1b29725 ffac9e80 81b94030 ffac9e38 nt!IopfCompleteRequest+0x2c4 WARNING: Stack unwind information not available. Following frames may be wrong. f89759e4 f1b207ef ffa70838 ffac9d80 00000000 fishbowl+0xb725 f8975a1c f1b216e1 ffa70838 ffac9d80 ffa708f0 fishbowl+0x27ef f8975a3c 804ec04f ffa70838 ffac9d80 f8975ab8 fishbowl+0x36e1 f8975a4c 8055b50d f8975ab8 ffafdc88 00000000 nt!IopfCallDriver+0x31 f8975a78 805ae37f ffa70838 f8975a94 00000000 nt!IopSynchronousCall+0xb8 f8975ab8 805054d6 ffafdc88 00000001 c00002ce nt!IopStartDevice+0x4a f8975ad4 805ae1e6 ffafdc88 ffafdc01 ffab4da8 nt!PipProcessStartPhase1+0x4c f8975d1c 805f9be6 81b49de8 00000001 00000000 nt!PipProcessDevNodeTree+0x171 f8975d4c 805142c6 00000003 80543d40 80548abc nt!PiRestartDevice+0x7e f8975d74 804ebd08 00000000 00000000 81fccda8 nt!PipDeviceActionWorker+0x150 f8975dac 80559026 00000000 00000000 00000000 nt!ExpWorkerThread+0xfe f8975ddc 8050f513 804ebc35 00000001 00000000 nt!PspSystemThreadStartup+0x34 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16 FOLLOWUP_IP: fishbowl+b725 f1b29725 a1f0c9b2f1 mov eax,[fishbowl+0xe9f0 (f1b2c9f0)] FOLLOWUP_NAME: MachineOwner SYMBOL_NAME: fishbowl+b725 MODULE_NAME: fishbowl IMAGE_NAME: fishbowl.sys DEBUG_FLR_IMAGE_TIMESTAMP: 3cb95807 STACK_COMMAND: kb (4) BUCKET_ID: 0x44_fishbowl+b725 Followup: MachineOwner --------- kd>
I've annotated the most important things in this output with callouts (1) in the right margin. These are, in order:
Except for the missing headings and FPO information (something we'll talk about in a later column), the stack display in the !analyze -v output looks exactly like what we obtained with the kv command. And the stack leads us directly back to the driver that caused the problem – at least, one of the two drivers. It appears that the "fishbowl.sys" driver called IopfCompleteRequest, which in turn detected the "multiple IRP complete requests" condition and called KeBugCheckEx.
A Less Easy Memory Dump
We're in the midst of this article because, for most crashes, it just isn't this easy. Here's a typical example:
kd> kv ChildEBP RetAddr Args to Child f08535f4 8045261e f085361c 8045cd0b f0853624 nt!PspUnhandledExceptionInSystemThread+0x18 (FPO: [1,0,0]) f0853ddc 80467122 80416bf2 00000001 00000000 nt!PspSystemThreadStartup+0x5e (FPO: [Non-Fpo]) 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16 kd>
We must tell you that there is a driver on the stack here – we just can't see it. This is one of those cases where opening the dump file with the debugger will not automatically pop open a source window with the "current line" pointing to the offending line. Even if the crash was in your driver and you do have your symbols and source path set correctly.
Let's see what !analyze -v has to say about this:
0: kd> !analyze -v [...] KMODE_EXCEPTION_NOT_HANDLED (1e) [...] Arguments: Arg1: c0000005, The exception code that was not handled Arg2: f07a973b, The address that the exception occurred at Arg3: 00000000, Parameter 0 of the exception Arg4: 0000000c, Parameter 1 of the exception [...] EXCEPTION_RECORD: f0853aa4 -- (.exr fffffffff0853aa4) ExceptionAddress: f07a973b (vicamusb+0x0000173b) ExceptionCode: c0000005 ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 0000000c Attempt to read from address 0000000c [...] STACK_TEXT: 00000000 00000000 00000000 00000000 00000000 vicamusb+0x173b [...] STACK_COMMAND: .cxr fffffffff08536fc ; kb [...] 0: kd>
The "STACK_TEXT" in this case doesn't look like a typical stack, but at least it references a driver. (As in the preceding case, this driver was deliberately "bugged" to cause this crash.) We note also that the "STACK_COMMAND" is more complex than it was last time; for this dump, it seems to take two commands to display the stack.
Now if you wanted to know just enough about the debugger to get by, we could at this point tell you:
and send you on your way. Or we could say
I suspect, though, that you want to know a little more about it than that.
Exceptions
The reason the debugger could not display a meaningful stack right away in the second example is this: The original cause of the system crash was an exception.
Bugchecks can be broadly and helpfully divided into two types: Those triggered by in-line tests, and those caused by unhandled exceptions in kernel mode. The bugcheck code we saw in our first example, MULTIPLE_IRP_COMPLETE_REQUESTS, is an example of the former: IoCompleteRequest detected – via an explicit in-line test – that an error condition had occurred, and called KeBugCheckEx. This is much like an "assert" from user mode, and we will refer to these as "assertion" bugchecks.
Our second case, KMODE_EXCEPTION_NOT_HANDLED, is obviously an example of the latter.
The debugger documentation on the bugcheck codes does not make a distinction between "exception" and "assert" bugchecks. And that's a shame, because recognizing which of these types of bugchecks you're looking at is vital to analyzing any crash. It turns out, by the way, that while most bugcheck codes that you'll find in the documentation are of the "assert" variety, "exception" bugchecks are by far the more common in terms of actual occurrence rates.
Exceptions are processor conditions raised as a side effect of executing instructions, usually to report problems of one sort or another. A very familiar and obvious example (but almost irrelevant to kernel mode debugging) is dividing by zero. If you try it, the DIV instruction will raise an exception. There is no possible numeric result that the instruction can return that correctly represents the "undefined" result of division by zero, so the instruction has to have some other way of reporting the attempt. It does so by raising an exception.
Now drivers seldom do any division at all, let alone by zero, so we won't often see divide by zero errors in kernel debugging. But we do very commonly see memory access violations, that is, attempts to reference undefined or inaccessible virtual addresses. Dereferencing a null pointer is a very frequent case of this. A page fault – an attempt to access a virtual address that is currently not "paged in" – is an exception as well, but one that the pager usually takes care of and dismisses.
Exceptions are like interrupts in the following way: When either an exception or an interrupt occurs, the processor is forced to suspend what it's doing and go off and execute code pointed to by a table in memory. In the x86 architecture this table is called the interrupt dispatch table, or IDT. Despite its name, it contains entries for both exceptions and interrupts.
Just before transferring control to wherever the IDT entry tells it to, the processor will push onto the stack the address of the next instruction "in line" – the next instruction to be executed had the exception not occurred. Well, to be precise, for some exceptions (faults) it pushes the address of the instruction that raised the exception; for others (traps) it's the address of the next instruction after that, but that detail isn't important now. In the case of a fault, this permits the processor to re-execute the faulting instruction in the case where the exception handling code fixes things up; page faults are a very common example. For some exceptions the processor also pushes other information about the exception before transferring control. If you want to know all the ugly details, they're of course in the Intel processor architecture references, specifically Chapter 5 of the IA-32 Intel® Architecture Software Developer's Manual, Volume 3: System Programming Guide.
Exceptions are like interrupts in another way: The processor can resume execution of the original code after, and if, the exception is handled. An exception handler is a particular type of procedure written for this purpose. For some exceptions, notably page faults, the operating system provides the exception handler.
Exception Processing in Kernel Mode
Whenever an exception occurs, the operating system gets control, in kernel mode, at whatever routine the IDT indicates for the exception. On the x86, each different exception has a different entry in the IDT, and therefore a different routine can be invoked for each. Most of the entries for exceptions in the IDT point to routines with names like KiTrap03, KiTrap04, etc.; these together comprise the kernel trap handler, which performs a sort of first-level triage for exceptions. One factor in these decisions is the original processor access mode: The kernel trap handler treats exceptions raised from user mode differently from those raised from kernel mode. We aren't particularly interested in exceptions raised from user mode; they won't cause system crashes. They are the app developer's problem.
For exceptions raised from kernel mode, the trap handler sorts the exception into one of three categories:
The first category isn't interesting to us; we'll never see them. In the second case, the trap handler calls the exception dispatcher, and we'll cover that scenario in just a moment.
Bugchecks Declared by the Trap Handler
In the third case – an exception that can't be handled at all – the kernel trap handler itself calls KeBugCheckEx. Since kernel mode code is supposed to be trusted, and exceptions indicate problems, an exception raised from kernel mode that cannot be handled is always considered fatal to the system. One common example is a memory access violation or page fault at IRQL greater than or equal to DISPATCH_LEVEL (IRQL 2). In such a case, the kernel trap handler calls KeBugCheckEx with the bugcheck code IRQL_NOT_LESS_OR_EQUAL.
Let's take a look at a stack trace from a crash of this type, displayed without any help or hints from !analyze -v. Please note that this is a different memory dump file than the one we were looking at a moment ago; we'll get back to that one in the next section.
kd> kv ChildEBP RetAddr Args to Child f645e62c fcc494f6 80465402 00000030 00000030 nt!KiTrap0E+0x27c (FPO: [0,0] TrapFrame @ f645e62c) f645e69c f7095181 004c0407 f645e74c f645e748 NDIS!NdisQueryBufferOffset+0x8 (FPO: [3,0,0]) f645e770 f70950c7 81381008 813090b8 813b0800 el90xbc5!SendPacket+0x61 (FPO: [Non-Fpo]) f645e794 fcc604d9 81381008 f645e7b4 00000001 el90xbc5!NICSendPackets+0xa5 (FPO: [Non-Fpo]) f645e808 fcc536a0 813090b8 813b0890 813090e0 NDIS!ndisMStartSendPackets+0x1fb (FPO: [Non-Fpo]) f645e82c fcc60643 812d1e48 813090b8 00000000 NDIS!ndisMProcessDeferred+0x37 (FPO: [Non-Fpo]) [...]
What happened here? Code in the routine NdisQueryBufferOffset incurred an exception – specifically, a memory access violation or page fault. (How would you know that? By looking at the stack after applying the techniques we're about to describe! Of course, as with most of our examples, the driver and routines that appear on the stack here aren't actually at fault; the system was deliberately "bugged" to cause this problem.) This caused the processor to vector off to KiTrap0E, part of the kernel trap handler. This routine determined that the memory access violation occurred at IRQL DISPATCH_LEVEL or above, and so called KeBugCheckEx.
The stack doesn't show a call to KeBugCheckEx, by the way, because the stack only shows the current instruction pointer value and calls from routines, not calls into routines... and in this stack, KeBugCheckEx hasn't called anything. If we really want to see the call to KeBugCheckEx within KiTrap0E, we can ask the debugger to unassemble (u) within KiTrap0E, starting a few bytes before the current instruction pointer value shown:
kd> u nt!KiTrap0E+0x27c-e nt!KiTrap0E+26f: 80464b12 7568 jnz nt!KiTrap0E+0x2d9 (80464b7c) 80464b14 56 push esi 80464b15 51 push ecx 80464b16 50 push eax 80464b17 57 push edi 80464b18 6a0a push 0xa 80464b1a e8f573fcff call nt!KeBugCheckEx (8042bf14) 80464b1f f7457000000200 test dword ptr [ebp+0x70],0x20000 kd>
(How did we know to look 0xE bytes in memory before KiTrap0E+0x27c? We didn't at first; we had to poke around a bit to find that. Or look at the disassembly window. We'll cover those techniques in a later article.)
Bugchecks Declared by the Exception Dispatcher
For Case 2 – exceptions that an exception handler may be able to handle – things are more complicated. The kernel trap handler calls an inner level set of procedures, common to all the KiTrap0x entry points, collectively called the exception dispatcher. The exception dispatcher's job is to go looking for declared exception handlers and call them. If it finds none, or if all that are found return EXCEPTION_CONTINUE_SEARCH, then the exception hasn't been handled, and the exception dispatcher calls KeBugCheckEx. Again, any unhandled exception raised from kernel mode is fatal to the operating system. In this case, though, it's the exception dispatcher rather than the kernel trap handler that makes the call.
Here's a stack (from yet another different memory dump file) showing this case:
kd> kv
ChildEBP RetAddr Args to Child
f71efac8 804624cb f71efae4 00000000 f71efb38 nt!KiDispatchException+0x30e (FPO:
[Non-Fpo])
f71efb30 8046247d 81288784 f71efb84 804b0b48 nt!CommonDispatchException+0x4d (FPO:
[0,20,0])
f71efb30 804676c8 81288784 f71efb84 804b0b48 nt!KiUnexpectedInterruptTail+0x1f4
(FPO: [0,0] TrapFrame @ f71efb38)
f71efbd0 804672a2 00000001 00000000 8049b588 nt!ExFreePoolWithTag+0x342 (FPO:
[Non-Fpo])
f71efbdc 8049b588 e1e40ea8 e1bbb80c 80495b70 nt!ExFreePool+0xb (FPO: [1,0,0])
f71efbe8 80495b70 e1bbb810 e1bbb7f8 81437740 nt!SepTokenDeleteMethod+0x1b (FPO:
[1,0,1])
f71efc04 8044c3b3 e1bbb810 e1bbb810 81272020 nt!ObpRemoveObjectRoutine+0xd6 (FPO:
[Non-Fpo])
[...]
ExFreePoolWithTag raised an exception in a context in which exceptions can theoretically be handled. So the kernel trap handler called the exception dispatcher, as
evidenced by the calls on the stack to CommonDispatchException and
KiDispatchException. However, KiDispatchException found no exception handlers that wanted to handle this problem, and
so called KeBugCheckEx – this time with the bugcheck code
KMODE_EXCEPTION_NOT_HANDLED.
Exceptions and Debugger Register Context
Now we can get to the crux of the matter: the reason the debugger sometimes needs help to display a meaningful stack and register contents from a memory dump file.
When the debugger first opens the crash dump file, its register context reflects that of the processor as if KeBugCheckEx had returned to caller. (The bugcheck mechanism actually does a little bit of fudging when writing the processor state to the dump file to support this; otherwise, the current instruction pointer in a crash dump would always be within KeBugCheckEx, or sometimes an inner routine it calls, KeBugCheckEx2.) For example, if we had source code, private symbols, etc., for the operating system version that produced the stack shown above, the debugger would pop open a source window and could show us the local variables within KiDispatchException.
But we don't want to debug KiDispatchException. We want to debug the routine that raised the exception in the first place. To do this we have to set the debugger's register context to what it was at the time of the exception. Since the instruction pointer (eip) register is part of the register context, the debugger will then show us the actual instruction that raised the exception. Since the stack pointer (esp) and other general purpose registers are part of the register context too, we'll probably be able to see why the instruction raised the exception.
Setting the debugger's register context is usually relatively easy to do, because the memory dump usually includes all of the necessary information. For some exception bugcheck codes, the debugger help will tell you how to do it; unfortunately, they omitted the description for a few others. And of course for "inline" or "assert" bugchecks there is nothing to do at all in the way of setting register context, but you do have to recognize that it is an assert bugcheck and that there's therefore no context to set.
What you are looking for on the stack is evidence of a trap frame, context records, or task state segment in which the system recorded its state just prior to the exception being raised. Here's how to do this.
1) Look for a pointer to a context record
If the exception dispatcher is going to look for an exception handler, it must first build two data structures that the exception handlers (if any) will need. These are called a context record and an exception record. To find these structures, see if the routine name PspUnhandledExceptionInSystemThread is at the top of the stack. We had an example of a stack showing this routine earlier. Here it is again:
kd> kv ChildEBP RetAddr Args to Child f08535f4 8045261e f085361c 8045cd0b f0853624 nt!PspUnhandledExceptionInSystemThread+0x18 (FPO: [1,0,0]) f0853ddc 80467122 80416bf2 00000001 00000000 nt!PspSystemThreadStartup+0x5e (FPO: [Non-Fpo]) 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16 kd>
If the first thing on the stack shows a call to this routine name, we can use the first argument passed to the routine to find the exception and context records in memory. The first argument to any routine shown on the stack (unless it's using the fastcall interface, which this routine never does) is in the first column of DWORDs under the "Args to child" heading in the stack trace. We've highlighted this in blue above.
This argument contains a pointer to two successive DWORDs in memory, which in turn are pointers to the exception and context record, respectively. We can use those pointers as arguments to the debugger's .exr and .cxr commands. We can do this on any memory dump where we see PspUnhandledExceptionInSystemThread at the top of the stack. Let's try it:
kd> dd f085361c L 2 f085361c f0853aa4 f08536fc
We used L 2 (length: 2 items) on the dd (display DWORDs)
command to keep the resulting output to a manageable size. We supply the DWORD
in memory at the indicated address to the debugger's .exr (display
exception record) command:
kd> .exr f0853aa4
ExceptionAddress: f07a973b (vicamusb+0x0000173b)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 0000000c
Attempt to read from address 0000000c
kd>
The debugger conveniently interprets the exception record, describing the
type of exception and the meanings of the exception parameters. Then we supply
the second DWORD to the debugger's .cxr (set context to exception record)
command:
kd> .cxr f08536fc
eax=00000000 ebx=802ac6c8 ecx=802ac610 edx=00000000 esi=802ac6c8 edi=802ac610
eip=f07a973b esp=f0853b6c ebp=00000000 iopl=0
nv up ei pl zr na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00210246
vicamusb+173b:
f07a973b 39450c
cmp
[ebp+0xc],eax ss:0010:0000000c=????????
kd>
In response to the .cxr command the debugger shows us the contents of the x86's general purpose registers at the time of the exception. One of these is the instruction pointer (eip), and the debugger interprets the instruction – a cmp (compare) instruction – that it finds at that address. It even tells us that the instruction was trying to access address 0xc, the same value reported from the exception record.
Furthermore, since the debugger uses the ebp register to begin walking the stack, and we now have a different value in ebp, we should now be able to get a more meaningful stack trace! Let's see:
0: kd> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. 00000000 00000000 00000000 00000000 00000000 vicamusb+0x173b
Hmm, that doesn't look very meaningful, does it? Actually, it is, but we'll have to put off analyzing this sort of stack display for a later article. The point for now is that we've gotten the debugger to show us the instruction that raised the exception, not the (much later) point where the bugcheck was declared as the result of that exception. We can use the u (unassemble) command, or the disassembly window, to look at the code "around" this instruction. If we have the source code and private symbols for vicamusb.sys, the debugger would have popped open a source window and so on.
2) Look for a trap frame address passed to KiDispatchException
If you see a call to KiDispatchException at the top of the stack, its third argument will be the address of a trap frame, and we can use this address with the debugger's .trap (Display Trap Frame) command to set the register context. The kernel trap handler builds the trap frame on the stack to capture the state of the processor at the time of the exception. If nobody handles an exception raised from kernel mode, the kernel trap handler or exception dispatcher will call KeBugCheckEx, and the resulting memory dump will initially show an "unhelpful" stacks, with the offending code not at the top. But if we can find that trap frame, we can set the debugger's register context to reflect the CPU state at the time of the exception.
Here's an example of a stack showing a call to KiDispatchException:
kd> kv
ChildEBP RetAddr Args to Child
f71efac8 804624cb f71efae4 00000000 f71efb38
nt!KiDispatchException+0x30e (FPO: [Non-Fpo])
f71efb30 8046247d 81288784 f71efb84 804b0b48 nt!CommonDispatchException+0x4d (FPO:
[0,20,0])
f71efb30 804676c8 81288784 f71efb84 804b0b48 nt!KiUnexpectedInterruptTail+0x1f4
(FPO: [0,0] TrapFrame @ f71efb38)
f71efbd0 804672a2 00000001 00000000 8049b588 nt!ExFreePoolWithTag+0x342 (FPO:
[Non-Fpo])
f71efbdc 8049b588 e1e40ea8 e1bbb80c 80495b70 nt!ExFreePool+0xb (FPO: [1,0,0])
f71efbe8 80495b70 e1bbb810 e1bbb7f8 81437740 nt!SepTokenDeleteMethod+0x1b (FPO:
[1,0,1])
f71efc04 8044c3b3 e1bbb810 e1bbb810 81272020 nt!ObpRemoveObjectRoutine+0xd6 (FPO:
[Non-Fpo])
[...]
The third argument passed to KiDispatchException (highlighted in blue) has the value 0xf71efb38. We use the .trap command on this address:
kd> .trap f71efb38 ErrCode = 00000002 eax=e1e40fc0 ebx=00000000 ecx=00000000 edx=00000000 esi=e1e40ea0 edi=81438428 eip=804676c8 esp=f71efbac ebp=f71efbd0 iopl=0 nv up ei pl zr na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246 nt!ExFreePoolWithTag+342: 804676c8 890a mov [edx],ecx ds:0023:00000000=???????? kd>
Just as with the .cxr command, this command tells the debugger to assume the register state reflected in the trap frame supplied. It displays the general purpose registers, tells us that the instruction pointed to by the instruction pointer (eip) in the trap frame was a mov (move) instruction, and interprets the address referenced by this instruction – zero. Furthermore, the ebp register now points to the correct place to allow the debugger's stack trace to display a more meaningful stack:
kd> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child f71efbd0 804672a2 00000001 00000000 8049b588 nt!ExFreePoolWithTag+0x342 (FPO: [Non-Fpo]) f71efbdc 8049b588 e1e40ea8 e1bbb80c 80495b70 nt!ExFreePool+0xb (FPO: [1,0,0]) f71efbe8 80495b70 e1bbb810 e1bbb7f8 81437740 nt!SepTokenDeleteMethod+0x1b (FPO: [1,0,1]) f71efc04 8044c3b3 e1bbb810 e1bbb810 81272020 nt!ObpRemoveObjectRoutine+0xd6 (FPO: [Non-Fpo]) f71efc28 80451ac4 00000002 81272020 f71efd3c nt!ObfDereferenceObject+0x149 (FPO: [Non-Fpo]) f71efc40 80451e5d 81272020 00000000 00000000 nt!PsImpersonateClient+0x191 (FPO: [Non-Fpo]) [...] kd>
The stack doesn't appear to have changed much, but the routine that raised the exception is now at the top of the displayed stack, and the rest of the debugger's register context also reflects this location in the code. The debugger can now show us correct source code, local variables, etc., for the failing routine, provided of course we have source and private symbols for that routine.
3) Look for a Trap Frame Address
The trap frame was easy to spot in the previous example. What if the kernel trap handler doesn't bother calling KiDispatchException? This will be true for exceptions that are not considered handleable at all. In that case, the trap handler will simply call KeBugCheckEx directly.
Fortunately, the debugger knows where to find the trap frame in this case, and tells us rather clearly. Here's an example:
kd> kv ChildEBP RetAddr Args to Child f645e62c fcc494f6 80465402 00000030 00000030 nt!KiTrap0E+0x27c (FPO: [0,0] TrapFrame @ f645e62c) f645e69c f7095181 004c0407 f645e74c f645e748 NDIS!NdisQueryBufferOffset+0x8 (FPO: [3,0,0]) f645e770 f70950c7 81381008 813090b8 813b0800 el90xbc5!SendPacket+0x61 (FPO: [Non-Fpo]) f645e794 fcc604d9 81381008 f645e7b4 00000001 el90xbc5!NICSendPackets+0xa5 (FPO: [Non-Fpo]) f645e808 fcc536a0 813090b8 813b0890 813090e0 NDIS!ndisMStartSendPackets+0x1fb (FPO: [Non-Fpo]) f645e82c fcc60643 812d1e48 813090b8 00000000 NDIS!ndisMProcessDeferred+0x37 (FPO: [Non-Fpo]) [...]
Just as before, we supply that address to the debugger's .trap command:
kd> .trap f645e62c ErrCode = 00000000 eax=004c0407 ebx=8128b7bc ecx=8128b7bc edx=f645e74c esi=813090b8 edi=004c0407 eip=fcc494f6 esp=f645e6a0 ebp=f645e770 iopl=0 nv up ei pl nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202 NDIS!NdisQueryBufferOffset+8: fcc494f6 8b4818 mov ecx,[eax+0x18] kd>
Then we can display the stack:
kd> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child f645e69c f7095181 004c0407 f645e74c f645e748 NDIS!NdisQueryBufferOffset+0x8 (FPO: [3,0,0]) f645e770 f70950c7 81381008 813090b8 813b0800 el90xbc5!SendPacket+0x61 (FPO: [Non-Fpo]) f645e794 fcc604d9 81381008 f645e7b4 00000001 el90xbc5!NICSendPackets+0xa5 (FPO: [Non-Fpo]) f645e808 fcc536a0 813090b8 813b0890 813090e0 NDIS!ndisMStartSendPackets+0x1fb (FPO: [Non-Fpo]) f645e82c fcc60643 812d1e48 813090b8 00000000 NDIS!ndisMProcessDeferred+0x37 (FPO: [Non-Fpo]) f645e848 f6ec1218 812d1c68 81309001 812d1e48 NDIS!ndisMSend+0x20b (FPO: [Non-Fpo])
Just as before, the stack hasn't changed much – we've just pared off the trap handler. Again, what is important is that the debugger's register state (and so the source code window, Locals window, and so on) will reflect the point of the exception, rather than the much later point where the system declared the bugcheck.
4) Look for a Task State Segment
The above three cases cover most of the exception bugchecks you'll ever see, but there's one more possibility. It's relatively rare unless you are experiencing a particular type of problem, in which case it will happen at least fairly reliably – if the word "reliably" makes any sense here!
The only time I've ever seen this case is when the CPU raises an exception known as a double fault. Older versions of the debugger documentation suggested that double faults indicated hardware problems; and it is true that if you are having certain types of hardware problems, you may see some double faults. However, double faults can have a purely software origin, one that is not terribly uncommon... particularly if certain guidelines for driver coding are ignored. I'm happy to report that Microsoft has updated the debugger documentation to reflect this.
The best way to describe this case is to start by looking at one. Here's a stack trace:
0: kd> kv ChildEBP RetAddr Args to Child 00000000 bfb2b7f4 00000000 00000000 00000000 nt!KiTrap08+0x41 (FPO: TaskGate 28:0) WARNING: Stack unwind information not available. Following frames may be wrong. f24391c0 bfda8aa8 00000001 81fa85e0 00000d12 xyzwdm+0x707f4 f24391e0 bfda8af5 00000001 81fa85e0 00000d12 xyzspud+0xeaa8 f243920c bfda09ba 00000001 81fa85e0 00000d12 xyzspud+0xeaf5 f243924c bfd9c06a 81f44be0 bfdb1951 81fa85e0 xyzspud+0x69ba f243929c bfb51f02 00000000 00000000 00000001 xyzspud+0x206a f24392ec bfb2cd9f 81176748 f2439344 e14aa508 xyzwdm+0x96f02 f2439304 bfb2ab10 00000000 00000000 00000000 xyzwdm+0x71d9f [...]
A double fault occurs when the processor, while trying to record the instruction pointer value and other information about one exception, encounters a second exception. An exception raised in the exception handling path can usually be dealt with serially, but not always. If not, the CPU raises a special "double fault" exception to report the fact that it couldn't report the original exception.
As with all other exception types, the "double fault" has its own entry in the IDT. Unlike most, however, this IDT entry specifies that the exception is to be handled in a new hardware task. In other words, the IDT entry specifies a task state segment, or TSS, other than the one under which most of the operating system runs.
The "task state segment" (TSS) is a data structure that provides new values for all the general purpose and some of the privileged registers in the CPU. An exception or interrupt through an entry in the IDT that specifies a new TSS will result in all of these registers suddenly acquiring new values, as reflected in the members of the TSS structure. The processor's previous state is recorded in the TSS that was previously in effect, so that execution of the previous "hardware task" can be resumed if possible.
Detailed discussion of hardware tasks, how x86 uses them, and how the Windows NT operating system family mostly ignores them, is far out of the scope of this article. Briefly, however, changing to a new hardware task is much like a thread context switch. In fact, task state switches could be used by an operating system to implement thread context switching, although Windows does not use them for this purpose, running almost all of the time in one hardware task.
The double fault exception has its own task state segment under Windows simply because it's assumed to be necessary. If a double fault occurs, nothing about the previous hardware state, not even things like the stack, can be assumed to be usable. So early on in the boot, Windows initializes a special hardware task – with its own stack, independent of the normal threads' kernel mode stacks – just for recording the evidence left behind by double faults.
When a double fault occurs, the CPU changes its state to that reflected in the IDT entry for the double fault exception, and records its previous state – the state at the time the original exception occurred – in the task state segment that was in use at the time. Task state segments are identified by small numbers rather than addresses, and the notation on the stack, "TaskGate 28:0", tells us that the processor state we want to look at was recorded in TSS 28. We supply this number to the debugger's .tss (Display Task State Segment) command:
0: kd> .tss 28 eax=bfb2b7eb ebx=81fafd80 ecx=81f44be0 edx=00000000 esi=81f44be0 edi=fe0c02c0 eip=bfb2b7f4 esp=f2438ed8 ebp=f24391c0 iopl=0 nv up ei ng nz ac po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00210296 xyzwdm+707f4: bfb2b7f4 57 push edi
This looks familiar; it's the same sort of display we get after a .trap or .cxr command. Apparently the push edi instruction raised an exception. About the only way a push from a register can raise an exception is if the stack itself is somehow inaccessible.
As in the previous cases, we have a new value for the ebp register, so the stack display should have changed; and it has, a little:
0: kd> kv ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. f24391c0 bfda8aa8 00000001 81fa85e0 00000d12 xyzwdm+0x707f4 f24391e0 bfda8af5 00000001 81fa85e0 00000d12 xyzspud+0xeaa8 f243920c bfda09ba 00000001 81fa85e0 00000d12 xyzspud+0xeaf5 f243924c bfd9c06a 81f44be0 bfdb1951 81fa85e0 xyzspud+0x69ba f243929c bfb51f02 00000000 00000000 00000001 xyzspud+0x206a f24392ec bfb2cd9f 81176748 f2439344 e14aa508 xyzwdm+0x96f02 f2439304 bfb2ab10 00000000 00000000 00000000 xyzwdm+0x71d9f [...]
Nothing looks very wrong here. Recall, however, that a stack trace begins with the address found in the ebp register. A push instruction, by contrast, will use the address in the esp (stack pointer) register, and we see from the register display above that esp has a somewhat smaller value than ebp. Let's see what the debugger tells us about the memory pointed to by esp:
0: kd> dd esp f2438ed8 ???????? ???????? ???????? ???????? f2438ee8 ???????? ???????? ???????? ???????? f2438ef8 ???????? ???????? ???????? ???????? f2438f08 ???????? ???????? ???????? ???????? f2438f18 ???????? ???????? ???????? ???????? f2438f28 ???????? ???????? ???????? ???????? f2438f38 ???????? ???????? ???????? ???????? f2438f48 ???????? ???????? ???????? ???????? 0: kd>
Those question marks tell us that the esp register contains a virtual address that is either undefined, or not visible to the debugger because it's paged out. To determine which, we can use the debugger's extremely useful !pte extension command. Note that the !pte command can't take a register name like esp, so we have to supply the register's contents, obtained from the register display or the dd command output:
0: kd> !pte f2438ed8
F2438ED8 - PDE at C0300F24 PTE at C03C90E0
contains 01001963 contains 7243C000
pfn 1001 G-DA--KWV not valid
PageFile 0
Offset 7243c
Protect: 0
0: kd>
It seems that the page containing the address in the esp register is paged out ("not valid" – specifically, its contents are, or rather were, in the paging file), so the debugger can't display them. Inspection of the addresses in esp and ebp shows that this page is the next page earlier in virtual address space than the page pointed to by ebp. What could cause a page of the kernel stack to be paged out, when the kernel stack of the executing thread is supposed to be nonpageable?
Suppose this page was not supposed to be part of the kernel stack at all?
A kernel stack overflow is in fact exactly what we're looking at here. It's worth thinking for a moment about just how this ends up as a double fault. The kernel stack is normally surrounded by a pair of inaccessible pages. When the last "good" location on the stack is used, the stack pointer (esp) points to the first byte in the last (lowest-addressed) valid page of the stack. A subsequent push of a DWORD (as was attempted here) decrements the stack pointer by 4 and then attempts to copy the DWORD into the location now pointed to by esp. However, after the decrement by 4, esp points to an invalid address. Therefore, the attempt to write to the stack raises an ordinary access violation or page fault exception.
The processor then traps through the IDT entry for the exception. As with all exceptions, before transferring control to the location pointed to by the IDT, it must push the address of the instruction that raised the exception onto the stack. But in doing this, it starts with the very same stack pointer value that the "push" instruction was using, that raised the access violation in the first place! If the original push instruction couldn't write anything there, the "automatic push" performed by the exception mechanism can't either.
That's the second of the two faults that make up this "double fault." In effect the processor now tries again, using an IDT entry specfically for double faults. But this IDT entry specifies a new hardware task in which to report the exception. The new hardware task has a new value for esp, pointing to a completely different stack. The processor state at the time of the exception is recorded in the task state segment previously in use. We can tell the debugger to set itself back to that state via the .tss command.
There are a few different ways in which drivers can overflow the kernel stack. In this case it appears that the driver's two components, here called "xyzspud" and "xyzwdm," seem to be calling each other back and forth in a recursive pattern:
0: kd> kv 20 ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. f24391c0 bfda8aa8 00000001 81fa85e0 00000d12 xyzwdm+0x707f4 f24391e0 bfda8af5 00000001 81fa85e0 00000d12 xyzspud+0xeaa8 f243920c bfda09ba 00000001 81fa85e0 00000d12 xyzspud+0xeaf5 f243924c bfd9c06a 81f44be0 bfdb1951 81fa85e0 xyzspud+0x69ba f243929c bfb51f02 00000000 00000000 00000001 xyzspud+0x206a f24392ec bfb2cd9f 81176748 f2439344 e14aa508 xyzwdm+0x96f02 f2439304 bfb2ab10 00000000 00000000 00000000 xyzwdm+0x71d9f f2439354 bfb2ba1e 81f37dc8 81fa83e0 f00dface xyzwdm+0x6fb10 f2439654 bfda8aa8 00000001 81fa83e0 00001003 xyzwdm+0x70a1e f2439674 bfda8af5 00000001 81fa83e0 00001003 xyzspud+0xeaa8 f24396a0 bfda09ba 00000001 81fa83e0 00001003 xyzspud+0xeaf5 f24396e0 bfd9c06a 81f44be0 bfdb1951 81fa83e0 xyzspud+0x69ba f2439730 bfb51f02 00000000 00000000 00000001 xyzspud+0x206a f2439780 bfb2cd9f 8131e308 f24397d8 e14aa508 xyzwdm+0x96f02 f2439798 bfb2ab10 00000000 00000000 00000000 xyzwdm+0x71d9f f24397e8 bfb2ba1e 81f37dc8 81f28970 f00dface xyzwdm+0x6fb10 f2439ae8 bfda8aa8 00000001 81f28970 00000d04 xyzwdm+0x70a1e f2439b08 bfda8af5 00000001 81f28970 00000d04 xyzspud+0xeaa8 f2439b34 bfda09ba 00000001 81f28970 00000d04 xyzspud+0xeaf5 f2439b74 bfd9c06a 81f44be0 bfdb1951 81f28970 xyzspud+0x69ba [...]
We also see that at one point in each sequence of calls, the stack pointer (represented in the "ChildEBP column") is decremented by a rather large amount. 700 or so (decimal) bytes is a lot to either store as local variables or pass by value, particularly if several procedures in your call tree are each doing the same thing. Careless use of C++ (passing data structures by value on the stack) can cause this, though in this case the culprit appeared to be local variables.
Incidentally, the driver represented here is a particular version of an actual driver distributed for a fairly popular sound card. We've changed the names to protect the guilty.
Conclusion
In this article we've presented the common types of register context issues you'll encounter with the debugger and how to resolve them. In a later article we'll have more to say about how to interpret the stacks and other information displayed.
Jamie Hanrahan has been writing device drivers for Windows NT and its successors since the 1992 Windows NT Driver Developer's Conference. His consulting practice operates under the name Kernel Mode Systems. He is also a partner, with Brian Catlin, in Azius Developer Training, developing and presenting seminars in kernel mode development, debugging, and troubleshooting.