Minidump Analysis 61 0: kd> lmv m dxg start end module name bf9c3000 bf9d4580 dxg (pdb symbols) Loaded symbol image file: dxg.sys Mapped memory image file: c:\websymbols\dxg.sys\41107B9311580\dxg.sys Image path: dxg.sys Image name: dxg.sys Timestamp: Wed Aug 04 07:00:51 2004 (41107B93) CheckSum: 0001D181 ImageSize: 00011580 File version: 5.1.2600.2180 Product version: 5.1.2600.2180 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 3.7 Driver File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operating System InternalName: dxg.sys OriginalFilename: dxg.sys ProductVersion: 5.1.2600.2180 FileVersion: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158) FileDescription: DirectX Graphics Driver LegalCopyright: © Microsoft Corporation. All rights reserved. 0: kd> lmv m win32k start end module name bf800000 bf9c2180 win32k # (pdb symbols) Loaded symbol image file: win32k.sys Mapped memory image file: c:\websymbols\win32k.sys\45F013F61c2180\win32k.sys Image path: win32k.sys Image name: win32k.sys Timestamp: Thu Mar 08 13:47:34 2007 (45F013F6) CheckSum: 001C4886 ImageSize: 001C2180 File version: 5.1.2600.3099 Product version: 5.1.2600.3099 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 3.7 Driver File date: 00000000.00000000 Translations: 0406.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operativsystem InternalName: win32k.sys OriginalFilename: win32k.sys ProductVersion: 5.1.2600.3099 FileVersion: 5.1.2600.3099 (xpsp_sp2_gdr.070308-0222) FileDescription: Win32-flerbrugerdriver LegalCopyright: © Microsoft Corporation. Alle rettigheder forbeholdes. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 62 PART 2: Professional Crash Dump Analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Minidump Analysis 63 SYMBOLS AND IMAGES Suppose we have a minidump with a stack trace that involves our product driver and due to some reason WinDbg doesn’t pick symbols automatically and shows the following stack trace and crash address pointing to driver.sys module: 1: kd> kL ChildEBP RetAddr WARNING: Stack unwind information not available. Following frames may be wrong. ba0fd0e4 bfabd64b driver+0×2df2a ba0fd1c8 bf8b495d driver+0×1f64b ba0fd27c bf9166ae win32k!NtGdiBitBlt+0×52d ba0fd2d8 bf9168d0 win32k!TileWallpaper+0xd4 ba0fd2f8 bf826c83 win32k!xxxDrawWallpaper+0×50 ba0fd324 bf8651df win32k!xxxDesktopPaintCallback+0×48 ba0fd388 bf8280f3 win32k!xxxEnumDisplayMonitors+0×13a ba0fd3d4 bf8283ab win32k!xxxInternalPaintDesktop+0×66 ba0fd3f8 80833bdf win32k!NtUserPaintDesktop+0×41 ba0fd3f8 7c9485ec nt!KiFastCallEntry+0xfc 1: kd> r eax=000007d0 ebx=000007d0 ecx=00000086 edx=bfb371a3 esi=bc492000 edi=bfb3775b eip=bfacbf2a esp=ba0fd0b8 ebp=ba0fd0e4 iopl=0 nv up ei pl nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202 driver+0×2df2a: bfacbf2a f3a5 rep movs dword ptr es:[edi],dword ptr [esi] es:0023:bfb3775b=e4405a64 ds:0023:bc492000=???????? Let’s get the timestamp of this module too: 1: kd> lmv m driver start end module name bfa9e000 bfb42a00 driver T (no symbols) Loaded symbol image file: driver.sys Image path: driver.sys Image name: driver.sys Timestamp: Thu Mar 01 20:50:04 2007 (45E73C7C) CheckSum: 000A5062 ImageSize: 000A4A00 Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0 We see that no symbols for driver.sys were found and this is also indicated by the absence of function names and the presence of huge code offsets like 0x2df2a. Perhaps we don’t have a symbol server and store our symbol files somewhere. Or we got symbols from the developer of the recent fix that bugchecks and we want to apply them. In any case if we add a path to Symbol Search Path dialog (File -> Symbol File Path Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 64 PART 2: Professional Crash Dump Analysis …) or use .sympath WinDbg command we are able to get better stack trace and crash point: 1: kd> .reload Loading Kernel Symbols . Loading User Symbols Loading unloaded module list . Unable to load image driver.sys, Win32 error 0n2 *** WARNING: Unable to verify timestamp for driver.sys 1: kd> kL ChildEBP RetAddr ba0fd0c0 bfabc399 driver!ProcessBytes+0×18 ba0fd0e4 bfabd64b driver!ProcessObject+0xc9 ba0fd1c8 bf8b495d driver!CacheBitBlt+0×13d ba0fd27c bf9166ae win32k!NtGdiBitBlt+0×52d ba0fd2d8 bf9168d0 win32k!TileWallpaper+0xd4 ba0fd2f8 bf826c83 win32k!xxxDrawWallpaper+0×50 ba0fd324 bf8651df win32k!xxxDesktopPaintCallback+0×48 ba0fd388 bf8280f3 win32k!xxxEnumDisplayMonitors+0×13a ba0fd3d4 bf8283ab win32k!xxxInternalPaintDesktop+0×66 ba0fd3f8 80833bdf win32k!NtUserPaintDesktop+0×41 ba0fd3f8 7c9485ec nt!KiFastCallEntry+0xfc 1: kd> r eax=000007d0 ebx=000007d0 ecx=00000086 edx=bfb371a3 esi=bc492000 edi=bfb3775b eip=bfacbf2a esp=ba0fd0b8 ebp=ba0fd0e4 iopl=0 nv up ei pl nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202 driver!ProcessBytes+0×18: bfacbf2a f3a5 rep movs dword ptr es:[edi],dword ptr [esi] es:0023:bfb3775b=e4405a64 ds:0023:bc492000=???????? Because WinDbg reports that it was unable to verify timestamp for driver.sys we might want to double check the return address saved when ProcessBytes function was called. If symbols are correct then disassembling the return address backwards will most Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Minidump Analysis 65 likely show ProcessObject function code and “call” instruction with ProcessBytes address. Unfortunately minidumps don’t have code except for the currently executing function: 1: kd> ub bfabc399 ^ Unable to find valid previous instruction for 'ub bfabc399' 1: kd> uf driver!ProcessObject No code found, aborting Therefore we need to point WinDbg to our driver.sys which contains executable code. This can be done by specifying a path in Executable Image Search Path dialog (File -> Image File Path …) or using .exepath WinDbg command. Now we get more complete stack trace and we are able to double check the re- turn address: 1: kd> .reload Loading Kernel Symbols . Loading User Symbols Loading unloaded module list . Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 66 PART 2: Professional Crash Dump Analysis 1: kd> kL ChildEBP RetAddr ba0fd0c0 bfabc399 driver!ProcessBytes+0×18 ba0fd0e4 bfabd64b driver!ProcessObject+0xc9 ba0fd104 bfac5aac driver!CacheBitBlt+0×13d ba0fd114 bfac6840 driver!ProcessCommand+0×150 ba0fd140 bfac1878 driver!CheckSurface+0×258 ba0fd178 bfaba0ee driver!CopyBitsEx+0xfa ba0fd1c8 bf8b495d driver!DrvCopyBits+0xb6 ba0fd27c bf9166ae win32k!NtGdiBitBlt+0×52d ba0fd2d8 bf9168d0 win32k!TileWallpaper+0xd4 ba0fd2f8 bf826c83 win32k!xxxDrawWallpaper+0×50 ba0fd324 bf8651df win32k!xxxDesktopPaintCallback+0×48 ba0fd388 bf8280f3 win32k!xxxEnumDisplayMonitors+0×13a ba0fd3d4 bf8283ab win32k!xxxInternalPaintDesktop+0×66 ba0fd3f8 80833bdf win32k!NtUserPaintDesktop+0×41 ba0fd3f8 7c9485ec nt!KiFastCallEntry+0xfc 1: kd> ub bfabc399 driver!ProcessObject+0xb7: bfabc387 57 push edi bfabc388 40 inc eax bfabc389 50 push eax bfabc38a e861fb0000 call driver!convert (bfacbef0) bfabc38f ff7508 push dword ptr [ebp+8] bfabc392 57 push edi bfabc393 50 push eax bfabc394 e879fb0000 call driver!ProcessBytes (bfacbf12) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Minidump Analysis 67 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 68 PART 2: Professional Crash Dump Analysis INTERRUPTS AND EXCEPTIONS EXPLAINED EXCEPTIONS AB INITIO Where do native exceptions come from? How do they propagate from hardware and eventually result in crash dumps? I was asking these questions when I started doing crash dump analysis more than four years ago and I tried to find answers using IA-32 Intel® Architecture Software Developer’s Manual, WinDbg and complete memory dumps. Let’s look at some findings. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Interrupts and Exceptions Explained 69 X86 INTERRUPTS How do exceptions happen in the first place and how does the execution flow reach KiDispatchException function? When some abnormal condition happens such as a breakpoint, division by zero or memory protection violation then the normal CPU execu- tion flow (code stream) is interrupted (I use the terms “interrupt” and “exception” interchangeably here). The type of interrupt is specified by a number called interrupt vector number. CPU has to transfer execution to some procedure in memory to handle that interrupt. CPU has to find that procedure, theoretically either having one procedure for all interrupts and specifying an interrupt vector number as a parameter or having a table containing pointers to various procedures that correspond to different interrupt vectors. Intel x86 and x64 CPUs use the latter approach which is depicted on the follow- ing diagram: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 70 PART 2: Professional Crash Dump Analysis CPU KiTrap00 (808347ca) Mouse Keyboard HD KiTrap01 (8083494a) KiTrap02 (80834a3d) . 8003f400: 8003f408: 8003f410: 8003f418: 8003f400 IDTR . . . nt!KiTrap00: 808347ca push 0 808347cc mov word ptr [esp+2],0 808347d3 push ebp 808347d4 push ebx 808347d5 push esi 808347d6 push edi 808347d7 push fs 808347d9 mov ebx,30h 808347de mov fs,bx 808347e0 mov ebx,dword ptr fs:[0] 808347e7 push ebx . . . . Bf972586: idiv eax, ebx ; ebx = 0 . 1. Exception (Vector 0) 3. Transfer execution 2: Interrupt dIspatch . . Vector 52 Vector b2 Vector 72 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... ds:0950:0028=???????? KiPageFault function was called and from the dumped IDT we see that it corresponds to the interrupt vector 14 (0xE) called on any memory reference that is not present in physical memory Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 82 PART 2: Professional Crash Dump Analysis Now I’m going to dump the raw stack around fffffade`4e890780 address to see data... for x86 Windows flat memory model so we can easily identify this pattern on raw stack data Why should we care about an interrupt frame? This is because in complete memory dumps we can see exceptions that happened in user space and processed at the time the dump was saved Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 84 PART 2: Professional Crash Dump Analysis Let’s look... crash dump These are OS and user-defined numbers The first 32 vectors are reserved by Intel Before Windows switches CPU to protected mode during boot process it creates IDT table in memory and sets IDTR to point to it by using SIDT instruction Let me now illustrate this by using a UML class diagram annotated by pseudocode that shows what CPU does before calling the appropriate procedure The pseudocode... nt!KiTrap00+0×88 f2178c90 bf94c23c driver!foo+0xf9 f2178d54 80833bdf driver!bar+0×11c Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 74 PART 2: Professional Crash Dump Analysis Dumping memory around ESP value (f2178c1c) shows the values processor pushes when divide by zero exception happens: 0: kd> dds f2178c1c-100 f2178c1c+100 f2178b80 00000000 f2178b84 f2178b50 f2178b88... Split-Merge on www.verypdf.com to remove this watermark 90 PART 2: Professional Crash Dump Analysis Stack trace doesn’t look good to us because there is no BaseProcessStart or BaseThreadStart function, perhaps because we specified ESP value twice instead of EBP and ESP Let’s hope to find EBP value by dumping the memory around ESP: 0: kd> dds 0012f934-10 0012f934+100 0012f924 00000000 0012f928 0012f934... nt!KiRaiseAssertion Exact matches: nt!KiXmmException = Let’s look at some dump BugCheck 1E, {ffffffffc0000005, fffffade5ba2d643, 0, 28} This is KMODE_EXCEPTION_NOT_HANDLED bugcheck and obviously it could have been an invalid memory access And indeed the stack WinDbg shows after opening a dump and entering !analyze -v command is: RSP fffffade`4e88fe68 fffffade`4e88fe70 fffffade`4e8905f0... PageFaultCount 232357 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 233170 DebugPort 899b6a40 We see that processA has a DebugPort and the presence of it usually indicates that some exception happened Therefore if we dump all processes by entering !process 0 1 command we can search for any unhandled exceptions Indeed if we switch to this process (we can also use !process 89a94800 ff command for dumps coming... +0×180 Prcb : _KPRCB _KGDTENTRY64 _KTSS64 _KPCR _KPRCB _KSPIN_LOCK_QUEUE _KIDTENTRY64 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 78 PART 2: Professional Crash Dump Analysis Next we dump the first entry of IDT array and glue together OffsetHigh, OffsetMiddle and OffsetLow fields to form the interrupt procedure address corresponding to the interrupt vector 0, divide by zero... complicated because the processor has to switch the current user-mode stack to kernel mode stack Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 72 PART 2: Professional Crash Dump Analysis The following diagram is for 32-bit x86 processor: Exception or interrupt (VectorNumber) in kernel mode push EFLAGS push CS push EIP push ErrorCode EIP := IDT[VectorNumber].ExtendedOffset . PART 2: Professional Crash Dump Analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Minidump Analysis 63 SYMBOLS AND IMAGES. watermark. Minidump Analysis 67 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 68 PART 2: Professional Crash Dump Analysis INTERRUPTS