Changes between Version 2 and Version 3 of AppDebugWin


Ignore:
Timestamp:
Apr 25, 2007, 11:12:31 AM (17 years ago)
Author:
Nicolas
Comment:

Required manual changes to automatic conversion. That was a big mess. Did a few typo corrections too.

Legend:

Unmodified
Added
Removed
Modified

  • AppDebugWin

    v2 v3  
    3636 This section is going to describe what the output of a crash dump looks like and what it contains.
    3737
    38 
    39 
    40 
    4138=== Debugger version ===
    4239
    … …  
    5148srv*C:\DOCUME~1\romw\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/symstore
    5249}}}
    53 ::This area provides some basic information about the version of the BOINC debugger  being used, when the crash occured, and what the internal version of the Windows debugger technology is being used.
     50
     51This area provides some basic information about the version of the BOINC debugger  being used, when the crash occured, and what the internal version of the Windows debugger technology is being used.
    5452
    5553Symbol search paths are used to inform the debugger where it might be able to find the symbol files related to the modules loaded in memory.  Entries prefixed with 'srv*' are used to denote a web based symbol store. DbgHelp will use them if symsrv can be loaded at the time of the crash.
    5654
    5755If you see a load library failure for either dbghelp.dll or symsrv.dll then there is a pretty good chance that most of the data in the dump will be useless.
    58 
    59 
    6056
    6157
    … …  
    7066    Product Version: 5.2.3790.1830
    7167}}}
    72 ::Information about which modules were loaded into the processes memory space can be found here. The first hexadecimal value is the address in memory in which the module was loaded;  the second hexadecimal is the size of the module.
     68
     69Information about which modules were loaded into the processes memory space can be found here. The first hexadecimal value is the address in memory in which the module was loaded;  the second hexadecimal is the size of the module.
    7370
    7471If a version record was found inside the module, it'll be dumped out as part of the module list dump.
    7572
    76 If the correct symbol file can be found, it'll display what symbol file type is in use.  The three most common symbol types in modern software are 'PDB', 'exports', and  '-no symbols-'. Symbol files are pretty large and so most projects do not like to include  them as part of the APP VERSION package. Luckily Microsoft has created a technology called  Symbol Stores which enable an application to be able to grab its symbol file from a web  server at the time of a crash in a compressed format. We will describe setting up a symbol store later in this document.
     73If the correct symbol file can be found, it'll display what symbol file type is in use.  The three most common symbol types in modern software are 'PDB', 'exports', and  '-no symbols-'. Symbol files are pretty large and so most projects do not like to include  them as part of the APP VERSION package. Luckily Microsoft has created a technology called Symbol Stores which enable an application to be able to grab its symbol file from a web server at the time of a crash in a compressed format. We will describe setting up a symbol store later in this document.
    7774
    7875PDB files are generated at compilation time and usually have to be turned on for release builds. This file will contain all the needed information to generate a pretty good  callstack which you can use to diagnose problems.
    … …  
    8178
    8279No symbols means that the runtime debugger could not determine a way to give you any  symbolic information. You'll only receive function pointers in the callstack.
    83 
    84 
    85 
    8680
    8781=== Process Information ===
    … …  
    109103WorkingSetSize: 19210240, PeakWorkingSetSize: 26361856, PageFaultCount: 6729
    110104}}}
    111 ::This is some overall useful information about the process. Most of the time the 'Virtual Memory', 'Pagefile', and 'Working Set' are the most useful indications  of whether or not the process was under low available memory pressure from the OS.
    112 
    113 
    114 
     105
     106This is some overall useful information about the process. Most of the time the 'Virtual Memory', 'Pagefile', and 'Working Set' are the most useful indications  of whether or not the process was under low available memory pressure from the OS.
    115107
    116108=== Thread Information ===
    … …  
    119111*** Dump of the Worker thread (a4): ***
    120112}}}
    121 ::This identifies the thread for which additional information is going to be  displayed. Both the thread name and thread ID are displayed. To set the thread name for any thread you have created in your program just call diagnostics_set_thread_name() as defined in diagnostics.h to set the thread name for the currently executing thread.
    122 
    123 
    124 
     113
     114This identifies the thread for which additional information is going to be  displayed. Both the thread name and thread ID are displayed. To set the thread name for any thread you have created in your program just call diagnostics_set_thread_name() as defined in diagnostics.h to set the thread name for the currently executing thread.
    125115
    126116==== General Information ====
    … …  
    130120Status: Waiting, Wait Reason: UserRequest, Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 38241696.000000
    131121}}}
    132 ::Status shows what state the thread was in when the snapshot was taken. If the thread is waiting, wait reason will describe why the thread is waiting. If the thread is running both the base thread priority and current thread priority will be displayed.
     122
     123Status shows what state the thread was in when the snapshot was taken. If the thread is waiting, wait reason will describe why the thread is waiting. If the thread is running both the base thread priority and current thread priority will be displayed.
    133124
    134125Kernel time, user time, and wait time describe how much time, in nanoseconds, the thread has spent in each of those states.
    135 
    136 
    137 
    138126
    139127==== Unhandled Exception Record ====
    … …  
    143131Reason: Breakpoint Encountered (0x80000003) at address 0x7C822583
    144132}}}
    145 ::This section if included in the thread describes what event ocurred that caused the runtime debugger to engage. Structured Exceptions in Windows are not the same thing as C++ exceptions. Unless you are using a compiler that knows about both types it is unlikely that a C++ catch is going to actually catch this type of exception.
    146 
    147 Further information about Structured Exception Handling can be found  [http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/about_structured_exception_handling.asp  here. ]
     133
     134This section if included in the thread describes what event ocurred that caused the runtime debugger to engage. Structured Exceptions in Windows are not the same thing as C++ exceptions. Unless you are using a compiler that knows about both types it is unlikely that a C++ catch is going to actually catch this type of exception.
     135
     136Further information about Structured Exception Handling can be found  [http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/about_structured_exception_handling.asp  here.]
    148137
    149138It is important to note that both hardware and software exceptions can bubble up from the operating system through this mechinism.
    150139
    151 The example above shows that  EXCEPTION_BREAKPOINT(PlatformSDK\Include\winbase.h) was raised at 0x7C822583.  EXCEPTION_BREAKPOINT is defined as STATUS_BREAKPOINT(PlatformSDK\Include\ntstatus.h)  which is defined as ((NTSTATUS)0x80000003L).
    152 
    153 
    154 
     140The example above shows that  EXCEPTION_BREAKPOINT(PlatformSDK\Include\winbase.h) was raised at 0x7C822583. EXCEPTION_BREAKPOINT is defined as STATUS_BREAKPOINT(PlatformSDK\Include\ntstatus.h) which is defined as ((NTSTATUS)0x80000003L).
    155141
    156142==== Registers ====
    … …  
    162148cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
    163149}}}
    164 ::This is a basic dump of the processor registers at the time the exception was raised and will look different for each process type.
     150
     151This is a basic dump of the processor registers at the time the exception was raised and will look different for each process type.
    165152
    166153In this example these are the registers and flags for the Intel based x86 processor.
    167 
    168 
    169 
    170154
    171155==== Callstack ====
    … …  
    17916300aaffec 00000000 00426840 00000000 00000000 00000000 kernel32!_BaseThreadStart@8+0x0 (c:\boincsrc\main\boinc\api\graphics_impl.c:75)
    180164}}}
    181 ::This describes the state in which the thread was in at the time of the exception. ChildEBP and RetAddr are not really useful unless you can reproduce the issue using the same OS version.
     165
     166This describes the state in which the thread was in at the time of the exception. ChildEBP and RetAddr are not really useful unless you can reproduce the issue using the same OS version.
    182167
    183168Args to Child are the first four parameters passed to the function.
    … …  
    185170The next piece of information has the following format: <Module Name>!<Function Name>@<Function Ordinal>+<Symbol Offset> <File/Line Information>
    186171
    187 
     172''' XXX need to fix this '''
    188173
    189174Module Name   The friendly name for the DLL or EXE.   Function Name   The name of the function.   Function Ordinal   Function ordinals only apply to functions that are publically exported from a DLL.   Symbol Offset   If the symbol file being used has private symbols stripped, then the symbol displayed may        not be the correct symbol.     You can use the symbol offset to lookup the correct symbol       from the object file or some other tool that can map symbols to source.   File/Line Information   Source file and line number information is not available in all symbol file formats, but       if it is there it'll be displayed.    PDB files are generally the best symbol file format to use.
     175
    190176=== Debug Message Dump ===
    191177
    … …  
    193179*** Debug Message Dump ****
    194180}}}
    195 ::This feature is disabled by default.
     181
     182This feature is disabled by default.
    196183
    197184What is allows for is capturing the debugger viewport data at runtime just as though you were running the application within a debugger. Since all applications use the same block of memory is can slow down any and all applications that want to write to the debugger viewport, even on release builds which is why it is disabled by default. Video capture, edit, and playback software tends to dump data to the viewport even when running a release build.
    198185
    199186The following regedit script demonstrates how to enable the debug message dump:
    200 
    201187
    202188{{{
    … …  
    206192"CaptureMessages"=dword:00000001
    207193}}}
    208 ::      To disable able the message capture use this regedit script:
     194To disable able the message capture use this regedit script:
    209195{{{
    210196Windows Registry Editor Version 5.00
    … …  
    213199"CaptureMessages"=dword:00000000
    214200}}}
    215 ::
    216 
    217201
    218202=== Foreground Window Data ===
    … …  
    225209    Window Thread ID : ae8
    226210}}}
    227 ::This shows which window has the user input focus. The feature was originally meant to detect potiential problems with 3rd party application injecting code into BOINC applications and displaying UI to the user.
    228 
    229 This feature turns out to be problematic since the foreground window might be hung  which would mean that trying to get the window name and class would cause the runtime  debugger to hang as well. This feature will probably be removed in the future.
    230 
    231 
    232 
     211
     212This shows which window has the user input focus. The feature was originally meant to detect potential problems with 3rd party application injecting code into BOINC applications and displaying UI to the user.
     213
     214This feature turns out to be problematic since the foreground window might be hung which would mean that trying to get the window name and class would cause the runtime debugger to hang as well. This feature will probably be removed in the future.
    233215
    234216== Symbol Stores ==
    235217
    236218=== Introduction ===
    237  In order to obtain useful diagnostic information in the event of an application crash,  it is necessary to dump a callstack and any other relevant information about what was  going on at the time of the crash.  Symbols are only needed during a crash event,  therefore they are stripped from most applications to cut down on the binary size and  bandwidth requirements to deploy a new release.
     219
     220In order to obtain useful diagnostic information in the event of an application crash,  it is necessary to dump a callstack and any other relevant information about what was  going on at the time of the crash.  Symbols are only needed during a crash event,  therefore they are stripped from most applications to cut down on the binary size and  bandwidth requirements to deploy a new release.
    238221
    239222Without symbols, callstacks tend to be nothing more than a list of function pointers  in memory.  A developer has to load the un-stripped executable in memory using the  same operating system and similar processor to jump to that memory address in order  to determine the function name and parameters.  This is very labor intensive and  generally not a very fun job.
    … …  
    243226With the BOINC Runtime Debugger for Windows framework a project can publish their  symbol files and only have to distribute the application to each of the BOINC  clients.  When a crash event occurs the runtime framework will download the symbol  file from the symbol store and then proceed to dump as much diagnostic information  as possible to help projects diagnose the failure.
    244227
    245 
    246 
    247 
    248228=== Requirements ===
    249  You'll need the latest stable release of the  [http://www.microsoft.com/whdc/devtools/debugging/default.mspx    Debugging Tools for Windows. ]
     229You'll need the latest stable release of the  [http://www.microsoft.com/whdc/devtools/debugging/default.mspx Debugging Tools for Windows. ]
    250230
    251231Verify that your executable is setup to generate PDB debugging symbols for a release build.
    … …  
    255235You'll need to explictly name both your EXE and PDB before compilation since the debugger bases the name of the PDB file off of information that is stored in the executable header.
    256236
    257 
    258 
    259 
    260237=== Project Symbol Store ===
    261  Specifying a project wide symbol store is as easy as adding the symstore element to your config.xml file for the project.
     238
     239Specifying a project wide symbol store is as easy as adding the symstore element to your config.xml file for the project.
    262240
    263241Below is an XML shred with an example symstore element.
    264 
    265 
    266 
    267242
    268243{{{
    … …  
    274249}}}
    275250
    276 
    277 
    278251=== Adding symbols to the symbol store ===
    279252 [http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/using_symstore.asp Symstore] is a utility to manage symbol stores.  You'll want to create a local symbol store on your Windows build machine in which you'll initially add new symbol files with each revision of your application.
    … …  
    283256Below is an example command which you can run from the Windows command line or cygwin command line.
    284257
    285 
    286 
    287 
    288258{{{
    289259symstore.exe add /l /f c:\SampleSrc\*.pdb /s c:\symstore /compress /t "Sample" /v "5.02" /o /c "Application Release"
    290260}}}
    291 ::
    292 
    293261
    294262=== Uploading symbols to the symbol store ===
    295  Most projects tend to use scp to copy files between Windows machines and their project server.
     263
     264Most projects tend to use scp to copy files between Windows machines and their project server.
    296265
    297266The example below copies the entire symstore to the target location.  After the copy operation you can delete all the subdirectories except '000Admin' to save time uploading for future application symbols.
    298267
    299 
    300 
    301 
    302268{{{
    303269pscp.exe -r -C -batch c:\symstore sample@project.example.com:projects/sample/html/user/symstore
    304270}}}
    305 ::
    306271
    307272
    … …  
    309274
    310275=== Introduction ===
    311  In this section we'll list a few things to look for when reading the dumps.  Please keep in mind that every application is different, but there should be enough similiarity that you should be able to figure something out.
    312 
    313 
    314 
     276
     277In this section we'll list a few things to look for when reading the dumps.  Please keep in mind that every application is different, but there should be enough similiarity that you should be able to figure something out.
    315278
    316279=== Breakpoint Encountered (0x80000003) ===
    … …  
    33730000b2ffec 00000000 0043f4ba 00000000 00000000 00000000 kernel32!_BaseThreadStart@8+0x0 (boinc\api\graphics_impl.c:74)
    338301}}}
    339 ::This kind of error is an intentional error.  Somewhere in the code base it encountered a breakpoint.
     302
     303This kind of error is an intentional error.  Somewhere in the code base it encountered a breakpoint.
    340304
    341305The callstack will point to the function that started the call to the breakpoint function.
    … …  
    343307To add manual breakpoints to your code for diagnostics purposes you can call the Windows API:
    344308
    345 
    346309{{{
    347310void DebugBreak( void );
    348311}}}
    349 ::
    350 
    351312
    352313=== Invalid Parameter (0xc000000d) ===
    353  Starting with Visual Studio 2005, Microsoft re-vamped the whole C Runtime Library.  Part of the re-vamp process was to do parameter checking on each function.  Places that would normally return a NULL value now cause a structured exception to be thrown.
    354 
    355 The nature of this structed exception is different than most as they specifically coded it so that it will not engage the BOINC Runtime Debugger and it'll display a dialog box asking the user if they wish to debug the error.  If the user cancels the error code 0xc000000d is returned without any more information.
     314
     315Starting with Visual Studio 2005, Microsoft re-vamped the whole C Runtime Library.  Part of the re-vamp process was to do parameter checking on each function.  Places that would normally return a NULL value now cause a structured exception to be thrown.
     316
     317The nature of this structured exception is different than most as they specifically coded it so that it will not engage the BOINC Runtime Debugger and it'll display a dialog box asking the user if they wish to debug the error. If the user cancels the error code 0xc000000d is returned without any more information.
    356318
    357319To get more information with this error you'll need to create a function like this:
    358 
    359320
    360321{{{
    … …  
    378339#endif
    379340}}}
    380 ::The following code block should be added after the call to boinc_diagnostics_init():
    381 
     341
     342The following code block should be added after the call to boinc_diagnostics_init():
    382343
    383344{{{
    … …  
    391352#endif
    392353}}}
    393 ::When this issues happens in the future it'll describe which CRT function call was passed an invalid parameter and it should dump out the callstack for all threads.
     354
     355When this issues happens in the future it'll describe which CRT function call was passed an invalid parameter and it should dump out the callstack for all threads.
    394356
    395357The function blocks above overwrite the default behavior of the CRT when an invalid  parameter is detected.
    396 
    397 
    398 
    399358
    400359=== Privileged Instruction (0xc0000096) ===
    … …  
    42138001e1ffec 00000000 006363b0 00000000 00000000 00000000 kernel32!_BaseThreadStart@8+0x0 (boinc\api\graphics_impl.c:75)
    422381}}}
    423 ::In this example it appears the processor took exception to the fact that a user mode process attempted to push a kernel mode address onto the stack without first switching to kernel mode.
     382
     383In this example it appears the processor took exception to the fact that a user mode process attempted to push a kernel mode address onto the stack without first switching to kernel mode.
    424384
    425385Look at the EBP register, 'ffffffff' when converted into a signed int is equal to '-1'  and when converted to an unsigned int it is equal to 4GB.  On Windows anything above 2GB  is considered a kernel mode address.  If the Windows machine supports PAE and the /3GB boot option is specified in BOOT.INI then kernel addresses will start at 3GB instead.
    … …  
    427387What has probably happened here is that a function is about to be called and a 'push EBP' instruction was called to push a new address onto the stack, the CPU threw the exception  since the address was outside user mode land. EBP should have had a similar progression  as all the other stack frames ChildEBP values.
    428388
    429 If EBP had some random kernel mode address it would be pretty easy to dismiss this as a CPU overheating.  'ffffffff' begs the question is the stack being overwritten by an error result from another function?
     389If EBP had some random kernel mode address it would be pretty easy to dismiss this as a CPU overheating. 'ffffffff' begs the question is the stack being overwritten by an error result from another function?
    430390
    431391Investigation of this issue is still ongoing.
    432392
    433 
    434 
    435 
    436393=== Stack Overflow (0xc00000fd) ===
    437  An application will throw this exception when one of it's threads exceed the 1MB stack size allotment.
    438 
    439 
    440 
     394
     395An application will throw this exception when one of it's threads exceed the 1MB stack size allotment.
    441396
    442397=== Out of Memory Exception (0xe06d7363) ===
    … …  
    4814360012fff0 00000000 0097abd2 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 (crt\src\crt0.c:315)
    482437}}}
    483 ::In the above example the new opterator was being requested to allocate 4GB of memory.  An out of  memory exception was thrown.
    484 
    485 
    486 
     438
     439In the above example the `new` operator was being requested to allocate 4GB of memory. An out of memory exception was thrown.
     440