wiki:MacBacktrace

Interpreting BOINC Crash Backtraces on the Mac

Last updated 8/17/12

Although this page is written primarily for the Mac, many of these techniques can be used on Linux and UNIX systems as well.

BOINC Manager, BOINC Client, and the BOINC library libboinc.a all generate debugging information in their respective stderr.txt files in the event of a crash, including a backtrace. When running on Mac OS 10.3 or OS 10.4, the backtrace is generated correctly by BOINC and project applications using libboinc.a for all BOINC versions since 5.10.3. However, BOINC versions earlier than 6.1.11 do not correctly dump a backtrace under Mac OS 10.5.

Executables released for general distribution usually have had most of their symbols stripped, so the backtrace generated when there is a crash in the field will typically only have hex addresses with few if any symbols.

Whenever we release BOINC for the Mac, we archive the entire executable files with symbols for the Client and Manager before stripping the symbols for general distribution. For BOINC version x.y.z, you can download these from: <http://boinc.berkeley.edu/dl/boinc_x.y.z_macOSX_SymbolTables.zip>

If you have a crash dump without debugging symbols, there are several ways to find the associated symbols. These same techniques can be used for project applications as well as the BOINC Manager and Client:

  1. Replace the stripped executable with the unstripped version and run again
  2. Use the atos command-line utility to convert hex addresses to symbols
  3. Use the GDB debugger to convert hex addresses to symbols

Replace the stripped executable with the unstripped version and run again

Replace the stripped executable with the unstripped one for the correct architecture, and run it again to reproduce the crash with full symbols.

The boinc_x.y.z_macOSX_SymbolTables.zip file contains a separate unstripped executable for the BOINC Manager and Client for BOINC version x.y.z.)

The BOINC Manager's executable is inside its bundle. You can replace the Manager executable on the command line thus:

sudo cp -f {path}/BOINCManager_i386 /Applications/BOINCManager.app/Contents/MacOS/BOINCManager

You should then confirm that the ownership and permissions have not changed:

ls -al /Applications/BOINCManager.app/Contents/MacOS

and you should see something like this:

-r-xr-sr-x  1 boinc_master  boinc_master  9197816 Mar 12 03:04 BOINCManager

Note that you must rename the unstripped executable to have the same file name as the original.

Similarly, to replace the BOINC Client inside the Manager's bundle:

sudo cp -f {path}/boinc_i386 /Applications/BOINCManager.app/Contents/Resources/boinc

Then confirm that the ownership and permissions have not changed:

ls -al /Applications/BOINCManager.app/Contents/Resources/boinc

and you should see something like this:

-r-sr-sr-x  1 boinc_master  boinc_master  1184600 Mar 12 03:06 boinc



Use the atos command-line utility to convert hex addresses to symbols

Use the atos command-line utility to convert the hex address in the crash backtrace to its symbol. This will work for project applications as well as the BOINC Manager and Client, but not for separate .dSYM files.

For example, to find a symbol in BOINC Manager version 5.10.45, at the command line in the Terminal application, first enter:

atos -o {path}/boinc_5.10.45_macOSX_SymbolTables/SymbolTables/BOINCManager_i386

then enter any number of hex values separated by spaces or carriage returns. For example, entering the following:

0x0009dc57 0x0009de45 0x000151f1 0x0007210a

will produce the following output:

   QCRGetBacktraceAtIndex (in BOINCManager_i386) (QCrashReport.c:623)
   QCRPrintBacktraces (in BOINCManager_i386) (QCrashReport.c:756)
   boinc_catch_signal (in BOINCManager_i386) (diagnostics.C:554)
   CProgressBar::UpdateValue(double) (in BOINCManager_i386) (sg_ProgressBar.cpp:104)


Use the GDB debugger to convert hex addresses to symbols

Use gdb to convert the hex address in the crash backtrace to its symbol. This will work for project applications as well as the BOINC Manager and Client, including separate .dSYM files.

If the symbols are in the same file as the executable, or if the separate .dSYM file is in the same directory as the executable, first enter the following on the command line in the Terminal application:

gdb {path to executable file}

If the executable file is not in the same directory as the .dSYM file, do this instead:

gdb
   symbol {path to .dSYM file}

Then, for each hex address of the form 0x12345678, enter the following:

info line *0x12345678

Be sure to include the asterisk!

For example, again examining BOINC Manager version 5.10.45:

   gdb {path}/boinc_5.10.45_macOSX_SymbolTables/SymbolTables/BOINCManager_i386
   info line *0x0007210a

produces the following output:

   Line 104 of "/Volumes/Sage/BOINC_Mac/boinc_5_10_45_tag/mac_build/../clientgui/
   sg_ProgressBar.cpp" starts at address 0x72102 
   <__MIG_check__Reply__io_service_close_t+29> and ends at 0x7210c 
   <__MIG_check__Reply__io_service_close_t+39>.

This method is described at: http://developer.apple.com/technotes/tn2004/tn2123.html#LISTPOSDEP. There is also much useful information on the subject at: http://developer.apple.com/tools/xcode/symbolizingcrashdumps.html (scroll down to the section "Symbolizing Crash Dumps".)

Notes: this web page says that the atos utility properly handles separate .dSYM files if you use the version of atos the which ships with XCODE Tools version 3.0. Unfortunately, this is not true. Also, these pages are focused on the crash reports automatically generated by the Mac OS, which are a bit different from the ones generated by BOINC.

Last modified 12 years ago Last modified on Aug 17, 2012, 4:11:07 AM