Interpreting BOINC Crash Backtraces on the Mac
Last updated 8/17/12
Although this page is written primarily for the Mac, many of these techniques can be used on Linux and UNIX systems as well.
BOINC Manager, BOINC Client, and the BOINC library libboinc.a all generate debugging information in their respective stderr.txt files in the event of a crash, including a backtrace. When running on Mac OS 10.3 or OS 10.4, the backtrace is generated correctly by BOINC and project applications using libboinc.a for all BOINC versions since 5.10.3. However, BOINC versions earlier than 6.1.11 do not correctly dump a backtrace under Mac OS 10.5.
Executables released for general distribution usually have had most of their symbols stripped, so the backtrace generated when there is a crash in the field will typically only have hex addresses with few if any symbols.
Whenever we release BOINC for the Mac, we archive the entire executable files with symbols for the Client and Manager before stripping the symbols for general distribution. For BOINC version x.y.z, you can download these from:
<http://boinc.berkeley.edu/dl/boinc_x.y.z_macOSX_SymbolTables.zip>
If you have a crash dump without debugging symbols, there are several ways to find the associated symbols. These same techniques can be used for project applications as well as the BOINC Manager and Client:
- Replace the stripped executable with the unstripped version and run again
- Use the atos command-line utility to convert hex addresses to symbols
- Use the GDB debugger to convert hex addresses to symbols
Replace the stripped executable with the unstripped version and run again
Replace the stripped executable with the unstripped one for the correct architecture, and run it again to reproduce the crash with full symbols.
The boinc_x.y.z_macOSX_SymbolTables.zip file contains a separate unstripped executable for the BOINC Manager and Client for BOINC version x.y.z.)
The BOINC Manager's executable is inside its bundle. You can replace the Manager executable on the command line thus:
sudo cp -f {path}/BOINCManager_i386 /Applications/BOINCManager.app/Contents/MacOS/BOINCManager
You should then confirm that the ownership and permissions have not changed:
ls -al /Applications/BOINCManager.app/Contents/MacOS
and you should see something like this:
-r-xr-sr-x 1 boinc_master boinc_master 9197816 Mar 12 03:04 BOINCManager
Note that you must rename the unstripped executable to have the same file name as the original.
Similarly, to replace the BOINC Client inside the Manager's bundle:
sudo cp -f {path}/boinc_i386 /Applications/BOINCManager.app/Contents/Resources/boinc
Then confirm that the ownership and permissions have not changed:
ls -al /Applications/BOINCManager.app/Contents/Resources/boinc
and you should see something like this:
-r-sr-sr-x 1 boinc_master boinc_master 1184600 Mar 12 03:06 boinc
Use the atos command-line utility to convert hex addresses to symbols
Use the atos command-line utility to convert the hex address in the crash backtrace to its symbol. This will work for project applications as well as the BOINC Manager and Client, but not for separate .dSYM files.
For example, to find a symbol in BOINC Manager version 5.10.45, at the command line in the Terminal application, first enter:
atos -o {path}/boinc_5.10.45_macOSX_SymbolTables/SymbolTables/BOINCManager_i386
then enter any number of hex values separated by spaces or carriage returns. For example, entering the following:
0x0009dc57 0x0009de45 0x000151f1 0x0007210a
will produce the following output:
QCRGetBacktraceAtIndex (in BOINCManager_i386) (QCrashReport.c:623) QCRPrintBacktraces (in BOINCManager_i386) (QCrashReport.c:756) boinc_catch_signal (in BOINCManager_i386) (diagnostics.C:554) CProgressBar::UpdateValue(double) (in BOINCManager_i386) (sg_ProgressBar.cpp:104)
Use the GDB debugger to convert hex addresses to symbols
Use gdb to convert the hex address in the crash backtrace to its symbol. This will work for project applications as well as the BOINC Manager and Client, including separate .dSYM files.
If the symbols are in the same file as the executable, or if the separate .dSYM file is in the same directory as the executable, first enter the following on the command line in the Terminal application:
gdb {path to executable file}
If the executable file is not in the same directory as the .dSYM file, do this instead:
gdb
symbol {path to .dSYM file}
Then, for each hex address of the form 0x12345678, enter the following:
info line *0x12345678
Be sure to include the asterisk!
For example, again examining BOINC Manager version 5.10.45:
gdb {path}/boinc_5.10.45_macOSX_SymbolTables/SymbolTables/BOINCManager_i386
info line *0x0007210a
produces the following output:
Line 104 of "/Volumes/Sage/BOINC_Mac/boinc_5_10_45_tag/mac_build/../clientgui/ sg_ProgressBar.cpp" starts at address 0x72102 <__MIG_check__Reply__io_service_close_t+29> and ends at 0x7210c <__MIG_check__Reply__io_service_close_t+39>.
This method is described at: http://developer.apple.com/technotes/tn2004/tn2123.html#LISTPOSDEP. There is also much useful information on the subject at: http://developer.apple.com/tools/xcode/symbolizingcrashdumps.html (scroll down to the section "Symbolizing Crash Dumps".)
Notes: this web page says that the atos utility properly handles separate .dSYM files if you use the version of atos the which ships with XCODE Tools version 3.0. Unfortunately, this is not true. Also, these pages are focused on the crash reports automatically generated by the Mac OS, which are a bit different from the ones generated by BOINC.
