Changes between Version 9 and Version 10 of ServerDebug

Timestamp:

Apr 29, 2014, 9:12:30 PM (11 years ago)

Author:

davea

Comment:

--

ServerDebug

@@ -1 @@
-= Trouble-shooting server components =
+[[PageOutline]]
+= Trouble-shooting a BOINC server =
 
-A grab-bag of techniques for debugging BOINC server software:
+== Trouble-shooting tools ==
 
-== Log files ==
+=== Log files ===
 
 Each server component (scheduler, feeder, transitioner, etc.) has its own log file.
@@ -15 +16 @@
 To control the verbosity of the log files:
 
- * Scheduler: set the desired [ProjectOptions#Scheduler:logging logging options]
+ * Scheduler: set the desired [ProjectOptions#Loggin logging options]
  * File upload handler: set [ProjectOptions#misc fuh_debug_level].
- * Other components: pass "-d N" (1=least verbose, 4=most verbose)
+ * daemons: pass the cmdline arg "-d N" (1=least verbose, 4=most verbose)
+  If you run server components with '''-d 4''', their database queries will be logged.
+  This is verbose but extremely useful for tracking down database-level problems.
+
+=== Examining the database ===
+
+The [wiki:HtmlOps admin web interface] provides a web-based interface for
+browsing your project's database.
+
+You can also use MySQL tools such as
+ * The [http://dev.mysql.com/doc/refman/5.0/en/mysql.html mysql interpreter].
+   The '[http://dev.mysql.com/doc/refman/5.0/en/show-processlist.html show processlist;]' query
+   is useful for diagnosing DB performance problems.
+ * [http://jeremy.zawodny.com/mysql/mytop/ mytop]: like 'top' for MySQL: shows running queries.
+ * [http://www.phpmyadmin.net/ phpMyAdmin]: general-purpose web interface to MySQL
+
+=== Examining shared memory ===
+
+The command
+{{{
+bin/show_shmem
+}}}
+will print a textual summary of the contents of the shared-memory structure
+that caches jobs and information about applications.
+
+== Trouble-shooting the job pipeline ==
+
+ * Are workunits (jobs) getting created correctly?
+   Examine the database to see.
+   If you're using a work generator, check its log file.
+ * Are results (job instances) getting created?
+   Examine the database to see.
+   If you don't see results, check the transitioner log file.
+ * Are jobs getting into shared memory?
+   Use show_shmem (see above).
+   You should see jobs.
+   If not, check the feeder log file.
+ * Is the scheduler sending jobs? If not, check its log file, preferably with the following log flags:
+  * <debug_version/>: show details of app version selection
+  * <debug_send/>: show details of job assignment
+  * <debug_quota/>: show details of quota enforcement
+ * Are clients processing jobs correctly?
+   Check the status and stderr output of completed jobs.
+ * Are output files getting uploaded?
+   Check the file upload handler log file.
+ * Are jobs getting validated?
+   Check the validator log file.
+ * Are jobs getting assimilated?
+   Check the assimilator log file.
 
 == Debugging the scheduler ==
 
-Start by setting the appropriate logging options and examining the log file.
-
-If this doesn't reveal the problem, or if the scheduler is crashing,
+If the scheduler is acting incorrectly or crashing,
+and you like mucking around in C++ source code,
 you can run it under a debugger like `gdb`.
 The scheduler is a CGI program;
@@ -41 +89 @@
 }}}
 
-This is useful for figuring out why your project is generating 'no work available' messages.
-
 As an alternative to this, edit `sched/handle_request.cpp`,
 and put a call to `debug_sched("debug_sched");` just before `sreply.write(fout, sreq);`.
@@ -55 +101 @@
 #define DUMP_CORE_ON_SEGV 1
 }}}
-
-== MySQL interfaces ==
-
-You should become familiar with MySQL tools such as
- * The [http://dev.mysql.com/doc/refman/5.0/en/mysql.html mysql interpreter] and in particular the '[http://dev.mysql.com/doc/refman/5.0/en/show-processlist.html show processlist;]' query.
- * [http://jeremy.zawodny.com/mysql/mytop/ mytop]: like 'top' for MySQL
- * [http://www.phpmyadmin.net/ phpMyAdmin]: general-purpose web interface to MySQL
-
-== Database query tracing ==
-
-If you run server components with '''-d 4''',
-their database queries will be logged.
-This is verbose but extremely useful for tracking down database-level problems.