Changes between Version 1 and Version 2 of LocalityNew

Timestamp:

Aug 14, 2012, 2:06:53 PM (12 years ago)

Author:

davea

Comment:

--

LocalityNew

@@ +1 @@
+[[PageOutline]]
 = Locality scheduling redesign =
 
@@ -4 +5 @@
 
  * A '''batch''' consists of a sequence of files F1 ... Fn
-        and a set of jobs J1 ... Jm operating on these files.
+  and a set of jobs J1 ... Jm operating on these files.
  * Each job uses a contiguous set of files.
  * A given file may be used by many jobs.
@@ -74 +75 @@
 workunit (new fields)
         cursor_id integer
+        job_num integer
 
 result (new fields)
@@ -86 +88 @@
  * create batch record
  * based on # of hosts, # of jobs, and job density,
-   create locality_job_range and locality_cursor records
+   create locality_cursor records
 
 === Feeder/scheduler ===
@@ -107 +109 @@
 ==== Assigning jobs ====
 
+For each batch:
+
+Enumerate unsent results for this host's cursor
+in order of increasing job num
+(i.e. finish old jobs before starting new ones).
+Send as many as the host can handle.
+
+Some type of synchronization is needed; maybe
+
+ * use a DB transaction (how much would this lock?)
+ * explicitly lock the locality_cursor record (is this possible?)
+ * use a semaphore per cursor
+
 ==== Deleting files ====
+
+If the host has a file that's used only by finished jobs,
+tell client to delete it.
 
 === Work generator ===
 
-=== Validator ===
+Try to maintain a cushion of N unsent jobs per cursor.
+Start generating jobs at cursor.first_ungenerated_job_num.
 
 === Transitioner ===
 
+When a workunit is completed (successfully or not):
+
+{{{
+if workunit.job_num == cursor.first_unfinished_job_num
+        cursor.first_unfinished_job_num++
+}}}
+
 === locality_daemon ===
+
+This runs periodically (every hour or day) and recomputes
+locality_cursor.expavg_credit.
+It doesn't have to be a separate program;
+it could be added to an existing daemon, like the feeder