Changes between Version 3 and Version 4 of VolunteerDataArchival

Timestamp:

Nov 23, 2011, 3:06:22 PM (13 years ago)

Author:

davea

Comment:

--

VolunteerDataArchival

@@ -70 +70 @@
 unreliable resources:
 
-'''Replication''': a file is divided into N pieces,
-and each piece is stored on M hosts.
+=== Replication ===
+
+With this technique, a file is divided into N chunks,
+and each chunk is stored on M hosts.
 If a replica is lost, and there another replica,
 that replica is uploaded to the server, then downloaded to another host.
+By increasing M, reliability can be made arbitrarily high.
 
-'''Coding''': with Reed-Solomon coding, a file is divided into N 'packets',
+Replication has advantages:
+
+ * Recovery from a failure is fast, since only one upload and download is done.
+  This minimizes the chances of another failure occurring during recovery.
+ * By making N large, the server storage needed for a recovery
+  can be made arbitrarily small.
+
+and disadvantages:
+
+ * It has an extremely high space overhead,
+  since M in general must be made large to provide reliability.
+ * Even if individual chunks are made reliabile,
+  the failure rate for the file as a whole increases exponentially with N
+
+=== Coding ===
+
+With Reed-Solomon coding, a file is divided into N 'packets',
 and an additional K checksum packets are generated.
 The original data can be reconstructed from any N of these N+K packets.
 
-In
+Coding has advantages:
+
+ * It can provide high reliability without high space overhead.
+   For example, if N=40 and K=20, we can tolerate 20 simultaneous host failures
+   with a space overhead of only 50%;
+   with replication the overhead would be 2000%.
+
+and disadvantages:
+
+ * Regenerating a chunk requires reassembling the entire file on the server,
+  defeating the purpose of distributed storage.
+
+== Hybrid reliability mechanisms ==
+
+Because of the above disadvantages,
+neither replication nor coding alone is sufficient for volunteer data archival.
+However, we can combine them in various ways that reduce the disadvantages.
+
+=== Multi-level coding ===
+
+=== Coding plus replication ===
+
+== The VDAB simulator ==