db_dump: Create md5 or other checksum of generated files

[Major Tom MIB]

db_dump should produce a checksum listing which can be used to validate the files after they have been downloaded to remote sites. Primary use by mirror sites, secondary by stats collection processes.

[davea]

Why is this needed? TCP is a reliable protocol, and wget returns an error code on failure.

[Nicolas]

Some servers may be configured to send a Content-MD5 header. If your mirror or stats site wants checksums, make sure you're parsing that header. Note that it's the MD5 of the message sent by HTTP. For example, if a download is resumed, the response header will have the hash of the requested range, not of the whole file.

There is also a proposed standard for an HTTP server to provide whole-file checksums, regardless of the range or content-coding. See ​RFC 3230 Instance Digests in HTTP. If your mirror or stats site wants checksums, send the Want-Digest header. I don't think any server currently implements it, but it's a good idea for your client to send it anyway. Otherwise server developers will stay in the usual chicken-and-egg problem of "no client sends that header, so why should be implement it?"