Serious sample_bitwise_validator issues with binary files

[Nicolas]

sample_bitwise_validator works by calculating MD5 hashes of the files to be compared, then comparing the hashes, not the file data itself.

To calculate the hash, it loads the whole file in memory, calls md5_string on the data, then deletes the data.

The read_file_string function it uses to read the file needs file_size*2 RAM and truncates the data at the first null byte. See #886 for that problem.

The result is that if two files are overall different, but identical in the data before first null byte, the validator will say they match. I tested, for example, that it considers all object files (.o) and all linked binaries of the server code as identical. I tested it on six zip archives with completely different contents, and all except one were marked as valid to each other, which is perhaps more worrying (because projects actually use those).

While fixing #886 will also fix the binary file issue here, it will still load the whole file in memory. I attach a patch that uses md5_file instead, making it have O(1) memory usage, and fixing this problem independently of #886.

[Nicolas]

Test run of bitwise validator (before this fix), comparing five zip files. OK means files matched.

[Nicolas]

Patch available on ​http://bitbucket.org/nicolas17/better-bitwise-validator/changeset/9af68ac219c2/raw/better-bitwise-validator-9af68ac219c2.diff

[davea]

(In [17966]) - sample bitwise validator: make it work for binary files

fixes #886, #887