I am a GIS specialist, part of this is to routinely copy vast quantities of data to disc – backup, archiving and delivery. Occasionally the files on the other disc do not match the source disc, in the order of for every 4TB copied there will be one error in one file (that's 0.000000025%).
Currently I am using RoboCopy or Windows Explorer copy to duplicate the files onto the other discs and WinDiff to verify the contents, which is how I know that there's a very small error rate. This process works but WinDiff is slow, sometimes taking more time to verify than it took to originally copy; RoboCopy does not have a and verify switch like XCopy.
I have previously used XCopy with the /v switch but was advised that that utility was depreciated in Windows 7 and I shouldn't be using it. Individually creating MD5, checksum or hash of files is not an option unless there's an efficient library/method/algorithm that I can implement in a programming language as there would be hundreds of thousands of individual files.
The question is: Is there a method that will verify the files as they are copied or a better verification program than WinDiff, or even better yet a guaranteed way to copy these files with 100% integrity so I don't have to verify them?
I can code for C# / VB.net / C++ / Python / DOS Batch provided the required libraries are free or built in. I'm not scared by command line, I can remember when there was only command line, so DOS utilities/instructions are fine.
So why is this now a problem if I have a working method?
It's not really, I have a working method but have been doing it the same way since Windows NT in the late 90's; in the late 90's it was Exabyte tapes and 100MB or less, then there was Optical media which could be verified in less than half an hour. Now, with media being high capacity and relatively inexpensive, I find that I am copying and verifying up to 16 TB for a single delivery and the verification time is overrunning the delivery date…. I need to at least look for a better way!
Best Answer
The program you've been looking for is xxcopy. I too had this desire to do a binary file compare when copying oh, a few hundred thousand files for forensic analysis. I didn't care it took a little longer, I didn't want to have to create a SFV or MD5 file and then run another program on the files afterward, I wanted something that did the DOS "fc" command on the fly.
http://xxcopy.com/xxtb_027.htm#tag_319
You want switch /V2
Have a good day.