The XCOMP/2 home page

What is XCOMP/2

Welcome to the home of XCOMP/2, a recursive file compare utility. XCOMP/2 is for comparing files what is XCOPY is for copying files, something I was definitely missing in the OS/2 product.

One note about the line containing the Throughput measurements. In that example I was comparing some Lotus Notes databases on a backup CD-RW with the orignal on a harddisk. You can see that the Source drive, a CD-ROM drive, delivers a not too high throughput (it seems that that this 8-16 speed CD-ROM drive isn't too comfortable reading CD-RWs), while the Target drive, a Ultra-Wide SCSI drive delivers excellent performance (ok, actually it's the HPFS386 cache ;-) giving an acceptable Total performance.

The performance given for Source and Destination is just the raw performance of the OS/2 DosRead() API without accounting for overhead like doing the file searched and the comparism itself. The Total performance does include all overhead, that is the timer is started when the first file is searched and stopped when all files have been compared. Thus, don't take the performance specified to serious!

If the Source and Destination drives are different physical drives you may reduce the run-time by specifying the option /MP. Using that option causes 2 threads to be used for reading the Source and Destination files simultaneously instead of doing that sequentially with a single thread.
This option is most useful when reading from 2 drives of similar speed e.g. a CD-ROM and a CD-RW drive (because the 2 slow accesses will be done in parallel instead of sequential), it's less useful when having drives with a great difference in speed (because reading the fast drive takes almost no time compared to reading the slow drive, thus the slow drive alone affects the overall performance).

Use the option /!C if you do not want the comparism to stop waiting for the user pressing a key for every miscompare. This option may only be useful when you use the /LOG option or redirect the output into a file, as otherwise the lines stating the miscompare scroll away too fast.

Use the option /!F if you do not want the comparism to stop whenever a file in the source location can not be found in the target location. This option may only be useful when you use the /LOG option or redirect the output into a file, as otherwise the lines stating the miscompare scroll away too fast.

You may also notice that you may have a different buffer size. This is normal, because XComp/2 takes the amount of physical memory into account to be efficient on one hand and to limit swapping on the other hand. Using the option /TINY XComp/2 will use 2 64kB buffers instead of a percentage of the RAM installed. This might be useful if you are running a memory constraint system.

Finally, to ensure that you can notice the progress, a percentage indicator will be displayed while comparing files that are larger than the given buffer size.

If you just want to see if all files exists without actually comparing them (thus saving much time for the price of possibly overlooking corrupt data), then invoke XComp/2 with the /SIMULATE option.

When comparing using CD, CD-R or CR-RW media, you may get alarmed by miscompares which disappear when running the same (or limited to a subdirectory only) comparism again. I'm not sure what causes that, but I suspect that misreads of the drive (e.g. weak bits) occured. Reading the same files again may just cause different read results due to the drive's head repositioning in a different way.

Examples

As with V2.00 the interpretation of the commandline arguments regarding the source and target paths became more intelligent, here are some examples on the usage (we are located at S:\Orig and have XCOPY'd all files from there into T:\Copy):

Command	Resulting comparism from XCOMP/2
XCOMP S:\Orig\* T:\Copy	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size The most trivial example using 2 fully qualified paths. Specifying the qualified filename with "*" could of course be replaced with one actually wants to compare, e.g. "*.ba*" or "????.ba?"
XCOMP S:\Orig\ T:\Copy	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size If you do not qualify the filenames to compare for the source, by default "*" is assumed.
XCOMP * T:\Copy	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size The source "*" is expanded to a fully qualified path by querying the current directory.
XCOMP . T:	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size If you specify a relative path like ".", it is expanded to a fully qualified path. As no qualified filename is specified too, "*" is assumed. Using ".*" instead would have avoided that assumption. As no directory is specified for the target path, the current directory on that drive is assumed. Having specified "T:." would have led to the same result.
XCOMP S: T:	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size As you didn't specify a path, the current directories "S:\Orig" and "T:\Copy" will be taken automatically. As no qualified filename is specified too, "*" is assumed.
XCOMP ..\Orig\ T:..\Copy	Comparing files qualified by * at S:\Orig\ with T:\Copy\ using 2097152 bytes buffer size The relative path "..\Orig\" is expanded to "S:\Orig\" and as no qualified filenames are specified "*" is assumed. The same applies to "T:..\Copy" which gets expanded to "T:\Copy\*".
XCOMP \\pka1002s\Orig\ \\pka1002d\Copy	Comparing files qualified by * at \\pka1002s\Orig\ with \\pka1002d\Copy\ using 2097152 bytes buffer size For UNC-Names, things are treated the same way, e.g. as no qualified filename is available here, "*" is assumed.

If you found any inconsistency please tell me (it's not trivial to guess what the user may have meant when not specifying fully qualified paths)!

Verifying burned CD-R's and CD-RW's

As said, my primary motivation to write XCOMP/2 was to compare what has been burned onto a CD-R or CD-RW with the original data, as I do not trust write operations that have no explicit verification, e.g even if it takes some time I do verify the data when copying something to a floppy disk media.

And from my personal experience I'm right being sceptical here. I already have encountered a few CD-Rs and CD-RWs that contained currupted data. I have seen effects that best can be described as weak bits, which used to be a copy-protection mechanism for games on floppy disks. Weak bits are bits read from a media sometimes incorrectly and sometimes correctly, and that rate may even be different between different CD-ROM drives.

The technical reasons seem to be (and looking around in the Internet confirms that):

• CD-Rs and CD-RWs just contain large amounts of data, so even if the probability of a single bit failing is very low, the high number of bits on a CD may cause some faults.
• CDs were initially developed to contain audio data where it does not make much difference if a few bits are read incorrectly, the human ear would not notice the difference. For data CDs even a single bit error will cause a problem, even if much more capacity on a CD is spent for error detection and correction.
• CDs were initially rated as single speed, today's recorders can burn a the tenfold or higher speed. That of course decreases the time to burn a single bit accordingly, that is the time to position the laser and turn the laser power on to heat up the substrate gets shorter and shorter. It is easy to imagine that this can increase the failure rate, and even if the CD-R or CD-RW is rated for a high speed you may get better results buring with a lower speed.
• Different CD-ROM drives have different characteristics and may produce different result, or may even be incompatible with some burners. Combine that with increasing writing speeds, and again we have the problem that the time to read a bit gets shorter and shorter.
• When using CD-RWs instead of CD-Rs the failure rate may increase because of the different design of such medias, especially regarding the reflection. Combine that with the fact that different materials require the writing laser to operate at different power levels, and you can just hope that the media correctly tells the burner which power level to use.
• CD-ROMs initially were specified to contain 74 minutes or 650 MBs of data, to increase the capacity the limits of those specifications were reached (e.g. for 80 minutes or 700 MB CDs) or even exceeded (e.g. for 90 minutes or 800 MB CDs).
• Last but not least CD-Rs and CD-RWs are getting cheaper and cheaper, and from that cost pressure it's not hard to imagine that this will put some pressure on manufacturing for being as cheap as possible.
• At least for CD-Rs it seems that the important reflection layer that is fundamental for the CD-ROM drive being able to read the CD is the same layer that covers the top side of a CD (the side where one writes the CD title onto). If that layer is scratched even slightly, you have some worthless plastic waste in your hands (and in contrast to the bottom layer that one can't be repaired as far as I know).

What's new

The following items have changed between V2.50 and its predecessor 2.40:

• Added a workaround for a possible TVFS bug (TVFS does sometimes return files during enumeration of all directories within a (sub)directory, may be related to the mix of TVLINKing files and directories mixed).
• Added the option /SIMULATE which does all XCOMP/2 would do, but does not actually compare the files (that is, XCOMP/2 recurses into all subdirectories, opens all files to check for their existance, checks their sizes and closes them again without acutally comparing their contents). This may be useful to ensure that all files exist, can be opened and are of the same size, neglecting any possible differences. Of course, this is much faster than acutally comparing the files.
• Added that XComp/2 will pause if the source file can't be opened due to being locked by a process (e.g. SWAPPER.DAT), unless option /!F is specified.
• Added a port to the WIN32 environment (as I wanted to confirm that OS/2 is the superior platform also for an development environment ;-).
• Fixed that error message processing sometimes contained redundant code, removed redundancy.

The following items have changed between V2.40 and its predecessor 2.30:

• Added the options /!C (which relaces the option /NOPAUSE), and /!F
• Added the option /LOG[:[drive:\|\\server\][path\]filename] to allow XCOMP/2 to write a logfile so that you can run it unattended while still ensuring that you will not miss any problem. If you specify a filename with that option, XCOMP/2 will log all problems into that file, otherwise it will log them into the file specified by the XCOMP environment variable or if that is not available into the file XComp.Log in the directory XComp.Exe was launched from.
• Added the option /LINE to enhance problem determination. You likely will only need that option during problem determination.
• Changed that XCOMP/2 will now by default pause not only when a compared files does not match but also when a file available at the source location has no equivalent in the target location. Thanks to Timur Tabi for that suggestion.
• Fixed that using "." in the source location when the current directory is the root directory failed with an error message.

• Added that XCOMP/2 will beep when it has terminated, this will alert you when XCOMP/2 is running in the background.
• Added that XCOMP/2 will now display the directories it traverses into in addition to the file it compares in those directories.
• Fixed that if XCOMP/2 gets an error accessing a directory to compare files or to find subdirectories within (e.g. due to access right or file system problems), it will now display the directory in error and either continue or terminate (depending on the type of the error). Thanks to Wilbur E. Hong for that suggestion!

• Fixed that if an error occurs during opening of the files no filename was displayed but just the error message preventing the user to see which file(s) failed.

• Added documentation for the /TINY commandline option which uses 2 64kB buffers for the file comparisms.
• Added to display the retry counter when retrying a comparism due to a miscompare.

• Fixed a severe bug that caused to ignore miscompares within files of the same length (was introduced when I added a retry count to handle weak bit errors when reading CD-Rs in older CD-ROM drives).
• Fixed that the executable contains the correct BLDLEVEL signature.
• Added a message showing the total files compared.

• The commandline option /NOPAUSE has been added to prevent the default behaviour of V2.00 to stop further comparism when an error is detected waiting for the user to press a key. This option may be useful if you log XCOMP's output into a file (using file redirection).
  Up to V1.50 the default option was not to stop when an error was detected, which was modelled after XCOPY's behaviour, but proofed to be a bad idea when comparing much files (e.g. CD-R's or CD-RW's) as the non matching files easily scrolled away.
• Checking of some return codes has been improved.
• Added a consistency check to be sure the correct parts of the files are compared.
• The major improvement is a more intelligent interpretation of the commandline arguments regarding the source and target paths. Please see the examples for more details.
• Fixed a race condition when using the option /MP that caused to show miscompares when there were none.

XCOMP/2 download

You are welcome to download XCOMP/2 V2.50 (inclusive its source written with VisualAge C++) from this site.

XCOMP/2 source code

XComp/2 includes the complete source code to recompile it. Just run Protect xcomp2v250 Source.zie to obtain the source code archive Source.zip out of the encrypted file Source.zie (in case you wonder why I'm not using e.g. password protected ZIP-files, well by the required proprietary decryptor only for the OS/2 platform, all people interested in the source need to obtain an OS/2 machine!). Then just unzip Source.zip, preferably at the path you want to compile XComp/2 from.

To compile XComp/2 for OS/2 you need:

• IBM VisualAge C++ V3.0 CSD8 for OS/2 (or equivalent)
• IBM OS/2 Warp Toolkit V4.5 (or equivalent)
• just type NMAKE ["NODEBUG=1"]

To compile XComp/2 for WIN32 you need:

• IBM VisualAge C++ V3.6 FP2 for Windows NT (or equivalent)
• MS SDK included in IBM VisualAge C++ V3.6 (or equivalent) just type cd WIN32 followed by NMAKE ["NODEBUG=1"]