Thanks for your helpful comments and suggestions. I’ve worked out what was happening, and it appears to be a bug in RootsMagic.
I was convinced it was much more likely to be software related rather than hardware, given how the behaviour was so consistent on several independent computers with different hardware, operating system versions and user privileges. This is how I sorted it, for the benefit of any other users who come across similar problems:
Firstly, I modified my source GEDCOM file to contain only details of individuals, with no other records included. Importing this into two separate databases crashed the compare process when compared against each other in either direction. This led me to suspect it was something in the Individuals data.
I manually edited the GEDCOM file to give me separate files for the first 50% and last 50% of the records and repeated the import process with each of these files separately. One pair worked fine, but the other crashed again with the same error.
Repeating this chopping on the “bad” GEDCOM file enabled me to iterate towards a block of about a dozen records that crashed the compare. This was small enough to examine the GEDCOM extract line by line, and I noticed that one individual did not have a name. This was correct, as a marriage record gave the groom as a widower, but I had no information regarding his first marriage. Family Historian, where my data originated, permits this addition of an unknown individual (the first wife) without having to define a dummy placeholder name.
Adding a dummy name to the original GEDCOM file (1 NAME _Unknown) enabled the comparison to run to completion once I had imported the file, so it was the lack of a name that caused the routine to crash, presumably because there was no error trap built into the code for this situation.
I checked RootsMagic carefully after inputting the “faulty” GEDCOM, and it appears that the program imported the record but did not allow me to correct it after import. No error was flagged, and the database integrity check showed no error. There was no unnamed person shown in the Index panel. The groom was correctly shown in his Family tab as having two spouses, but when clicking the “2” button, only the named second spouse was shown. The program therefore seemed to be aware of the unnamed person, but there did not appear to be any direct way to display her details for later editing (for example, when I find out who she was!), or to delete the entry (I did find a work-around, by adding a dummy child, which makes the spouse visible, editing the spouse, then deleting the non-existent child, but that’s horribly clunky).
This behaviour within RootsMagic is inconsistent. It seems to insist on having a name everywhere else, but accepts a GEDCOM input without one. If naming individuals is fundamental to how the data are stored, it should flag this when the original import is made. It should also be able to trap the error when comparing databases without crashing.
There were also some oddities in the reported comparison output, with individuals not showing as equivalent even when a lot of facts were recorded, but I can live with a few false positives as long as it reliably finds those that really are different.