I'm afraid you have discovered one of the limitations of the Mac version of RM. From a technical perspective, the problem stems from the inability of the stripped down version of Internet Explorer that is packaged with RM for Mac to handle CHM files. There are a number of free applications in the Mac App Store that can handle such files, and I will append some additional suggestions later today.
The program that I use is called CHM Reader and it is available in the Mac App Store - and is free (though there are enhanced options for a charge).
Use that to open the RM help file which is called RootsMagic.chm. The challenge is to find that file. What I have done is find the file once and copy it to a folder within my Documents folder. Note that you cannot access the file using the standard Open process within CHM Reader without copying it to somewhere accessible.
To find the file, look for RootsMagic.app in your Application folder. Right click it and then select Show Package Contents. This will display a Contents folder. Expand this and follow down the hierachy Contents/SharedSupport/rootsmagic/support/RootsMagic/drive_c/Program Files/RootsMagic 7, expanding each level in turn, and in that last folder you will find the file RootsMagic.chm. Copy this to somewhere easier to find.
Although I am not sure, the downside could be that you lose the automatic positioning into the Help that RM could (? does) provide by knowing from where you initiate Help.
I hope this helps.