The error message "Common/ChunkCompression.cpp:50: Error: unsupported compressor 8" does appear to relate to a High Sierra bug concerning both the new AppleFS File System that is a default in High Sierra, and 32 bit applications, of which Roots Magic is one example.
As in, every time I start Roots Magic that error appears in system.log as an error in the Rootsmagic application. Here's one example:
Nov 25 19:07:52 NewiMac RootsMagic: /BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.1.1/Common/ChunkCompression.cpp:50: Error: unsupported compressor 8
I have a very drastic workaround for this.
I uninstalled High Sierra, reformatted the disk as HFS+ and reinstalled High Sierra without upgrading the disk to the new filesystem.
Its not great, but it worked.