Jump to content


Photo

Roots Magic 7 for MAC GEDCOM Generator

GEDCOM Speed

  • Please log in to reply
23 replies to this topic

#1 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 28 March 2016 - 09:48 AM

I am trying to assess Roots Magic 7 for MAC for use by my family. I have a very capable MAC Mini, and the import worked fine. I am concerned about data portability, and decided to test your GEDCOM generator. I started an export yesterday between 5 and 6 in the evening, and the program had done ~ 3500 people by 9 AM this morning. As this is < 5 % of the people in the tree, I calculate at this rate that the export of the people alone will take ~ 29 days. 

 

I have a hard time accepting that this could be the case, so I thought I would see if anyone could suggest something I might be doing wrong, or whether or not anyone else has had the same experience. I, too, am hopeful that some time in the future we can all be less concerned about data portability, but this family is not there yet and this result is not reassuring.

 

2.6 GHz Quad Core Hyperthreaded Late 2012 MAC Mini

16 GB RAM

1 TB Fusion Drive (integrated 128 GB SD data cache)

 

Between Wine and Roots Magic, it keeps 1 core's worth fairly busy. My .rmgc file - just loaded yesterday from a .ftmb - is 595.5 MB.

 

Any suggestions for what I might do to speed this up would be appreciated.

 

I could fire up a VM if someone could tell me that the Windows version is a lot faster, but the improvement would have to be exponential to be worth the trouble.

 

Thanks!

 

Steve



#2 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8490 posts

Posted 28 March 2016 - 10:27 AM

Restart your Mac computer. Then go to File>Database Tools in RootsMagic and run all options there in consecutive order, before generating the GEDCOM again


Renee
RootsMagic

#3 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 28 March 2016 - 12:09 PM

Renee, thanks very much for the quick feedback.

 

MAC restarted. DB tested, reindexed, de-phantomed and compacted. File (.rmgc) now 592.9 MB (vs. prior 595.5 MB).

 

I calculated out (better than gut feel - 3500/15 hours/60 minutes ~ ) 3.88 people/minute before. Now I am at 130 people and ~ 25:30 minutes, which is in a similar range just over 5 people/minute still and seems fairly steady given the count increment for the software is 10.

 

Any other suggestions? I can give it more time, but it doesn't look to be speeding up. Indexes is what I would have suspected as well. 

 

May I ask what you expect in terms of GEDCOM generation? Is my situation unusual? 

 

Thanks again!

 

Steve



#4 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8490 posts

Posted 28 March 2016 - 12:15 PM

No that's not a usual situation at all. Try doing a drag n drop of your database into a new blank one. It uses GEDCOM in the background so it might have the same issue. If it will create a new database then see if GEDCOM is any faster.


Renee
RootsMagic

#5 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 28 March 2016 - 12:25 PM

Hmmmm. Okay. I'll give that a try when I'm sure I understand what you are saying. Can you give me a step-by-step - at least at a bit lower level. I'm experienced, but not with RM7. 

 

Also, is there any way to stop the GEDCOM generator other than a force quit? There is no cancel button, and the red close window button on the dialog box is not responsive.



#6 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 28 March 2016 - 06:11 PM

I poked around a bit and found the drag and drop video for RM 6 and it got me started. Unfortunately, as you suspected it appears to be copying at just about the rate the GEDCOM was generating... Thanks again for trying, and any further help will be appreciated.



#7 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8490 posts

Posted 29 March 2016 - 08:41 AM

Open a support ticket and send a backup of your database for testing. If the file is to large to attach to the ticket then we can send you a link to upload through.

 

http://support.roots...us_requests/new


Renee
RootsMagic

#8 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 10:20 AM

I'll do that shortly. I hope I didn't cause too much aggravation. That certainly isn't my goal.



#9 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 11:36 AM

How can I tell if the upload failed? It says it is still uploading, but it's been at it for 20-30 minutes already. Network activity on my system is continuing, but it fairly sporadic.



#10 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 12:06 PM

I have to get out and around a bit. It appears that I need a "link to upload through," although your web site still reports it is uploading. My .rmgb file is 133.7 MB.



#11 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3606 posts

Posted 29 March 2016 - 02:05 PM

I'm a Windows user, not a Mac user. My rmgb file is about 18MB with 60,000 people in my database, so 133.7MB sounds pretty big for an rmgb file. How many people in your database?

 

Jerry



#12 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8490 posts

Posted 29 March 2016 - 02:45 PM

Open the support ticket and then request the link for upload. Though if processing this on your computer is having a hard time uploading it does sound more like a computer issue. They both might be related.


Renee
RootsMagic

#13 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 05:19 PM

Hi, Jerry - thanks for the feedback. Right now I am trying a new import. I THINK I imported an .ftmb file last time - I plan to repeat that, but this time I'm importing the .ftm file - hoping against hope that I'll find a difference in the export time. The import is at about 45 minutes and reports being 20 % done - I take that back - it's now sitting on a blank grey dialog box - which if memory serves indicated it was just about finished. My wife's .ftm file weighs in at 3.38 GB per finder. Some people do short/fat trees. Some do tall skinny trees. My wife is a careful documenter and she has a tall/fat tree. Database properties from RM7 are: Created: 3/29/2016, 4:23:58 PM; Modified: 3/29/2016, 5:36:14 PM (likely import start/stop time so duration ~ 1:13 hours); People: 124,543; Families: 47,034; Events: 455,329; Alternate names: 5041; Places: 63,135; Sources: 2288; Citations: 863,055; Repositories: 26; Multimedia Items: 113,105; Multimedia Links: 695,455 for a total of 2,369,011 records - assuming database records is indeed what this is summarizing. The tree is meticulously researched and documented and she's spent 10's of thousands of hours on it. A fresh TreeSync from Ancestry - where the tree originates - onto FTM3 for MAC - takes more than 1/2 day, and the media takes well over a week to download after the sync completes. The total space consumed is well over 150 GB, although most of that is likely irrelevant to this discussion.

 

Can you tell me approximately how long it takes you to export a GEDCOM for your tree? During the export process on my MAC, I did notice the wine/crossover software Roots Magic uses to "MAC-ify" their app consumes up to 1/3 of the core RM7 uses, but that's not nearly enough to explain the glacial pace of the process for this tree though. SOMETHING is wrong SOMEWHERE. :-) I'm thinking it's the sheer volume of records, but the theory has been slow to gain traction with the current "audience."

 

Steve



#14 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 05:54 PM

Renee, thanks for sticking with me. I've performed a standard disk check with no problems reported. I also decided to perform the Apple prescribed "Apple Hardware Test, Version 3A243" short version, which reported "No Trouble Found" after 4:41 Minutes of testing. There is only one indication that there MIGHT be something wrong with my highly capable system - and that is that my newly imported tree does not export as expected using RM7. I'm not asking you to read too much into this, but I am a computer engineer, and I have been developing SQL database applications for many years now - on my MAC Mini and my MacBook and prior systems. Again, the above is worth as much as the time it took to type it, but I am only a newbie on RM7. I have an open mind to suggestions as to what might be wrong, but "it does sound more like a computer issue" isn't all that helpful. It sounds to ME like I experienced a problem with your web site. How big a file is it able to handle? If you have a MAC expert who is able to offer suggestions for specific problems, I'd love to hear about it. I am a pretty decent MAC/Windows/Linux expert, however, and my system looks pretty solid to me.

 

I just reimported - as you can see by my last post in response to Jerry. There are some stats in that post that you might want to have a careful look at. The import took ~ 1:13 hours and the export is at 120 people and counting, but it moves by 10 people every 20 seconds or so - just like it did before.

 

I know I probably sound snarky and impatient - actually I am. I have spent the entire day discussing this in the RM7 Facebook User Group, and most of the time has been spent helping me figure out what's wrong with my system and I have had to patiently try this and that, etc. I will be happy to follow any reasonable suggestion - and I mean it. But I won't be bashful about telling you if I think it's unreasonable or unhelpful. Your CEO even piped in to say that it has to be my computer, that the number of records is irrelevant to your software, and there is no way my file size is big enough to be a problem. This is unfortunate and I wish he hadn't taken such a stance. My experience with relational database application development would suggest there is a fair chance he's wrong. I hope we can get to the point where we can figure out the answer. I just want to use the software and have data portability. I've taken about 30 minutes to write this and the export is up to 260 people now. :-/

 

I honestly do thank you for your support. I'm not an enemy. I am not a "plant." I AM a very serious user of Family History software.



#15 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3606 posts

Posted 29 March 2016 - 07:45 PM

Can you tell me approximately how long it takes you to export a GEDCOM for your tree? 

 

About 2 minutes. My machine is a four year old Dell Inspiron laptop running 64 bit Windows 7 Home Premium (but RM is a 32 bit program), 8GB memory, two processors with two cores each (but i think an RM's export only uses one of the cores).

 

Jerry

 

P.S. I'm just exporting a GEDCOM to disk. It's not going anywhere on the Internet. Are you doing something that sends your GEDCOM to the Internet?



#16 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 10:23 PM

I'm just exporting a GEDCOM to disk. It's not going anywhere on the Internet. Are you doing something that sends your GEDCOM to the Internet?

The result is going straight to disk.

 

Interesting on the 32 bit - I had that column excluded in my activity monitor. Mostly 32 bit here as well.

 

I would actually love it if someone could point me to a problem where my system disk writes could be sped up by several orders of magnitude, but I don't even consider it worth talking about. I am otherwise quite happy (might I say amazed) by the system performance I get from these systems. I decided to load up my Windows 10 VM, and load Roots Magic there as well. I have an "old" 110,000 person version of the tree there from FTM 2014 that loaded up in ~ 20 minutes vs. 1 hour and 13 minutes for the 145,000 person tree on my MAC. I started the export at 7:37 PM just after the import finished and it stands at 3960 people while the MAC export I also have running for significantly longer is at 1840 people. On Windows, that's ~153 minutes for now and 4000 people and change =~ 26.14 people/minute. This is well over a 5X speedup. The wine crossover processing for MAC-ifying is consuming at 32 % of a core, and RM7 is consuming ~ 70%. With Unix, that's about as close as it will keep things to 100 % in my observation. Total CPU usage for my system - including RM7 on OS X exporting it's heart out, RM7 on a Fusion Windows 10 Virtual Machine exporting it's heart out, an Ubuntu Linux Fusion VM running PostgreSQL, and an assortment of other processes including a development web server... You get the idea. I will say my fan doesn't usually run but I can hear it on low cooling the CPU a tiny bit running at a 27-33 % load with no bottlenecks.

 

On Windows 10, RM7 is using 54.3 % of available CPU "space," and the disk is swapping between 0MB/s and 0.1MB/s - oh, and 8 MB of memory. I am sure there are other explanations for this behavior, but the way I would likely read it until proven otherwise is that the DB can't keep up with the CPU. Overall the disk is toggling between 0% and 1%, Memory at 54 % and CPU at 56%. I just checked, and my windows instance has 2 cores and 4 GB of RAM allocated. By the way, I'm not seeing where it is favoring one CPU over the other - they seem to be operating just about in lockstep. I don't see where Resource Monitor talks about 32 bit vs. 64 bit, but I think this is probably something that can be ignored. Without the wine crossover processing, Windows is giving it quite a bit more processing - they are the same processors. If the application is somehow limiting to roughly 1 CPU worth of processor cores, then this could be an alternate explanation I guess. I think one of the things I noticed when I converted FTM 2014 over to Windows 10 was how much better 10 handled multiprocessing, but I don't have data. What I do know is that I went from almost never being able to SYNC this tree to often being able to SYNC this tree - it was a highly significant improvement. FTM3 for MAC SYNCs pretty much every time, but if I wait too long and my wife adds too many records it can get confused enough where I judge it's time to start from scratch - which is VERY painful but pretty reliable.

 

If you are hearing that this has been a bit of a problem tree, you would be hearing right. BUT, NO ONE has ever said there is a problem with it other than it is "too large" to be handled gracefully between FTM and Ancestry. I'll give it to Ancestry that they didn't boot us out, but we did get pretty frustrated with each other at times. It got pretty tiresome to start every interaction by reaching out to a first level phone support person every time and justify taking up their time. I think I've finally figured out that the way to avoid some of this is a well thought out email to support@ancestry.com, but even then one or the other of us would have to jump through hoops because our problems were almost never the easy problems.

 

I have a few smaller/simpler trees, so I imagine I'll import and export one of those just to show that it's NOT my system conclusively and then we can stop the largely useless discussion (and I greatly appreciate your part in it!) and get back to what is actually happening.

 

It might be interesting for the RM7 folks if you could also summarize your record counts like I did so that they can use your tree as a data point if they need to. If your tree is - say - 500,000 "records", then dealing with the linkages between those vs. dealing with the linkages between 2,400,000 "records" can be huge (probable exponential relationships which are a possible explanation for what I am seeing) depending upon where you run your software relative to inflection points. It is always a balance and unless you really know what you are doing - which I am still learning - then things can pop up and you have to work hard to beat them back down (read speed things up). RM7 import speed - acceptable. Responsiveness after import - excellent so far but I've spent most of the time I could be exploring defending myself. Export - disaster for this tree, and so far it appears it is only this tree. Like I said, I hope someone tells me how I can get a 10,000X improvement in disk write speed. That would actually be dramatically better for me than being right about database thrashing (I don't believe in that kind of magic though).



#17 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 29 March 2016 - 11:13 PM

I dug up 2 smaller trees to illustrate what my system is capable of doing under more favorable circumstances. The first was 368 people, and I didn't have time to push the start button on my stop watch for that export (I admit I wasn't ready for it, so it may have been 2 seconds or so). I then loaded up a tree with 6712 people, and this export took 27.88-ish seconds. If I calculate that out, it comes to 14,445 people/second. Can we stop talking about the capabilities of my system as yet? I'd be happy to screen share and demo if someone remains unconvinced.

 

Steve

 

P.s. I tried one more time to upload my big file, but it wouldn't go so I opened a ticket and requested a link per instructions.

 

p.p.s. Folks, if it indeed turns out to be a bottleneck in your system, why is that a bad thing? This appears to be an extraordinary tree - we've always thought it might be, but it's hard to get such data to know for sure. If it is, you were going to hit it at some point anyway. Make it a celebration. Offer a bounty for people who break your software. I imagine Apple is thinking seriously about the bounty approach for privacy issues right now.



#18 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3606 posts

Posted 30 March 2016 - 05:41 AM

People: 56109
Families: 19445
Events: 108609
Alternate Names: 72
Places: 7743
Sources: 1943
Citations: 92913
Repositories: 7
To-do tasks: 239
Research logs: 24
Multimedia items: 917
Multimedia links: 1329
Addresses: 44
Correspondence: 0
 
I running at about 509 people per second.  Actually, it's higher than that on the "person" part of the export. The 509 figure is an aggregate based on the total number of people and the time to export everything - people, families, alternate names, etc.
 
Jerry


#19 zhangrau

zhangrau

    Advanced Member

  • Members
  • PipPipPip
  • 1520 posts

Posted 30 March 2016 - 06:21 AM

-- I'm running RM 7.1.0.7in Windows 7 with 12 GB ram and 177 GB free on my data drive (E:)
-- File - Properties reports 385,586 people and 2,175,605 citations (that's just 2 of the statistics)
-- The RMGC file is 1.08 GB, the most recent RMGB file is 222 MB
-- I started a full GEDCOM export to "test_05.ged" at 1:25 am
-- The GEDCOM completed at 2:58 am (just over 1-1/2 hours) and is 681 MB
-- 385,586 / 90 minutes = approx. 69 indiv./second
-
rB3wDBP.png


#20 StephenMWatts

StephenMWatts

    Member

  • Members
  • PipPip
  • 27 posts

Posted 30 March 2016 - 09:02 AM

Thanks, Zangrau and Jerry. Data!!!! I let my Windows 10 instance run overnight, and I just calculated out 27.5 people/minute and not even 1/4 of the way through the people as yet. It will be fascinating to see what causes your 2 trees - which on the surface bracket the tree I'm working with in terms of complexity - to run at a higher rate while also being so large. I'm in the process of submitting this tree to Renee to see how it goes in the RM shop. Either something wrong with the tree, or something structurally significant to the software...? I'd still love to hear that there is something wrong with my computer that will allow it function several orders of magnitude faster... I'm voting something structurally significant, but Zangrau has put an interesting twist on the discussion I must say(!).

 

My software reports Version 7.0.1 (13.0.1.27705local) on MAC and 7.1.0.7 (14Mar2016) on Windows 10.