Jump to content


Photo

RM4 is SLOW


  • Please log in to reply
72 replies to this topic

#21 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 08 February 2010 - 09:39 PM

If there is a safe way to do it, I'd love to try it. Has anyone tried this? I could backup before I do it

I'm sure that Bruce and Mike have tried it and found it wanting, else why isn't it incorporated? It would be a trivial programming task to include it.

It's difficult for outsiders to experiment with it because we do not have the proprietary RMNOCASE collation sequence required for many of the fields and indexes. Without it, VACUUM should just report an error. I have been able to spoof it out with a different collation but, what are the longer term consequences as tables change and indexes are updated by the internal sequence spec? This different collation may be alright for the unaccented English character set but not for others; I'm reasonably sure that's why RM4 has an extended collation sequence so that it can equalise for sorting/indexing upper and lower cases for a multi-national character set.

I VACUUMed my main database using the substitute collation and the file shrank by 1.1%. Exported both versions to GEDCOM and compared the two GED files - identical except for the 1 FILE line, as desired, so that's encouraging.

This is a small file (<1000 persons) so I'm not seeing speed issues.

If you want to experiment, I can tell you how I did it.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#22 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3420 posts

Posted 09 February 2010 - 05:07 AM

If there is a safe way to do it, I'd love to try it. Has anyone tried this? I could backup before I do it

I'm hoping Bruce & Team are going to find a safe way to do it, there are so many problems within RM at present that are linked to indexing not to mention speed that the repack function really needs to make a come back.

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#23 isis

isis

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 February 2010 - 07:01 AM

I'm hoping Bruce & Team are going to find a safe way to do it, there are so many problems within RM at present that are linked to indexing not to mention speed that the repack function really needs to make a come back.

I strongly suggest its a merge, purge & delete issue in RM4, there was a similar issue with my sql on a phpnuke website, but there was a button in the programme, you just pressed to optimise the database, which presumably deleted , deleted records & re-indexed it to make it run faster

I notice that PAF & RM3 had the pack & rebuild index feature, why was it removed from RM4

I find it a bit annoying that you can remove the space consuming, alternative name view in the sidebar list of names, but you cant remove the alternative names in the duplicate name merge option.

This makes the search for people to merge with, more confusing, because , in my case, I may have merged 8 people with the same name or letter different, but the list of names is showing me all of the names that I have already merged , when I'm just looking for the names that I need to merge with. Ive got a few trees imported a couple of times family members, so it just fills the screen up with names I've already resolved, ie in list of 15 visible names 10 are variations of the name I merged & only 3 are not merged yet. Its better to allow this feature to be switched off in merge session & surely it slows down the indexing, because its indexing the same person in 12 different places, but the merged info is the same

ie made up example
Mary Queen of Scotland b 1037 d1067 ie Scotland, Mary (Queen of)
is merged with

Name list
Scots, Mary queen of 1037 1067 &
Stuart, Mary Queen 1037 1067
Stewart, Mary (Queen) 1036 1067
Queen of Scotland, Mary 1034 1067
Scotland, Marye Queen 1037 1067
England, Mary (Queen of Scots) 1037 1067
etc

After merging , all of the above different methods of listing the same person, are in the list of names to merge with, (like ghosts),which means they also stay in the wrong locations, like "queen of scotland" as surname in imported gedview which shouldnt be then continualy filed there.

Its also disheartening as it makes you think theres more left to merge than there really is, because the database is bloated with loads of copies of the same name. Please allow for viewing these alternative names in the duplicate search & merge to become optional as well. Its enough to be able to see the alternative names in the persons (edit details). If youre comparing people to merge with, your more likely to look in the persons profile, partners , children & parents, then go round RM4 lists of names to see other names for that person as spouses child etc.

I m merging approx 3 versions of royalty databases inc. german & english, french versions going back to vikings & there can sometimes be 10 name versions of the same person, ie Henry , Heindrik , Henri etc but dates, partners, places, children all match but the database just gets littered with names youve already resolved .

I dont have this problem in RM3, you just get 1 merged person with alternative names visible when you click on that person

The merging is slow enough as it is, (up to 2 mins now, to search for a name on the duplicate name list). We dont also need to, without a choice, search the litter of already merged names. I know theres a blue box next to some, but it fills up the screen , unecessarily,when youve done lots of similar name merges & its harder to find your next merge. It must be indexing alternatives to all over the database, where the names originally were which must slow down the search immensly.

Thanks

#24 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3420 posts

Posted 09 February 2010 - 07:14 AM

If I remember correctly, the option to repack the database was deemed to be unnecessary due to the new database structure, in hindsight that opinion may have been a little premature and now need of a rethink.

There are many issues within RM at present where recommended course of action (from RM) is to export the complete gedcom file and import it again into a fresh database (sledgehammer & nuts comes to mind). Anyway that course of action introduces data problems and really only allows RM to create a fresh database with valis/correct indexes, need I say more. <_<

Bring back the repack/rebuild option......

Oh, do be aware that merging individuals can break media linkage in many instances in the current 4.0.7.1 build.

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#25 Romer

Romer

    Advanced Member

  • Members
  • PipPipPip
  • 2069 posts

Posted 09 February 2010 - 02:53 PM

Also thanks to Brenda for the reporting tip. I tried to describe the error fully as perhaps theres a simple fix on the horizon if its just a deleted items error , slowing down the database search.

isis, would you mind expanding on this topic a bit? If it refers to a discussion in another thread, could you post a link to it?

#26 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3589 posts

Posted 09 February 2010 - 09:43 PM

I'm late to this topic, but I would like to try to make a distinction. It seems to me that people are describing two kinds of slowness which I will call macro-slowness and micro-slowness. An example of macro-slowness might be when you start the program and open a data base - how long before you can start using the program? Is it 5 seconds, 10 seconds, or 5 minutes? Examples of micro-slowness have to do more with whether the program (and probably the underlying Windows) can keep with with the keyboard and mouse. A prime example of this has already been noted in this thread, namely CTL-F to open RM Explorer followed immediately by typing in a name. The micro-slowness can cause some of what you type to be swallowed or misinterpreted entirely. And the same thing can happen with the mouse sometimes.

For whatever reason, I do not seem to be much afflicted by macro-slowness. I do have a very old machine, but I do have 1GB of memory. That's not really enough memory these days, but it's not like it's 256MB or something like that. Plus, it doesn't seem to matter to me all that much if the program opens in 5 seconds vs. 10 seconds. I have about a 60,000 person database, and it usually opens for me in just a second or two, anyway. So for me at least, things like that are fine.

On the other hand, I'm constantly bothered by the micro-slowness. It may be because I'm a fairly fast touch typist. I don't know. But truly small delays in responding to the mouse and keyboard that may seem picayune - a quarter second delay, a half second delay, a second delay, for example - can be fairly devastating. Such small delays can cause you to lose your train of thought, make mistakes, and create serious errors in your data base.

I'm not sure if macro-slowness and micro-slowness have a common set of underlying causes or not. But except for really extreme cases of macro-slowness, I would prefer that any efforts that might be made to improve performance be focused on eliminating micro-slowness.

#27 Jim Belanger

Jim Belanger

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Posted 09 February 2010 - 09:44 PM

If there is a safe way to do it, I'd love to try it. Has anyone tried this? I could backup before I do it


Also thanks to Brenda for the reporting tip. I tried to describe the error fully as perhaps theres a simple fix on the horizon if its just a deleted items error , slowing down the database search. I think if it was just that it couldnt handle large files, then it wouldnt have sped through the same big database for the 1st few days of use. I think after merging over 500 it really started slowing down hard. I didnt delete any records, I just merged items ie linked & unlinked people

Also, I may try running it on a mac windows emulator, to see if its faster. I found large videos ran faster on my mac than on my pc.

Thanks All


JIM.B.NH

#28 Jim Belanger

Jim Belanger

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Posted 09 February 2010 - 09:51 PM

My database is only 155,000, half of what someone else mentioned. My computers are 1ghz ram and 1ghz processor.
I have not merged any names nor places. I have never ever deleted anyone, I simply re-use them.
Yet, every time I paste or memorize a source, I have to sit back and wait. If I press CTL-F then P too quickly, I am in the add a parent screen instead of the find screen with the first letter of a surname P. I have tried to go back to RM3 via Gedcom and that was a disaster. A GedCom export takes 20 minutes.
So, I live with it and pray for an upcoming solution. And, Yes, I have sent in complaints about speed and begged Bruce to reply to our requests for help and even suggested replies like 1. We are working on it, 2. We don't plan to address that. 3. We don't think this is a problem. Anything but silence would be appreciated.
JIM.B.NH

#29 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3420 posts

Posted 09 February 2010 - 10:17 PM

So, I live with it and pray for an upcoming solution. And, Yes, I have sent in complaints about speed and begged Bruce to reply to our requests for help and even suggested replies like 1. We are working on it, 2. We don't plan to address that. 3. We don't think this is a problem. Anything but silence would be appreciated.

It seems quite a long time since Bruce tgave any indication on the board <_< I only hope he still reads it.......

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#30 isis

isis

    Member

  • Members
  • PipPip
  • 13 posts

Posted 10 February 2010 - 02:11 PM

I'm late to this topic, but I would like to try to make a distinction. It seems to me that people are describing two kinds of slowness which I will call macro-slowness and micro-slowness. An example of macro-slowness might be when you start the program and open a data base - how long before you can start using the program? Is it 5 seconds, 10 seconds, or 5 minutes? Examples of micro-slowness have to do more with whether the program (and probably the underlying Windows) can keep with with the keyboard and mouse. A prime example of this has already been noted in this thread, namely CTL-F to open RM Explorer followed immediately by typing in a name. The micro-slowness can cause some of what you type to be swallowed or misinterpreted entirely. And the same thing can happen with the mouse sometimes.

I'm not sure if macro-slowness and micro-slowness have a common set of underlying causes or not. But except for really extreme cases of macro-slowness, I would prefer that any efforts that might be made to improve performance be focused on eliminating micro-slowness.

I dont think it has anything to do with processor & memory , as i have a 900,000 people database, which sped through all of the RM4 features speedily , for a few days, until I'd merge a few hundred individuals & places, on a 2GB 2GHZ Vista business laptop.I expected the programme to be much slower with this database & I havent found another geneaology programme apart from RM3, that can handle it

Its only after doing 100's of merges , by 3 days later that it started to run manual duplicate merges at a slow place. I think the lack of the rebuild /repackage/purge database which is in RM3 & PAF, slows the database index name search, because it may be searching deleted/merged records as well as new merged individuals, which must really slow it down

Has anyone else, experienced it slowing down the more you merge

I had tried RM4 just after beta mode last year on a 500k memory 1GHz processor XP machine & it did run very slow from the off & clearly that was a memory etc problem, but as the new version of RM4 ran quickly , with the same large database initially, then it leads me to think its a merge & purge problem


Also despite the 900k people , RM4 reported no errors during use, and all features , worked, only problem, it just got slower & slower to merge, my only real complaint

#31 Jim Belanger

Jim Belanger

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Posted 17 February 2010 - 07:44 AM

I just compared my SLOW RM4 program in one laptop against the exact same database using Rootsmagic Essentials and Essentials doesn't have the "SLOW" problem and runs circles around the full version. Does that give anyone a hint???
JIM.B.NH

#32 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 17 February 2010 - 08:10 AM

I just compared my SLOW RM4 program in one laptop against the exact same database using Rootsmagic Essentials and Essentials doesn't have the "SLOW" problem and runs circles around the full version. Does that give anyone a hint???

Did Essentials come out after 4.0.7.1? If so, maybe you'll see a speed-up in the next release of the full version. There could be tweaking of table indexes, query design, cache management that have happened along the way. Then again, maybe it is the suppression of some features that has lifted the burden.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#33 Alfred

Alfred

    Advanced Member

  • Members
  • PipPipPip
  • 5734 posts

Posted 17 February 2010 - 09:07 AM

A while ago, shortly after downloading and applying the 4.0.7.1 patch, I was curious about all of the questions about Essentials so I downloaded it. Installed it in it's own folder and ran it, It wasn't Essentials, it was 4.0.7.1. Since I had installed it on the same computer as I already had the registered copy of RootsMagic 4, it picked up the registration code. I came to the conclusion that the 4.0.7.1 update download and the Essentials download were exactly the same thing.
Everything in the folder was the same size and date as the earlier upgraded 4.0.7.1. (If I can remember correctly.)

Jim, Do you have both on the same computer? I couldn't try Essentials, I thought it was because I couldn't find how to remove the registration code.
Alfred

#34 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3420 posts

Posted 17 February 2010 - 01:11 PM

The next update should be in about 2 to 3 weeks.

RootsMagic, Inc.

Just in case anyone missed the recent news we should find out soon.

By then it will be pushing to almost 4 months since the 4.0.7.0 release so I would not expect a minor fix list. ;)

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#35 Jim Belanger

Jim Belanger

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Posted 17 February 2010 - 06:20 PM

A while ago, shortly after downloading and applying the 4.0.7.1 patch, I was curious about all of the questions about Essentials so I downloaded it. Installed it in it's own folder and ran it, It wasn't Essentials, it was 4.0.7.1. Since I had installed it on the same computer as I already had the registered copy of RootsMagic 4, it picked up the registration code. I came to the conclusion that the 4.0.7.1 update download and the Essentials download were exactly the same thing.
Everything in the folder was the same size and date as the earlier upgraded 4.0.7.1. (If I can remember correctly.)

Jim, Do you have both on the same computer? I couldn't try Essentials, I thought it was because I couldn't find how to remove the registration code.


No Alfred. I didn't install the Essentials in the same computer as the regular version. I had my brother download and Install Essentials on his computer, sent him a backup of my database and had him restore it. Yesterday, I visited with him (he is in Florida and I am in NH) and we both opened our respective programs in our own laptops. His opened immediately while mine had the hour glass for several minutes. Then a CTL-F on his brought up the list of names immediately while mine took a while. I noticed he could not save reports in RTF and a few other bennies he has missing.
JIM.B.NH

#36 kbens0n

kbens0n

    Advanced Member

  • Members
  • PipPipPip
  • 3459 posts

Posted 17 February 2010 - 07:36 PM

I had my brother download and Install Essentials on his computer, sent him a backup of my database and had him restore it.

Sounds like the real ultimate comparison would have been for you to close out all open databases, close RootsMagic, open it, and then restore from the same backup that you sent your brother (and compare speeds).

---
--- "GENEALOGY, n. An account of one's descent from an ancestor who did not particularly care to trace his own." - Ambrose Bierce
--- "The trouble ain't what people don't know, it's what they know that ain't so." - Josh Billings
---Ô¿Ô---
K e V i N


#37 Alfred

Alfred

    Advanced Member

  • Members
  • PipPipPip
  • 5734 posts

Posted 17 February 2010 - 07:42 PM

It isn't RootsMagic's fault that it is running so slow.

There is something else on your computer that is slowing it down.
Either some kind of malware filter, an indexing program of some kind or something.
Because, Essentials is exactly the same program, all you have to do is add the registration code to get the full version.
Alfred

#38 Jim Belanger

Jim Belanger

    Advanced Member

  • Members
  • PipPipPip
  • 658 posts

Posted 18 February 2010 - 07:42 PM

It isn't RootsMagic's fault that it is running so slow.

There is something else on your computer that is slowing it down.
Either some kind of malware filter, an indexing program of some kind or something.
Because, Essentials is exactly the same program, all you have to do is add the registration code to get the full version.


I respectfully disagree. There are way too many of us with large databases experiencing the same slow response.
Thank you to everyone for your suggestions. Now, it's my turn.
Why don't some of you load up GedCom file after GedCom file into a test database until you get 155,000 or 175,000 names and then try your program. If you can tell us that your program then is FAST, I will rethink my opinion.
JIM.B.NH

#39 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 18 February 2010 - 09:03 PM

Sounds like the real ultimate comparison would have been for you to close out all open databases, close RootsMagic, open it, and then restore from the same backup that you sent your brother (and compare speeds).

That would test whether backup and restore has any effect on the database, such as the old Pack or Reindex utilities. From my own observations of database properties, backup/restore has no effect - it is just a ZIP utility.

What other differences might there be between the two laptops that might give rise to such pronounced differences in performance and between Jim's early fast experience and progressive slowing? Cache size and management come to mind. If Jim's computer does a lot of disk thrashing and his brother's does not, then maybe RM is taking advantage of a greater availability of RAM. SQLite (for that matter any large database system) is susceptible to performance problems with large datasets, indexes on long keys, mistuned cache and page size. I've only just started monitoring the sqlite forum and have seen discussions above my head on slow performance with 1,000,000 records and fast performance with 20X that - very dependent on many variables.

Alfred suggests interference from other apps on the one computer and not the other. That may be and cannot be ruled out without controlled testing.

What about different Program options? The File options will have travelled with the backup and should be the same. Multimedia files are probably missing from his brother's. Are there things that Essentials dos not have to maintain that the full version does?

Ideally, these slow files need to be analyzed by the developers to ascertain where the bottlenecks may be. It certainly gives lie to the advertising claims of unlimited this and that.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#40 isis

isis

    Member

  • Members
  • PipPip
  • 13 posts

Posted 18 February 2010 - 09:21 PM

It isn't RootsMagic's fault that it is running so slow.

There is something else on your computer that is slowing it down.
Either some kind of malware filter, an indexing program of some kind or something.
Because, Essentials is exactly the same program, all you have to do is add the registration code to get the full version.


I have an approx 900,000 people database of various families, which have german, & english versions of family trees, going back to vikings, egypt etc

In RM3 , this runs very fast with no delays, irrespective of 1,000's of merges. However on the same PC , I installed the new version of RM4 & initially for the 1st few days, it was a gazelle, with no delays. However after a few 100 people merges & a few days later, it slowed from taking, approx 1 sec to go from 1 manually merged name back to the list of names to select the duplicate to merge with , to up to 10 mins. This became so frustrating I've stopped using the programme.

As you'll see from my earlier posts , I ve tried many things to try to speed up my windows software to see if that was the problem, although my machine runs fast. Ive submitted the programme errors to the helpsite & had no solutions offered.

I had the same problem when the programme 1st came out, although it was worse, ie the programme was running like a snail, from the very 1st time you used it, then it slowed to up to 30mins to go back to the merged list. I thought the problem was solved in the new version, as it ran superfast even though it had 900,000 individuals. It was only when I did 100's of people & place name merges, that it then started to slow down & not before. This I think is because the rebuild & package indexes, feature has been removed in RM4. It doesnt delete merged records entirely, evidenced by the fact that when you merge variants of a name, you are still shown all of the names you merged in the manual duplicate list of names , this must slow the programme down as the index to thge database stays the same size despite having lots of deleted/merged records. Its like having a warehouse with 1 full cereal box and 1,000's of now empty cereal boxes. But it has to keep searching the clutter of the empty cereal boxes to find the one you want. You can't even switch off the alternative name view, in the manual merge, primary, 2ndary merge function. You can only switch off alternative name view in the main left sidebar of the programme. But theres no need to see it.

My posts above are:
http://forums.rootsm...dpost__p__33357

http://forums.rootsm...dpost__p__33330

Disappointly, I reported this fault last year & this year & nothing has been done about it , I havent even had a constructive reply to my bug reporting, earlier, they ve just said theyve logged my bug report, so I've had to go back to Rm3, which is faster

Come on RM4 , its time to put the trash out.