Jump to content


Photo

RM4 is SLOW


  • Please log in to reply
72 replies to this topic

#61 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6256 posts

Posted 21 June 2010 - 01:40 PM

Only IPBoard Administrators can delete messages. Three consecutive messages were deleted, one each by Ludlow Bay, vyger and John James - in this thread. I don't know if any other threads have had deletions as no one has raised the question.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#62 mapleleaf

mapleleaf

    Advanced Member

  • Members
  • PipPipPip
  • 559 posts

Posted 21 June 2010 - 03:05 PM

Renee, how do you delete you own message? I've never found out how if it is possible. Thanks, Laura


Now Renee's post was deleted! Before Laura's.

I've also noticed other posts in other threads deleted now and then.
~ Debbie

#63 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3426 posts

Posted 22 June 2010 - 05:31 AM

Last night I was using the Duplicate Search Merge feature and moving happily from one possible match to the next in about 1 second. Happy days, I thought, RM have fixed this although I don't remember reading about it.

Not the case, today this feature is as painfully slow as it ever was although I am now worried about what type of merging RM was doing so quickly last night and totally confused as to the difference in performance. I tried 4 different computers today with different builds and operating systems and none performed at a speed you would expect bearing in mind the processing involved.

You can read the results and times here and I can only hope someday soon the inherent slow processing operations within RM4 can be finally overcome. :(

http://www.vyger.co....cate-merge.html

I have to add a little to this test and will update my web page next.

RM has asked for the file concerned and once I remove some stuff I will make it available, I have already removed some duplication and my living family and run the Duplicate Search Merge again using the default options. The time to display the initial merge screen was more than halved and the time to step from one possible match to the next also considerably improved.

In my original example I was running Soundex, Birth & Death years the same and Do Not compare Birth or Death Places, which was giving me a more reliable match list. I should have stated this in my original example and my apoligies for any confusion caused. I will update the web page to more accurately reflect my findings.

The other thing I noticed while reviewing my previous versions of this file was the size on disk. This file originated when I merged all my smaller reference files into one and the file size on disk was 284Mb, my next version was ~118Mb on disk?. I thought I had lost some information so compared the files, all properties were as expected,

I ran all automerges on the 284Mb file and the size remained unchanged.
I exported to gedcom and imported to a new file and hey presto ~118Mb.

I don't know how this might affect RM speed of operation but it does seem to add some weight to the argument for a pack/compress routine in a future version.

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#64 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8475 posts

Posted 22 June 2010 - 09:53 AM

I have administrator rights so I can delete postings. The delete button is down at the bottom of a post right next to "Edit". You would need to email us to delete a posting. So John James didn't delete his post it was done internally with approval from Bruce.
Renee
RootsMagic

#65 DerickH

DerickH

    Advanced Member

  • Members
  • PipPipPip
  • 173 posts

Posted 05 July 2010 - 04:48 PM

Last night I was using the Duplicate Search Merge feature and moving happily from one possible match to the next in about 1 second. Happy days, I thought, RM have fixed this although I don't remember reading about it.

Not the case, today this feature is as painfully slow as it ever was although I am now worried about what type of merging RM was doing so quickly last night and totally confused as to the difference in performance. I tried 4 different computers today with different builds and operating systems and none performed at a speed you would expect bearing in mind the processing involved.

You can read the results and times here and I can only hope someday soon the inherent slow processing operations within RM4 can be finally overcome. :(

http://www.vyger.co....cate-merge.html



#66 DerickH

DerickH

    Advanced Member

  • Members
  • PipPipPip
  • 173 posts

Posted 05 July 2010 - 05:01 PM

At first I thought it was my 4 year-old H-P Pavillion and XP Pro that was the cause of the painful SLOWNESS of RM4. I do have 166,000 individuals with extensive notes in my database but the same database in RM3 works quickly enough to satisfy me. I loaded RM4 GEDCOM to a Vista laptop and to a new computer with Windows 7. Same problem. RM4 is just plain SLOWWWWW. In every case, it takes over a full minute to bring up the index! Same for selecting a merge target.

MicroSoft never actually disavowed Vista but replaced it with Win7. It now appears that RM4 is a dog, like I have read on nearly 200 responses to the subject: RM4 IS SLOW. Too SLOW for me to wait 73 seconds for an index to come up.

I have reached the point where I am ready to switch over to FTM. A friend has over 200,000 individuals with extensive documentation on FTM and it runs like a breeze. Until I can complete the switch-over, I'll be happy with RM3, sans the bells and whistles I have come to enjoy. This is really disappointing as I started with Family Origins 1.0 when it was a Parsons Software product. I had hoped that after a couple of years the authors could overcome the SLOWNESS handicap and give us something a little more user friendly.

#67 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6256 posts

Posted 05 July 2010 - 07:25 PM

I ran all automerges on the 284Mb file and the size remained unchanged.
I exported to gedcom and imported to a new file and hey presto ~118Mb.

I don't know how this might affect RM speed of operation but it does seem to add some weight to the argument for a pack/compress routine in a future version.

A pack/compress routine at the database engine level would be of little benefit - RootsMagic uses the SQLite database engine; I have tried its VACUUM function and seen <1% reduction in file size. The problem is that RM is sloppy about cleaning up after itself when some things are merged or deleted. Being a relational database, a fact about a person involves multiple tables; merging or deletion might eliminate one row from one table but leave the others intact, resulting in orphaned data. That's mainly why your database did not shrink after merging.

Whether there is much penalty paid in having a larger file with orphaned data vs a smaller file without orphaned data, I cannot say. If a virus scanner is triggered to scan the whole file every time there is a write to it, then it would likely be a bigger hit on the system than if not. Likewise for such utilities as DropBox that attempt to synchronise files across the Internet. Similarly for indexing apps. If they must be allowed to work on the file, the smaller the better. Better still to isolate the RM datafile from them.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#68 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3426 posts

Posted 06 July 2010 - 06:03 AM

I have reached the point where I am ready to switch over to FTM. A friend has over 200,000 individuals with extensive documentation on FTM and it runs like a breeze. Until I can complete the switch-over, I'll be happy with RM3, sans the bells and whistles I have come to enjoy. This is really disappointing as I started with Family Origins 1.0 when it was a Parsons Software product. I had hoped that after a couple of years the authors could overcome the SLOWNESS handicap and give us something a little more user friendly.

I wouldn't go rushing off just yet; yes there is an inherent slowness in various RM4 operations but I do believe these things must be getting looked at and expect they will improve in the future. I know there is also the problem of users starting to accept this speed of operation as normal whereas direct comparisons with RM3 show that it is not.

Some day to day functional improvement could be achieved by more efficient buffering of key strokes so that when the program does catch up at least what is on screen is what was desired.

Maybe we will get some indication from Renee later, disclosing that work is ongoing with respect of speed of operation can hardly give the competition any sort of edge. ;)

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#69 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8475 posts

Posted 06 July 2010 - 10:33 AM

Maybe we will get some indication from Renee later, disclosing that work is ongoing with respect of speed of operation can hardly give the competition any sort of edge. ;)

I have confirmed with the RootsMagician, that yes, we are looking into the speed issue.
Renee
RootsMagic

#70 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3426 posts

Posted 16 July 2010 - 05:42 PM

I have confirmed with the RootsMagician, that yes, we are looking into the speed issue.


Whilst I still maintain a modest database in RM4 I have taken the decision to revert to RM 3.2.6 for my merge project of all my orphan files in anticipation of a speed increase and also the introduction of Dynamic Groups. So I exported everything with an edit date since June 2009 as safe keeping for when I revert to RM4 and began merging those files again.

At this point the database holds 28420 individuals and since I was about to run my first Duplacate Search Merge on the file I thought a comparison between RM3 and RM3 was appropriate.

Time to display the initial merge screen - RM3 = 18.6 seconds - RM4 = 41.0 seconds
From merging 1 to display of next - RM3 = < 1 second - RM4 = ~1.6 seconds

On a very clean file this is possibly more representative than my previous test and whilst the time to display the initial merge screen shows a massive difference the additional checks for Place Details etc might account for the difference in moving from one merge to the next.

* The other things I would note is that making a custom fact exactly as a built in fact will merge them on gedcom export/import.
* Merging of duplicate Sources and Repositories works OK in RM3.
* It is much easier for me to reconcile the Place List with external editors for example searching for contains "Church" and making corrections, something which the RM4 Place List could benefit from.

Personally I would be happy with these times providing they were repeatable which has not always been my experience, some allowance has to be made for the extra information like Place Details etc which RM3 does not cater for.

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#71 Vyger

Vyger

    Advanced Member

  • Members
  • PipPipPip
  • 3426 posts

Posted 18 July 2010 - 11:48 AM

Well, Bruce found the problem - it is related to Place Details index, and it will be fixed in the next release...
Thanks to RM & Bruce for such a quick response.

The above news is from a separate thread and it makes some sense to me now. The above speed example file of mine has NO Place Details whereas the previous very slow example had a lot.

Well done Bruce and team, that find is good news.

We are all limited by our visions and abilities

Whilst we can borrow from the visions of others we cannot always deliver.

 

User of Family Historian 6.2.7, Rootsmagic 7.6.2, Family Tree Maker 2014 & Legacy 7.5

 

Excel to Gedcom conversion - simple getting started tutorials here

 

Root


#72 Romer

Romer

    Advanced Member

  • Members
  • PipPipPip
  • 2073 posts

Posted 18 July 2010 - 12:29 PM

Here's that related thread:

http://forums.rootsm...m4-is-very-slow

#73 RootsMagician

RootsMagician

    Administrator

  • Admin
  • PipPipPip
  • 826 posts

Posted 19 July 2010 - 09:34 AM

The above news is from a separate thread and it makes some sense to me now. The above speed example file of mine has NO Place Details whereas the previous very slow example had a lot.

Well done Bruce and team, that find is good news.


To be clear, this fix is only for the incremental search in the place list with large databases (ones with a large number of places in them). It doesn't matter how many place details there are (it occurred even in databases with no place details).
RootsMagician