Jump to content


Photo

Extreme Source Splitting with RM's Source Templates


  • Please log in to reply
12 replies to this topic

#1 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3519 posts

Posted 17 December 2013 - 10:21 PM

The following message has been inspired by a number of messages from TomH over a long period of time, plus a recent problem I encountered with my strategy of extreme splitting of sources within RootsMagic.

In brief, the extreme source splitting strategy involves using only the Master Source part of a citation and leaving the Source Details part of each citation completely empty. The initial impetus for the extreme splitting strategy was to be able to make a change to a single instance of a citation that had been memorized and pasted many times, and then for that change immediately to appear in all the other copied and pasted instances of the same citation. It has turned out serendipitously that this extreme source splitting strategy also means that my sources can be exported via GEDCOM very reliably, even if I use RM's source template facility. It has also turned out serendipitously that this extreme source splitting strategy means that my sources can be interchanged between RM and FamilySearch Family Tree very reliably, even if I use RM's source template facility.

Even so, I have been applying my source splitting strategy while still using RM's built-in Free Form source template rather than using RM's source template facility. And I have been very happy with the results. Until now. The recent problem is that I now have between two dozen and three dozen obituary sources that have been built as extremely split sources, and I keep tweaking my footnote sentences. I will think that I have my footnote sentences for obituaries exactly the way I want them, and then I will change my mind. So I will have to change my footnote sentences for obituary sources between two dozen and three dozen times, once for each extremely split obituary source. As we speak, the exact number of extremely split obituary sources is 32, and I expect the number to grow to hundreds or even thousands. Obviously, I don't want to have to make the same change to hundreds or even thousands of extremely split obituary sources. I really don't even want to make the same change 32 times.

Before proceeding further, I should make note of fact that this little glitch does not invalidate my basic strategy. Quite the contrary. Obituaries tend to be cited many, many times. For example, if each of my extremely split obituary sources were cited 10 times then my tweaking would consist of fixing 320 citations rather than fixing 32 Master Sources. Yuck! (Well, assuming that the tweaks involved the Source Details part of the citation, which they usually do unless I'm using extremely split sources.)

So I decided to listen to the wisdom of TomH and to try to develop my own source template for obituaries, where my own source template would place all the data only in the Master Source. It turns out that developing such a source template was really quite easy, and my new source template works great for newly created obituary sources. But what about those 32 obituary sources that are already out there using the built-in Free Form source template, and each of them is cited many, many times?

Well, assuming you know SQL, it's almost trivial using SQLite to convert those 32 obituary sources so that they use my newly developed obituary source template. In particular, all the citations stay in place so that there is no subsequent problem in trying to find and fix all the hundreds of citations of those 32 obituary sources. However, the glitch is that the original and carefully crafted footnote, short footnote, and bibliography sentences are lost when my obituary sources are converted from the built-in Free Form template to using instead my obituary source template.

Not all is lost. My idea was to convert each of the obituary sources to my new obituary source template one at a time, and that I would copy the citation sentences into Notepad before the conversion and copy pieces of the sentences back into the obituary sources from Notepad after the conversion. Just to be clear, a footnote sentence for an obituary citation typically includes such things as the name of the deceased, the name of the newspaper, the date of the obituary, the page number of the obituary, etc. With the Free Form template, all this data is in one large data element called [Footnote]. With my new obituary source template (or really, with anybody's source template), the same information is distributed into several smaller data elements such as [Name], [Newspaper], [Date], [Page], etc. So I was copying the entire [Footnote] data element into Notepad and then copying the pieces such as [Name], [Newspaper], etc. back into my own obituary source template.

Tom's brilliant suggestion was to add the [Footnote], [ShortFootnote], [Bibliography], and [Page] data elements into my newly developed obituary source template before converting my obituary sources from Free Form to using my new source template. That way, the old data would still be there in each of the 32 obituary sources. I would still have to copy and paste from the old data fields into the new data fields, but all the data would still be right there in RM and I wouldn't have to mess around with Notepad at all. Better still might be to write some computer code (probably very complicated) that would move the data correctly. But there would be so many exceptions and exceptions to exceptions that the code would be very complicated and error prone, and I only had 32 sources to convert. So that's what I did, and it works great. I don't envy anybody who tries to write the computer code to do all this as a fully automated process.

By the way, the old [Footnote], [ShortFootnote], etc. data elements are still there, but they are not used for anything since the conversion is complete. Instead, the footnote sentence, the short footnote sentence, and the bibliography sentence are built from the [Name], [Newspaper], etc. data elements and the associated punctuation that I have included in the source template.

Moving forward, what this all means is that even if I reach hundreds or even thousands of obituary sources, I can tweak the footnote, short footnote, and bibliography sentences just once in the source template and the tweaks will apply immediately to all my obituary sources.

Jerry

#2 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3519 posts

Posted 20 December 2013 - 04:04 PM

Moving forward, what this all means is that even if I reach hundreds or even thousands of obituary sources, I can tweak the footnote, short footnote, and bibliography sentences just once in the source template and the tweaks will apply immediately to all my obituary sources.


I should have said the following to indicate the full import of this approach.

Moving forward, what this all means is that even if I reach hundreds or even thousands of obituary sources, I can tweak the footnote, short footnote, and bibliography sentences just once in the source template and the tweaks will apply immediately to all my obituary sources and to all of the citations of those sources.


Jerry

#3 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3519 posts

Posted 20 December 2013 - 10:57 PM

Since this thread is about changing the source template associated with a source from one source template to another source template, it's worth reiterating a few points.
  • The only way to make such a change is with an UPDATE statement in an SQL script that runs in SQLite. This is a programming task that requires a pretty substantial understanding both of SQL and of the RM database structure. It's a little dangerous and scary, and I'm not really advocating it as any sort of standard operating procedure. Nevertheless, RM itself doesn't support this functionality, and there's a pretty critical need for the functionality if source templates are to be used most effectively.
  • As I play around with the source templates, the need to change a source to a different source template manifests itself all the time. It's often really hard to pick the correct template. For example, for something so simple as a book, there are 7 different source templates to choose from. How are you to choose the correct source template? And more importantly, if you should choose the wrong one there is no way from within RM itself to switch over to a different source template. And even if you do luck out and choose the correct source template, there is no way to tweak the sentences in the template if the template you chose is one of the built-in templates.
  • For these reasons, the conventional wisdom of experienced RM users is never to use the built in templates directly. Rather, pick the built in template you wish to use and make a copy of it. Then use the copy with your source. But there is nothing in RM itself that leads you in this direction.
For those reasons, I would recommend the following changes be made to RM (which I will also post to the wish list).
  • One possibility to improve the situation would be for RM not actually to allow the built in source templates to be used directly. Rather, when a user chooses a built in source template, RM could behind the scenes and without any extraneous clicks just make a copy of the chosen source template and associate the copy of the source template with the source. If this approach were deemed too harsh and disruptive, there at least could be an option in the create source process to request that the copy of the source template be made and used as I just described.
  • Instead of or in addition to #1, it should be possible for any source to request that its source template to be copied and for the source to use the copy. This would be somewhere on the List->Sources screen. Such a change would allow the user to convert a source from using a non-editable source template to using an editable source template while still leaving all the built-in source templates as non-editable.
  • As a consequence of #2, the built-in Free Form template should become a real template that can be copied. At the present time, this template shows up in the source template list when you are in the add new source dialog, but it does not show up in List->Source Templates screen itself.
  • Finally, it should be possible from within the List->Source Templates screen to select a Source Template, request that it be copied, and also request that all references to the Source Template from Master Sources be pointed to the copy instead.
Jerry

#4 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3519 posts

Posted 21 December 2013 - 08:13 AM

But what about those 32 obituary sources that are already out there using the built-in Free Form source template, and each of them is cited many, many times?

Well, assuming you know SQL, it's almost trivial using SQLite to convert those 32 obituary sources so that they use my newly developed obituary source template.


I didn't belabor the following point in my original message. It was more implied. But what if you don't know SQL, or even if you do what if you don't want to take the risk of updating your RM database by using SQL instead of by using RM screens? How would I change my 32 obituary sources (and soon to be many more) from Free Form sources into being templated sources?

The only alternative for each of my 32 obituary sources would be to make 32 new obituary sources, each one of which would use my newly developed obituary source template. And then each of the newly created obituary sources would have to be memorized and pasted an average of 10 times each for a total of about 320 memorize and paste operations. And of course the biggest problem in doing so is to find all the right places to paste the new sources and to keep track of whether I have found all the correct places or not. Assuming I got through that very manual, labor intensive, and error prone process ok, then I would go back to the Soure List and delete the original 32 obituary sources, one at a time - and with lots of clicking because all the orginal sources are still in use even as I'm deleting them.

All that extra work and all those potential data entry errors could be avoided which is why I was so highly motivated to use an SQL solution for this particular problem.

Jerry

#5 billbremer

billbremer

    Member

  • Members
  • PipPip
  • 22 posts

Posted 17 January 2014 - 09:57 PM

This may be a bit off-topic, but here goes:

"The initial impetus for the extreme splitting strategy was to be able to make a change to a single instance of a citation that had been memorized and pasted many times, and then for that change immediately to appear in all the other copied and pasted instances of the same citation."

Yes, a very worthwhile feature indeed. But why must we resort to "extreme" source splitting to achieve this? Why not add to RM the ability to share or link source citations? Linked citations would always be identical to each other; an edit to any one of them would automatically propogate to all the other citations in that chain.

Here's how a user would create linked citations:

Create the citation.
Memorize it.
Go to the other person/fact and press the (new) "Paste Link" button. (The "Paste" button function would not change in any way)

Each press of the "Paste Link" button creates a new, linked citation that is identical to all the others in that linked chain.
Linked citations should be clearly highlighted as such and an "Unlink" button could provide a method of removing a particular citation from the chain.

Is this not possible? Am I missing something or otherwise oversimplifying a complex issue?

#6 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6211 posts

Posted 18 January 2014 - 07:28 AM

It is not impossible but is, indeed, a somewhat complex issue requiring redesign of the database structure. Currently, the CitationTable contains the citation details AND the 1:1 link between the citation record and the event, person, family supported. Mem/paste creates additional records duplicated except for the unique link values. To achieve what you describe would probably require a new table containing just the links between the singular CitationTable records and these other tables. Call it CitationLinkTable. Superficially, I think it is a good idea but there may be complications in other processes.

I think it may be desirable to add this as a Master Citation or Duplicate Citation feature while preserving the current functionality which is a Replicate Citation function that supports individual tweaks.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#7 billbremer

billbremer

    Member

  • Members
  • PipPip
  • 22 posts

Posted 18 January 2014 - 08:29 AM

The approach I suggest does not remove any current functionality. It merely adds the choice of creating a group of citations that will remain "synchronized" across multiple events, persons, and families. These shared citations would keep all the properties of the normal, unlinked citations we already know and love.

RM already recognizes the need for identical citations; hence the memorize/paste citation function and the report generator's automatic aggregation of footnotes when citations are identical. All my suggestion does is allow for automatic "synchronization" of the citations that you wish to keep identical.

Like any change, it's work for the developers, but I doubt you and I are the only two RM users tryiing to achieve this capability.

#8 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3519 posts

Posted 18 January 2014 - 10:11 AM

But why must we resort to "extreme" source splitting to achieve this? Why not add to RM the ability to share or link source citations? Linked citations would always be identical to each other; an edit to any one of them would automatically propogate to all the other citations in that chain.


I chose to go the "extreme source splitting" route because that was something I had control over and that I could do myself. I can't add a new ability to RM to share or link source citations.

Jerry

#9 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6211 posts

Posted 18 January 2014 - 10:36 AM

And there are other advantages to "extreme" (I say "ultimate") "source splitting" as summarised at Sources - Adventures in Extreme Splitting which could be eliminated with other developments in the software, but, how long do we wait?

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#10 billbremer

billbremer

    Member

  • Members
  • PipPip
  • 22 posts

Posted 18 January 2014 - 11:21 PM

By no means am I trying to be critical of your approach. I'm simply pining, or is it whining, for an easier way.

#11 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6211 posts

Posted 19 January 2014 - 07:39 AM

Understood. This is a Wish List item and Renee will undoubtedly acknowledge that during her work week. And maybe this discussion has ventured into the How the enhancement might be accomplished more so than before - that's a good thing. I think it might be do-able as I described above with the addition of a new table, the CitationLinkTable. It would contain, at minimum, these fields:

CitationLinkID: a new, primary field unique to each record in the table
CitationID: the link to the master citation in the CitationTable
OwnerID: the link to the EventTable, PersonTable, FamilyTable, NameTable as directed by...
OwnerType: defines which table OwnerID points at
Quality: assuming we have to allow that the quality of a master citation varies among the facts it supports

Each unique citation created by the current Mem/Paste function gets a new record in CitationTable as it does now plus a corresponding record in the new CitationLinkTable. The proposed Mem/Link function creates no new records in CitationTable; only additional linking records in the new CitationLinkTable.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#12 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8406 posts

Posted 20 January 2014 - 11:35 AM

Confirming enhancement request is in our tracking system.
Renee
RootsMagic

#13 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6211 posts

Posted 20 January 2014 - 02:49 PM

A further thought about shared or linked citations pertains to the Detail Text (Research Notes and Comments) and that is, just as the Quality of the evidence may vary with the fact being supported, so too may there be a need to vary the Detail Texts in some ways to make them better fit. While that may seem to run counter to the whole idea of being able to edit the master and have it ripple through to all instances, it is a virtue of "ultimate splitting" in the sense that the text in common is placed in the Master Text and, even though there may be no fields in Source Details, there still remains the Detail Text fields unique to each citation. Shared or linked citations functionality would have to have similar flexibility in order for it to achieve that result of ultimate splitting.

A solution would be for the footnote generated from a shared citation to output the Detail Text from the Master Citation unless the corresponding field of the Slave Citation is non-empty, in which case the latter is outputted. This could be made even more flexible if the sharing citation had these user-selected output options:
  • Slave's text instead of the Master's (default)
  • Append the Slave's text to the Master's
  • Append the Master's text to the Slave's
This change would require the addition of fields Comments and ActualText to the suggested CitationLinkTable, and depending on whether the output options are set at the Master Citation of individually at the Slave Citation, an OutputFlags field to either this new table or to the CitationTable.

Of course, the Edit Source form would have to be carefully revised to support Master/Slave citation functionality including previews of outputs and easy identification of all instances where the citation is used and navigation thereto. Not trivial...

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.