Jump to content


Photo

Custom template plus switches plus punctuation

templates punctuation gedcom

  • Please log in to reply
22 replies to this topic

#1 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 17 November 2015 - 05:53 PM

So I will present my problem primarily in picture form with a little narration.

 

I am working backwards. I deveoped my full footnote as I want it to be and then began figuring out how best to enter it into RM. Full freeform is still an option but I would essentially end up with a master source PER household. And have the duplicate bibliography issue that goes along with that. Not my ideal. I ended up "creating" a template which is "freeform" at the top but with 3 source details fields to allow me to enter things as I want them.

 

Pic 1: A screenshot of my custom template.

bk5nT7q.jpg

 

 

So next, here is a pic of my sample data when filling it out as a source. The results on the right hand side of the screen are exactly what I want.

vGmCtXj.jpg

 

 

OK, so far so good you would think. And yes, it is. I have my source. I can "lump" all 1841 England from this database title together. My footnote is exactly how I want it, as is my Short Footnote and Bibliography.

 

The problem (as always?)...is with punctuation missing and replaced during gedcom export.

 

Here is a pic of my gedcom file (irrelevant data removed) showing the problem.

9zvAEB2.jpg

 

 

So I am looking for input? Is there a change I can make to my custom template to mitigate or fix my export problems? I have changed some things around in my template several times but this is about the closest I have gotten it though I am still playing.

 

Thanks.



#2 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3221 posts

Posted 17 November 2015 - 07:02 PM

I may be missing the essence of what you are trying to accomplish, but it appears to me that you are using switches when switches aren't necessary. It is not necessary to use switches for very simple templates that are still conditional on fields being present or absent.

 

For example, I think you could replace <?[Accessed]|:[Accessed]> with <: [Accessed]> and get the effect you are after. In other words, if you have entered an access date, then a colon will appear followed by a blank followed by the date. Otherwise, nothing will appear for an access date.

 

Similarly, I think you can replace <?[POI],|[POI]> with <, [POI]> and I think you can replace <?[TNAUK]:[TNAUK]> with <: [TNAUK]>. My apologies if I'm not quite understanding what you are trying to accomplish.

 

Jerry

 



#3 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 17 November 2015 - 09:00 PM

Jerry,

 

I think you are right about the switches. I got hung up in the RM6 book on the section that illustrated the use of the switches and should have flipped back a page.  I will experiment with that.

 

My primary question...and perhaps I should have led with the 3rd picture...is: why did my punctuation go missing or change?

 

The : between the url and the viewed date disappeared when exported to gedcom.  Also, the , that appears after the "accessed" date and before the "POI" changed to a semi-colon.  How can I keep my punctuation exactly as I have it when exported to gedcom? Or can I?  I had experimented with several different tests and wasn't having any luck. I will try with your simple switches though I doubt the behavior will change.



#4 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5997 posts

Posted 17 November 2015 - 09:22 PM

The changed punctuation is due to standard GEDCOM having only one field or tag (PAGE) for what RootsMagic calls Source Details. So RM cannot use your sentence template in its entirety and concatenates your Source Details fields values in the order they are listed in your entry screen and separates them with "; ". Hence, any template with more than one Source Details field will have the semi-colon separator in its exported PAGE value.

 

You might find this experiment of interest: Source Templates - Versatile Free Form Hybrids

and this one: Source Template, A Better Free Form.


Edited by TomH, 17 November 2015 - 09:30 PM.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#5 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 17 November 2015 - 10:07 PM

Side question to this mess:

Does RM store the "full footnote" anywhere in the database after it is created or does it always generate it "on the fly" when needed? Because if it is stored somewhere, why couldn't that be exported in it's entirety to the TITL field?  

 

OK, back to the rest:

So the semi-colon can not be avoided. Bummer.I guess one option is to craft my source in such a way that I could do a search and replace on the gedcom when needed. I just need to make sure I set it up so that I would not be accidentally replacing it in other citations where it shouldn't be replaced.

 

Hmm...off for a few more experiments.

 

Of course, I may just say "four letter word" it and have a master source per household and forget the possibility of ever having a useable bibliography.  I guess I should double check this, but isn't it the case that if you have multiple sources that have the same bibliography entry that you will end up with multiple identical entries in the bibliography?



#6 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5997 posts

Posted 17 November 2015 - 10:45 PM

Whatever you put into the Footnote field of a Free Form source is exported to the TITL tag. That can be the full footnote sentence if you leave the Page field empty, i.e., an ultimately split source.

 

If different Master Sources have the same Bibliography field value, that value appears only once in the report's Bibliography page, iirc...


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#7 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3221 posts

Posted 17 November 2015 - 11:05 PM

The full footnote is generated on the fly.The data you enter for the Master Source and Source Details is stored as XML, where the XML tags correspond to the fields in the source template. The full footnote is generated on the fly based on the data from the XML as formatted by the sentence template that is a part of the source template.

 

Tom is absolutely correct about the GEDCOM. There are details about exactly how templated source data goes into GEDCOM that are not specified directly in the source template itself, and Tom is much more knowledgeable about how that works than I am. In essence, GEDCOM supports RM's source data in a straightforward way only if you use free form source template - either RM's built in one or Tom's enhanced one - or if you create your own source templates and your source templates place all the data into the Master Source. If you choose the latter course of action, you are choosing to be an extreme source splitter and most users don't want to go that route.

 

On your bibliography question, I indeed am an extreme source splitter and one consequence is that it's impossible in any practical sense to  produce a useful bibliography because of the multiple identical bibliograhic entries..  If RM (or even Microsoft Word, after the fact) would merge the multiple identical bibliographic entries, then it would be easy to make good bibliographies as a source splitter, but not otherwise. I have chosen to go exclusively with endnotes, not footnotes and not bibliographies. As such, the Short Footnotes and the Bibliography entries produced by my source templates are never used nor ever can be.

 

Jerry

 



#8 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 18 November 2015 - 04:42 AM

On your bibliography question, I indeed am an extreme source splitter and one consequence is that it's impossible in any practical sense to  produce a useful bibliography because of the multiple identical bibliograhic entries..  ...   I have chosen to go exclusively with endnotes, not footnotes and not bibliographies. As such, the Short Footnotes and the Bibliography entries produced by my source templates are never used nor ever can be.

 

That is sadly what I thought I remembered reading. Thanks for the confirmation.

 

So, my choices at the moment are:

1. revisit my ideal citation and decide if I can re-arrange the components in such a way that I can live with only the use of semi-colons

2. become a splitter or an extreme splitter and give up the idea of a neat bibliography (either via straight free-form or completely split with no detail fields)

3. go with the template I have that displays exactly like I want it to display in RM and hope RM remains my program of choice for the next 30-50 years (and either manually edit a gedcom or live with incorrect punctuation if I share with other (non-RM) people)

 

To be fair, when I exported to gedcom and then imported into Legacy, the results for this one aren't unforgiveable. Just not as designed.  The imported footnote in Legacy was:

"1841 England, Wales & Scotland Census," database with images, FindMyPast (http://www.findmypast.com, viewed 06 November 2015); entry for John Hunter [household], Harwood Dale, Yorkshire; citing [The National Archives of the UK] HO107/1265/4, folio 3, pg. 1 (stamped), lines 19-23.

 

Obviously Legacy (and RM probably does this too) put a comma in between the TITL and PAGE fields on import. And of course the semi-colon after the accessed date was the way RM exported. It is not unforgiveably bad. And of course, since the "Short Footnote" and "Bibliography" entries don't transfer anyway, if you did share a file or move to another program, there would be a ton of work to do to clean up sources anyway!

 

I know I'm making a big deal about this and many either don't care about how it exports or just move on with it, but it bugs me...I want the trifecta of being able to moderately lump aka use detail fields, use a bibliography, and transfer the citation to other people/programs the way it looks when complete in RM.  That doesn't seem like much!



#9 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3221 posts

Posted 18 November 2015 - 10:20 AM

If different Master Sources have the same Bibliography field value, that value appears only once in the report's Bibliography page, iirc...

 

I was wondering if I was remembering this correctly, so I double checked with a little test report. When different Master Sources have the same Bibliography field value, that value actually does appear multiple times in the report's Bibliography page.

 

It seems to me that this is a bug. But whether it's a bug or whether it's according to design, it would be extremely useful if the behavior could be changed so as not to produce duplicate Bibliography field values.

 

I was trying to think through whether this problem is a self inflicted wound on my part because of my choice to use extreme source splitting, or whether the problem also exists if you choose to use RM's built-in source templates. The closest thing I could find to addressing the question in any of the Webinars is Webinar #38 Creating a 1940 U.S. Census Citation in RootsMagic from about 39:36 to about 45:20. Bruce explains how to make a copy of one of the built-in source templates and move a couple of fields from the Master Source to the source detail, thereby making sources based on the template be a little bit more lumped. The more fields that are in the Master Source the more split is the source, and the more fields that are in the source details the more lumped is the source. 

 

As Bruce pointed out, if you copy and modify a built-in source template in this manner, you shouldn't have to change any of the sentence templates at all (but I think you might need to change one of the source sentence templates - see below). So I was thinking about the Bibliography sentence template in the Census, U.S. Federal (online images) template that Bruce was using for his example. This sentence template does include jurisdictional information that Bruce moved from the Master Source to the source details in his example. The bibliography sentence doesn't change when you make the source more lumped as Bruce did in his demo. But what happens with the Bibliography list itself with respect to duplicated entries? So I did a more extensive testing, using Bruce's demo as my model for the test.

 

The answer seems to be that you get one bibliographic entry in the Bibliography for each distinct Master Source that is referenced anywhere in the report. That behavior on RM's part is very simple and easy to understand, and on it's face it doesn't seem to have any problems. However, I see at least two problems with it.

 

The first problem is that two different Master Sources can produce identical bibliographic entries. I suspect that in what you might call normal use of the built in source templates, this situation would seldom or never occur. You have to create the problem in some sort of pathological manner such as by being an extreme source splitter. In my testing, I made a copy of a Master Source and made a citation for both the original Master Source and for the copy. Both Master Sources obviously had the same Bibliography sentence and the identical sentences both appeared in the Bibliography. So I guess the solution is "don't do that". But a better solution would be if RM based it's construction of the Bibliography based on Bibliography sentences rather than based on Master Sources.

 

The second problem with RM's behavior in creating the Bibliography is a little more subtle and is exemplified in Bruce's example from the Webinar. After Bruce changes a source template to move a couple of items from the Master Source to the source detail, he doesn't change any of the source sentence templates. Not changing the Footnote sentence template is perfectly fine, and I think also changing the Short Footnote sentence template is perfectly fine. Indeed, being able to move the items from Master Source to the source detail without having to change the source template sentence is the whole point of the exercise.

 

But I think there is a problem with the Bibliography sentence template in the Webinar. Namely, it contains a data field that is moved from the Master Source to the source detail. It seems to me that nothing from the source detail should be in the Bibliography sentence template. I suppose reasonable people could disagree on this point, but if you want any data from the source detail to appear in your bibliographic entries, you will be sorely disappointed. Once again, that's because the bibliographic entries are constructed on the basis of one per Master Source that is referenced in the report rather than on the basis of one per bibliographic sentence that is referenced in the report. So in the example in the Webinar, after moving jurisdiction to the source details and not changing the bibliographic sentence template, one bibliographic entry will appear for that Master Source containing the jurisdiction for the first reference to the Master Source and the other jurisdictions will not appear in the bibliography. I would contend that either all or none of the jurisdictions should appear as bibliographic entries - user's choice depending on whether or not the jurisdiction is in the Bibliographic sentence template. But that's not the way it works. You only get one of the jurisdictions in the bibliography unless you leave the jurisdiction in the Master Source. And you get one of the jurisdictions in the bibliography instead of none unless you remove jurisdiction from the bibliography sentence.

 

Jerry



#10 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 18 November 2015 - 02:57 PM

 

I was wondering if I was remembering this correctly, so I double checked with a little test report. When different Master Sources have the same Bibliography field value, that value actually does appear multiple times in the report's Bibliography page.

...

The answer seems to be that you get one bibliographic entry in the Bibliography for each distinct Master Source that is referenced anywhere in the report. That behavior on RM's part is very simple and easy to understand, and on it's face it doesn't seem to have any problems.

...

The first problem is that two different Master Sources can produce identical bibliographic entries. ... But a better solution would be if RM based it's construction of the Bibliography based on Bibliography sentences rather than based on Master Sources.

 

But I think there is a problem with the Bibliography sentence template in the Webinar. Namely, it contains a data field that is moved from the Master Source to the source detail. It seems to me that nothing from the source detail should be in the Bibliography sentence template. 

 

Wow...thanks for all the analysis, info, etc. I've read over it a time or two or three!

 

I do agree with you that "nothing from the source detail should be in the Bibliography". Bibliography or Source List is *not* the place for details at all.

 

In the meantime I've been letting this dilemma percolate in my head a bit. I had pretty much decided to go with my option 3 approach (use my desired template that displays correctly in RM and live with incorrect punctuation if/when shared via gedcom).  However, while thinking about this, I did do a google search for how to remove duplicate paragraphs in Word and I did find 2 results. I have not tried them yet. One option uses a VB script and the other option used an Advanced Find technique.  So I am about to put a bunch of identical "paragraphs" in a word doc and give this a try. If it works, I may reconsider my approach.



#11 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5997 posts

Posted 18 November 2015 - 03:07 PM

Here is another approach. Use your source templates as a source lumper but when you need to export, convert them to fully split sources. The entire footnote sentence is exported as per your template, no matter how complex, to the TITL tag. How to split hundreds or thousands of sources? Until something is changed in the database design to render it ineffective, this SQLite procedure can do so for all your sources in a matter of seconds: http://sqlitetoolsfo...treme Splitting

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#12 Laura

Laura

    Advanced Member

  • Members
  • PipPipPip
  • 4276 posts

Posted 18 November 2015 - 03:31 PM

I use free form sources and enter data in the Page number in the Source details.

I added the same source to a fact with different entries in rhe Page number of the Source detail.

The source was printed once in the Biblography with no place details.

If I had created a Master source for each household with the data that would otherwise be in the source details in each Master source, two Master sources would have been printed instead of one.

 



#13 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5997 posts

Posted 18 November 2015 - 03:53 PM

If I had created a Master source for each household with the data that would otherwise be in the source details in each Master source, two Master sources would have been printed instead of one.

That's the way it is but, arguably as Jerry has explained, not the way it should be. The Bibliography should contain unique Bibliography entries, no duplicates. If I wish to split Master Sources, I would want to lump my Bibliography entries as in texas_nightowl's example "1841 England, Wales and Scotland Census...". I cannot think of any rationale for duplicating the Bibliography list item when there is only a difference in the Master Source values.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#14 Laura

Laura

    Advanced Member

  • Members
  • PipPipPip
  • 4276 posts

Posted 18 November 2015 - 05:34 PM

Master sources for each household in a census with the source details in the Master source:

1940 U.S. census, Kansas, Seward County, population schedule, NARA microfilm publication T627-1258lWashington, D.C.: National Archives and Records Administration, n.d.: Whatever Township, Page 234, Household 25, family 25, H/H James Doe.

H1940 U.S. census, Kansas, Seward County, population schedule, NARA microfilm publication T627-1258, Washington, D.C.: National Archives and Records Administration, n.d.;  Whatever Township, Page 267, Household 278, familt 279, H/H Gerge Anderson.

When you create a Master source for each household and put different data in each Master source, you have created two separate Master sources.

You can choose to just print 1940 U.S. census, Kansas, Seward County, population schedule, NARA microfilm publication T627-1258lWashington, D.C.: National Archives and Records Administration, n.d.: in the Bibliography or both sources.

 

but, that doesn't change the fact that what is printed in the Bibliography is from two different Master sources even though both Master sources have some of the same data in both.

RM is going to print every Bibliography for every Master source in the report as it should do.

It is the user's choice to decide what those Master sources are going to be.
 



#15 texas_nightowl

texas_nightowl

    Advanced Member

  • Members
  • PipPipPip
  • 79 posts

Posted 18 November 2015 - 05:52 PM

RM is going to print every Bibliography for every Master source in the report as it should do.

It is the user's choice to decide what those Master sources are going to be.
 

 

Respectfully, when generating a report, if 2 bibliography "entries" are identical, there is no reason they shouldn't be combined regardless of coming from separate Master Sources. A Bibliography/Source List should not have multiple exactly identical listings in it.



#16 zhangrau

zhangrau

    Advanced Member

  • Members
  • PipPipPip
  • 1410 posts

Posted 18 November 2015 - 07:11 PM

I suspect that RM uses a key reference value to select & sort the bibliography entries, under the assumption that each Master Source has a unique Bibliography sentence. It would be much more complex programming to compare all of the Bibliography sentences, leaving Footnotes unmerged and then merging the Bibliography entries. Possible? Sure. But definitely a lot more complex to program -- and then subject to minor variations (punctuation, spacing, spelling, etc.) that would cause the comparison routine to still report near-identical Bibliography sentences as separate items in the Bibliography list. The USER must make decisions based on what is actually in the program's design..... I'm a source lumper, and the program design works very well for me.



#17 Laura

Laura

    Advanced Member

  • Members
  • PipPipPip
  • 4276 posts

Posted 18 November 2015 - 07:29 PM

The Bibliography is part of the Master source, so how should Bruce program RM where the Bibliography is seperate from rhe Master souce?

And, would the Bibliographies export and import as part of the source in another program.

I don't think I want to wait for my report while RM compares every Bibliography in every Master source to combine Bibliographies that have the exact same data entries.

The more Master sources I have, the longer that would take.

I have 916 Master sources and 33,565 citations

I have a Master source for a county and state for the 1900 census linked to 750 people so there are also that many source details.

If I do an average of 5 people in one household and put the source details for each household into a separate Master source, I would have 150 Master sources for each household with separate names and with the same Bibliography.

Multiple that by all the other census sources.  Even with not considering other source types, that could be thousands of Master sources for RM to have to compare and combine Bibliographies.
 



#18 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5997 posts

Posted 18 November 2015 - 09:45 PM

In response to zhangrau and Laura, your arguments against unique, non-duplicated items in the Bibliography are specious. Surely you must agree that there should be no repetition in the Bibliography list? Regardless of the degree of lumping and splitting of Master Sources and recognising that within one database there can be a spectrum of where the split may fall. Just because you are comfortable with the way it works for you does not make it the right solution for all.

 

What is so hard to program and time consuming for computers to carry out? The current process has to build a list of Bibliography sentences from the Master Sources and Source Details used by the report through the associated Source Template sentence template for Bibliography. Then it has to sort it alphabetically. There can be no other way to arrive at a sorted list. The hard work has been done. The added step is pretty trivial - output the distinct values from the list.

 

Here's an example, starting from a list with many duplicates: 156,000 surnames in a RootsMagic database. A simple SQLite query returns the alphabetically sorted list of 10,645 unique surnames in ~1/10 second on an older, economy notebook. Admittedly, it takes advantage of a pre-existing table index and the surnames are not as long as a typical Bibliography but one should not worry that the lifting to be done is so heavy as to warrant not doing it. So what if it takes 10 or 100 times longer to process 156,000 citations for unique Bibliographies?

 

There is no change required in the database structure nor in the RM-enhanced GEDCOM export/import. This is purely processing to be done in report generation, the only place Bibliographies are output for consumption by humans who cannot stand needless and wasteful repetition. The program does deal with repetitious footnotes by substituting Short Footnotes and "ibid" - so it should for the Bibliography.

 

Here's the challenging bit of programming to extract the distinct surname values from a table:

SELECT DISTINCT Surname FROM NameTable; 

Did I say "trivial"?

 

I'm inclined to think that this is another piece of unfinished or short-sighted development. Having allowed the user to arbitrarily set the degree to which they lump/split sources, the boundary condition of fully split should have been considered and the need to lump/group Bibliography items would have been apparent. It is somewhat ironic that this issue has received attention as a consequence of people struggling to get around other shortcomings in the Source Template system, i.e., citations that export well, if not identical to report footnotes.   

 

@Renee, if an enhancement request to eliminate repetition in Bibliography lists is not in the system, it should be now! 


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#19 Laura

Laura

    Advanced Member

  • Members
  • PipPipPip
  • 4276 posts

Posted 18 November 2015 - 10:35 PM

Tom, as long as there is no problem with gedcoms, and it doesn't take forever to generate the report, I have no problem with it.

#20 zhangrau

zhangrau

    Advanced Member

  • Members
  • PipPipPip
  • 1410 posts

Posted 18 November 2015 - 11:21 PM

Tom, as long as there is no problem with gedcoms, and it doesn't take forever to generate the report, I have no problem with it.

 

Ditto.

 

I have a much larger database than Laura, with nearly 400K Individuals, 13K Sources, and 2.2 million Citations. I wait frequently for "background" operations -- creating & sorting indexes, database tools, backups, etc. My larger reports can take several hours to generate on a Win7 (64-bit) computer with 12 GB RAM and 300 GB free hard drive space (these are a good time to run errands or sleep !!!).