Jump to content


Photo

No Source Template Fields in GEDCOM


  • Please log in to reply
35 replies to this topic

#1 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 10 May 2010 - 08:27 PM

I recently re-exported a GEDCOM file and noticed that source template fields no longer appear in GEDCOM source records. The variable fields are missing. For example, an old GEDCOM has:

0 @S4074@ SOUR
...
1 TITL 1930 US Census, Allegheny County, Pennsylvania, population schedule, [C
2 CONC ivilDivision], enumeration district (ED) [ED], [PageID], [HouseholdID], [
...

The new GEDCOM has:

0 @S4074@ SOUR
...
1 TITL 1930 US Census, Allegheny County, Pennsylvania, population schedule, , ; d
2 CONC igital images, ...

All the variable fields like [CivilDivision], [ED], [PageID] are missing. I have the option for "Extra details (RM Specific)" enabled when I export. I have not exported a GEDCOM in a while, so I cannot say in which version this stopped working. I'm running 4.0.8.4.

#2 kbens0n

kbens0n

    Advanced Member

  • Members
  • PipPipPip
  • 3458 posts

Posted 10 May 2010 - 10:24 PM

It'll most likely be helpful to know which version of RootsMagic 4 that you're running?
4.0.8.3 added some GEDCOM template fixes/enhancements as noted in the blog:

http://blog.rootsmagic.com/?p=669

---
--- "GENEALOGY, n. An account of one's descent from an ancestor who did not particularly care to trace his own." - Ambrose Bierce
--- "The trouble ain't what people don't know, it's what they know that ain't so." - Josh Billings
---Ô¿Ô---
K e V i N


#3 RootsMagician

RootsMagician

    Administrator

  • Admin
  • PipPipPip
  • 826 posts

Posted 11 May 2010 - 10:19 AM

The template fields were removed from the TITL, AUTH and PUBL tags because those tags are what other programs read when they import sources.

The template fields are still provided in the custom tags RM uses so that RM can read the template information back in.
RootsMagician

#4 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 11 May 2010 - 01:08 PM

I write third-party software to take advantage of the RM templates as exported via GEDCOM. I understand that templates are not part of the GEDCOM standard and therefore they are handled in a proprietary manner. However I hope you will reconsider removing the variable fields from the TITL, AUTH, PUBL and _BIBL tags. I depend on those fields to reconstruct source citations and bibliography entries. Perhaps you could keep them when the "Extra details (RM Specific)" option is enabled. Or perhaps you could provide proprietary tags that would duplicate the TITL, AUTH, PUBL and _BIBL tags, except they would include the variable fields.

What you have left in the TITL tag is a lot of empty spaces with punctuation, for instance:

1 TITL 1930 US Census, Allegheny County, Pennsylvania, population schedule, , ; d
2 CONC igital images, ...

How can another program construct a meaningful title without some way of knowing where in the string to substitute actual field values from the source record and source citation references?

#5 RootsMagician

RootsMagician

    Administrator

  • Admin
  • PipPipPip
  • 826 posts

Posted 13 May 2010 - 04:39 PM

How can another program construct a meaningful title without some way of knowing where in the string to substitute actual field values from the source record and source citation references?

You can reconstruct them using the extra fields RM puts in the GEDCOM when you have it include the RM specific data. That includes the template and all the fields with their data delimited for you already.
RootsMagician

#6 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 13 May 2010 - 10:06 PM

That includes the template and all the fields with their data delimited for you already.


Thanks for responding. The template is what I'm looking for, but I cannot find it in the GEDCOM. I have a check mark in "Extra details (RM Specific)" when I export a GEDCOM. For example, consider a census fact for an individual:

 1 CENS
 2 DATE 2000
 2 PLAC Lansing, Michigan
 2 SOUR @S4@
 3 PAGE Sample Civil Division; Sample ED; Sample Page; Sample Household; Sample Person; Sample Access Type; 1 January 2000
 3 _TMPLT
 4 FIELD
 5 NAME CivilDivision
 5 VALUE Sample Civil Division
 4 FIELD
 5 NAME ED
 5 VALUE Sample ED
 4 FIELD
 5 NAME PageID
 5 VALUE Sample Page
 4 FIELD
 5 NAME HouseholdID
 5 VALUE Sample Household
 4 FIELD
 5 NAME Person
 5 VALUE Sample Person
 4 FIELD
 5 NAME AccessType
 5 VALUE Sample Access Type
 4 FIELD
 5 NAME AccessDate
 5 VALUE 1 January 2000

The source record appears as:

 0 @S4@ SOUR
 1 ABBR Sample Census
 1 TITL Sample Year, Sample Jurisdiction, sample schedule, , ; sample item type
 2 CONC , <i>Sample Website</i> (Sample URL : accessed ); Sample Credit Line.
 1 _SUBQ Sample Year, Sample Jurisdiction, Sample Schedule, , , .
 1 _BIBL Sample Country. Sample Jurisdiction. Sample Year, sample schedule. Samp
 2 CONC le Item Type. Sample Website. Sample URL : .
 1 _TMPLT
 2 TID 43
 2 FIELD
 3 NAME Country
 3 VALUE Sample Country
 2 FIELD
 3 NAME CensusID
 3 VALUE Sample Year
 2 FIELD
 3 NAME Jurisdiction
 3 VALUE Sample Jurisdiction
 2 FIELD
 3 NAME Schedule
 3 VALUE Sample Schedule
 2 FIELD
 3 NAME ItemType
 3 VALUE Sample Item Type
 2 FIELD
 3 NAME WebSite
 3 VALUE Sample Website
 2 FIELD
 3 NAME URL
 3 VALUE Sample URL
 2 FIELD
 3 NAME CreditLine
 3 VALUE Sample Credit Line

In previous RM versions, I used the TITL tag, which contained the variable template fields. I replaced the variable fields with the corresponding field values (NAME/VALUE pairs) from both the source citation and from the source record itself.

Without the variable fields, I cannot reconstruct a proper title for the source citation (nor for the bibliography). Is there another tag that holds the template with the variable fields? The TITL tag as it is now cannot be used to construct a meaningful title, even in conjunction with the PAGE tag. In effect some important data has been discarded.

The GEDCOM architecture is meant to use one source record, where each source citation is simply a different "page" in the same source. With source templates, a source record can actually be more than one source, through the substitution of variable fields. If you do not provide the variable template fields, you will have to generate multiple source records, one for each source citation that references that source. The title of the source can change based on the source citations that reference it (via source templates). For instance the TITL now does not include the [CivilDivision] (among others), which would change on a per-citation basis. Without the variable field, you need a separate source record for each different [CivilDivision] referenced.

I hope you will provide the full source template with variable fields, either in the TITL tag, or in another proprietary tag.

#7 kbens0n

kbens0n

    Advanced Member

  • Members
  • PipPipPip
  • 3458 posts

Posted 14 May 2010 - 01:41 AM

The template is what I'm looking for, but I cannot find it in the GEDCOM.

*SNIP*

 1 _TMPLT
 2 TID 43

*SNIP*

I'm not a programmer, but I'm guessing that you mean the GEDCOM now isn't enough because for any source templates that are customized within the database ... RM goes the route of a template <Export> function using a button (instead of additional GEDCOM tag representations) and you would thusly need access to those exported Source Template file(s) in order to parse them for the variables decided upon by the user, instead of the defaults represented in, say, the Rootsmagic.st file (for example) below:

  </Template>
  <Template Id="43">
    <Name>Census, U.S. Federal (Online images)</Name>
    <Description>online commercial site; Evidence style; Place and year as lead source list elements; Census ID as lead full note element; Uses :Abbrev function in several fields to substitute abbreviations if entered; if no abbrev data entered, field used as is; Credit line not included in Source List. For 1790 to 1930 (and beyond)</Description>
    <Category>[EE, QC-6, p 240; QS, front cover] </Category>
    <Footnote>[CensusID], [Jurisdiction]&lt;, [Schedule:Lower]&gt;, [CivilDivision]&lt;, enumeration district (ED) [ED]&gt;&lt;, [PageID]&gt;&lt;, [HouseholdID]&gt;, [Person]; &lt;[ItemType:Lower]|digital images&gt;, &lt;i&gt;[WebSite]&lt;/i&gt; ([URL] : &lt;[AccessType]|accessed&gt; [AccessDate])&lt;; [CreditLine]&gt;.</Footnote>
    <ShortFootnote>[CensusID], [Jurisdiction:Abbrev]&lt;, [Schedule:Abbrev]&gt;, [CivilDivision:Abbrev]&lt;, ED [ED]&gt;&lt;, [PageID]&gt;, [HouseholdID:Abbrev], [Person].</ShortFootnote>
    <Bibliography>&lt;[Country]. &gt;[Jurisdiction:Reverse]. [CensusID]&lt;, [Schedule:Lower]&gt;. &lt;[ItemType:Caps]|Digital images&gt;. [WebSite]. [URL] : [AccessDate:Year].
    </Bibliography>
    <Field>

      *SNIP*

    </Field>
  </Template>

---
--- "GENEALOGY, n. An account of one's descent from an ancestor who did not particularly care to trace his own." - Ambrose Bierce
--- "The trouble ain't what people don't know, it's what they know that ain't so." - Josh Billings
---Ô¿Ô---
K e V i N


#8 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 14 May 2010 - 10:25 AM

I think the GEDCOM should contain everything needed to reconstruct the data. It is not just user-created templates, but standard templates as well.

Templates add a layer of complexity, as I alluded to. What was a single source record can now actually be several different sources. The source citation contains fields that will actually change the title of the source. Several different citations can point to the same source record, and because it is a template, the information it contains can be different for each citation. That's why you cannot simply remove the variable fields from the template title. This discards important information. Without the variable fields, you would have to generate several source records, one for each unique source citation. That would keep the GEDCOM file consistent with the GEDCOM standard.

I have a GEDCOM to HTML conversion program. After I switched to using RM, I found my source citations displayed the variable template fields rather than the actual values. So I modified my program to support RM's source templates. If the GEDCOM exported from RM includes templates, it should provide everything needed to properly display source citations. Otherwise, the GEDCOM should be standard, meaning RM must generate all the source records necessary. Rather than a template for a given source, there may need to be several source records generated.

#9 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 14 June 2010 - 06:54 PM

I hope this issue is being addressed. Without valid source citations, I cannot use the family data I've exported to a GEDCOM file and I'm unable to update my family website.

#10 Alfred

Alfred

    Advanced Member

  • Members
  • PipPipPip
  • 5734 posts

Posted 14 June 2010 - 08:09 PM

I cannot figure out what is missing.
I tried creating a custom source template by copying one of the book templates, then I gave it new name by adding TEST to the beginning of it.
I used it for a source for my birth fact and filled in a bunch of the fields with trivia.
I then exported a GEDCOM of the database and looked at the GEDCOM file with WordPad.
The citation page number, reference number and source number, @S7@ are listed under my birth fact.
I pressed the [Ctrl]+[f] combination to find and entered the source number, @S7@ and found the source listed farther down with the other sources.

The footnote, short and bibliography texts were there for those programs which do not understand these templates, follower by the template fields and what I had entered into each.

When I imported that GEDCOM into an empty database, I got the whole thing which looks just like it did in the original database.

I am using RootsMagic 4.0.9.3 right now.
Alfred

#11 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 15 June 2010 - 06:54 PM

What's missing are the template fields in the source record, where you would substitute the field NAME/VALUE pairs from the source citation. If there are any NAME/VALUE pairs in the citation under your birth fact, you will not find those field names down below in the source record. The "Source Details" values are kept with each source citation, not in the source record.

As in my previous example, I created a sample census citation. Its title in a resulting GEDCOM file is:

1 TITL Sample Year, Sample Jurisdiction, sample schedule, , ; sample item type
2 CONC , <i>Sample Website</i> (Sample URL : accessed ); Sample Credit Line.

Notice the , , ; near the end of the first line. In prior versions there were template fields there, like [CivilDivision]. Those have now been removed, leaving empty punctuation. These template fields were formerly substituted with values from each citation that referenced the source template. Thus each source template could represent numerous different sources. Without the template fields, there is no way to reconstruct these sources. Important information is missing, which makes the source effectively meaningless.

RootsMagician has not acknowledged this is a problem, and there have been a number of RM updates since. So I'm left wondering if the issue is being addressed, or if I'm missing something.

Does anyone have experience in reverting to a previous version of RM? I could use a version that generated the full source citations. I'm concerned that newer RM versions may have made database schema changes that affect my data. In other words, I'd use an older RM version on a database updated for the most recent version.

#12 kbens0n

kbens0n

    Advanced Member

  • Members
  • PipPipPip
  • 3458 posts

Posted 15 June 2010 - 08:05 PM

RootsMagician has not acknowledged this is a problem, and there have been a number of RM updates since. So I'm left wondering if the issue is being addressed, or if I'm missing something.

Actually Bruce clearly addressed it in his first response. He expressly stated that he removed them from those standard (not custom) tags to avoid incompatibility with other software.

Does anyone have experience in reverting to a previous version of RM? I could use a version that generated the full source citations. I'm concerned that newer RM versions may have made database schema changes that affect my data. In other words, I'd use an older RM version on a database updated for the most recent version.

I don't have any experience, but I'm sure you could revert if you have an earlier installer version (I have 4.0.7.1 - not sure if that's far enough back). Unfortunately, as you are already aware, you'd likely need to create fresh databases with that older version and populate them in order to avoid schema differences and that it would leave you unable to avail yourself of bugfixes and feature enhancements going forward.

---
--- "GENEALOGY, n. An account of one's descent from an ancestor who did not particularly care to trace his own." - Ambrose Bierce
--- "The trouble ain't what people don't know, it's what they know that ain't so." - Josh Billings
---Ô¿Ô---
K e V i N


#13 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 15 June 2010 - 08:36 PM

I think the RM specific export will include the full template for a user defined template (as of 4.0.8.3) but not for a built-in template (without checking the current veracity of this statement). If your source uses a user-defined template, the GEDCOM should contain all you need to construct your full source citation. If it uses a built-in template, the GEDCOM is incomplete, as you have complained. Knowing the template ID for a built-in template, you would have to look it up in the RootsMagic.st file; then you have all the information needed to construct your full source citation.

Should those template ID numbers ever change, then you may be screwed. Interestingly, Bruce has changed RootsMagic.st and the template changes propagate (to some degree, at least) to an older database when it is opened by the new version of RM.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#14 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 16 June 2010 - 08:17 AM

Actually Bruce clearly addressed it in his first response. He expressly stated that he removed them from those standard (not custom) tags to avoid incompatibility with other software.


Removing those template fields is the problem. The following title is now even more incompatible with other software (red fields added for emphasis).

1 TITL Sample Year, Sample Jurisdiction, sample schedule, [missing], [missing]; sample item type
2 CONC , <i>Sample Website</i> (Sample URL : accessed [missing]); Sample Credit Line.

I could read the template ID (TID) field and then parse RootsMagic.st to reconstruct the template, as you suggested May 14. Traditionally GEDCOM files have been self-contained "transmissions." Certainly having to parse an accompanied XML file along with a GEDCOM file would be incompatible with all other software. The two files could also get out of synchronization, meaning the XML could change long after the GEDCOM was generated. So I don't think that is a viable solution.

#15 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 16 June 2010 - 09:05 AM

The template fields were removed from the TITL, AUTH and PUBL tags because those tags are what other programs read when they import sources.

The template fields are still provided in the custom tags RM uses so that RM can read the template information back in.


Yes, the field NAME/VALUE pairs are provided and RM can export and re-import its own GEDCOM. However I suspect that RM ignores the TITL/AUTH/PUBL/_BIBL tags and instead reads RootsMagic.st in order to reconstruct the template. In the future when RM attempts to read an old GEDCOM, and RootsMagic.st has since changed, RM will also be unable to reconcile the older NAME/VALUE pairs with a new (modified) template.

I notice that the "Master Source" template fields are not in the TITL/AUTH/PUBL/_BIBL tags. You did have the "Source Detail" template fields, but have since removed them. If you remove them, you will have to substitute those fields with the actual values from the Source Details too. Not only that, but for every reference where the Source Details are not exactly identical, you must generate a completely new source record. You cannot simply remove the Source Detail template fields from the the TITL/AUTH/PUBL/_BIBL tags and give other software no way to reconstruct the template. I suspect this design is the reason there are problems with RM generating websites with mismatched citations/footnotes/bibliography (http://forums.rootsm...ting-a-website/, http://forums.rootsm...bsite-creation/)

#16 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 16 June 2010 - 09:12 AM

I could read the template ID (TID) field and then parse RootsMagic.st to reconstruct the template, as you suggested May 14. Traditionally GEDCOM files have been self-contained "transmissions." Certainly having to parse an accompanied XML file along with a GEDCOM file would be incompatible with all other software. The two files could also get out of synchronization, meaning the XML could change long after the GEDCOM was generated. So I don't think that is a viable solution.

Short of persuading Bruce to revise the GEDCOM export to suit your needs, here are a couple of alternative suggestions:

1. Uncheck the Extra Details (RM specific) checkbox when you do your GEDCOM export. I think all your citations and sources are converted to free form. That should be easy to handle.
2. Convert all your sources to user-defined templates. Check the Extra Details (RM specific) checkbox when you do your GEDCOM export. Your GEDCOM should now contain all the required information without having to refer to RootsMagic.st.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#17 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 16 June 2010 - 09:35 AM

Thanks TomH, but with RM Specific unchecked, the title still contains empty punctuation where the Source Detail fields should appear. And, I can't ask the users of my GEDCOM-to-HTML program to convert all their sources to user-defined templates.

Besides, it's not a matter of changing the GEDCOM export to suit my needs. I've tried to argue the point from several different angles that the GEDCOM file is indeed invalid. It seems however that Bruce and others don't see a problem. But if they do, I'd at least like to know whether they are addressing it.

#18 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8457 posts

Posted 16 June 2010 - 09:53 AM

To update you on this situation, the RootsMagician is looking into this, but we don't have a good solution yet.
Renee
RootsMagic

#19 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6254 posts

Posted 16 June 2010 - 10:15 AM

Thanks TomH, but with RM Specific unchecked, the title still contains empty punctuation where the Source Detail fields should appear.

I agree that there is a bug in the template to free-form conversion routine.

And, I can't ask the users of my GEDCOM-to-HTML program to convert all their sources to user-defined templates.

I did not realize that your software was for other than personal use. What is the URL of your software website? It sounds potentially interesting.

Besides, it's not a matter of changing the GEDCOM export to suit my needs. I've tried to argue the point from several different angles that the GEDCOM file is indeed invalid. It seems however that Bruce and others don't see a problem. But if they do, I'd at least like to know whether they are addressing it.

I'm not sure if the way proprietary additions use a GEDCOM makes the file any more invalid than having proprietary additions at all. If there is a GEDCOM principal that requires all GEDCOMs to be complete and self-contained, then maybe so. If the way RootsMagic exports user-defined source templates is acceptable, then it is not much of a stretch to do the same thing with built-in templates. You should have a look at whether the current method of exporting user-defined templates meets your needs. If so, then I think Bruce has a fairly easy way to make the GEDCOM exports more self-contained than they are at present for the transmission of sources and citations using built-in templates.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#20 TreeTraverser

TreeTraverser

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 16 June 2010 - 01:48 PM

Thanks Renee. I was looking for some assurance that you agree it is a problem. I've gone to a lot of trouble switching my family data to RM and have been waiting for source citations to be corrected before I can update my family tree website.

Since the source TITL/AUTH/PUBL/_BIBL tags are not complete, I consider the GEDCOM invalid. Not invalid according to the GEDCOM standard, but invalid in the sense the data are incomplete. A simple solution would be to export the template using user-defined tags, for instance _TITL and _BIBL under _TMPLT in the source record. (Although there is still the problem of missing data in the actual TITL tag etc.)

I added RootsMagic support to my program, GED-GEN, for a forthcoming release. It's in the form of an option that enables RootsMagic-style source templates. I took care to generate unique footnotes, recognize identical footnotes (for ibid processing), eliminate duplicate bibliography entries and create new ones caused by a single source record being used as multiple different sources. Then I noticed all the empty punctuation fields and discovered important template fields had been removed.

I copied a census template to make a user-defined template and I get the same problem:

 0 @S5@ SOUR
 1 ABBR Test-Name
 1 TITL Test-Year, Test-Jurisdiction, test-schedule, , ; test-item, <i>Test-Web
 2 CONC site</i> (Test-URL : accessed ); Test-Credit.
 1 _SUBQ Test-Year, Test-Jurisdiction, Test-Schedule, , , .
 1 _BIBL Test-Country. Test-Jurisdiction. Test-Year, test-schedule. Test-Item. T
 2 CONC est-Website. Test-URL : .
 1 _TMPLT
 2 TID 10000

Or should I export the user-defined template from the Source Templates dialog and create an .rmst file? Again, having an accompanied XML file just to reconstruct the title of a source record is certainly contrary to the GEDCOM standard.