Jump to content


Photo

convert one census format to another


  • Please log in to reply
9 replies to this topic

#1 NEreswearcher

NEreswearcher

    Advanced Member

  • Members
  • PipPipPip
  • 38 posts

Posted 05 March 2018 - 09:50 AM

I am trying to clean up all my mistakes and poor choices that I have made in RM in the past. None of these issues should reflect poorly om RM as they are all of my own doing but I hope that someone may have also run into this situation and can provide a brilliant solution.

I will use census records as an example of my problem. When entering a citation and source for a census record I have not always made the right choice regarding the correct template to use. There appear to be about 20 different templates for a US Federal census. I have not always chosen wisely. I am trying to get a printable genealogy and the footnotes and bibliography are not always correct because of some missing data.

I have looked at all the various ways in RM to correct a census source to be in the correct template but so far I have found nothing except going manually through every source and citation to make it correct. The merge function works but data is lost during the merge. I have looked at SQL and found that for every citation there is in a blob field with XML code for the fields for that source. Unless I wright a program to parse the XML and then match that up with what should be in the correct format for the new source template there is no SQL solution.

Before I continue with this manual process I was hoping someone may have a better solution.



#2 Nettie

Nettie

    Advanced Member

  • Members
  • PipPipPip
  • 1557 posts

Posted 05 March 2018 - 10:53 AM

This can be a big discussion because many look at how to do this their way. 

okay, 

1. Are your census citations/Sources

by individual

by same template per census year [example: all 1850 census records found in one template no matter, what state, what county, etc.]

free form

 or.......

Back when RM 3 was written, I was going thru this same process.  I set up my own way to do censuses that made sense to me.  Others have done it different.  

"Using one census template for each census year."  From <http://forums.rootsm...mat-to-another/>

 

In RootsMagic V 5 topics under Tips and Hints

You will find this article:  

"Census Records using one record per census year"

From <http://forums.rootsm...er-census-year/>

Tom H. said the same thing another way "I think you mean "One Master Source per census year", for which there would be many citations (source lumping)."

From <http://forums.rootsm...er-census-year/>

 

Read some of the many discussion since RM3 then 4 about how to do this.  

The above templates for US Census are copied and changed  as mention in the article.  I did the same thing for a book/Journal with different Chapters etc....


Genealogy:
"I work on genealogy only on days that end in "Y"." [Grin!!!]
from www.GenealogyDaily.com.
"Documentation....The hardest part of genealogy"
"Genealogy is like Hide & Seek: They Hide & I Seek!"
" Genealogists: People helping people.....that's what it's all about!"
from http://www.rootsweb....nry/gentags.htm
Using FO and RM since FO2.0 


#3 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3089 posts

Posted 05 March 2018 - 12:15 PM

I think the basic answer for most users is that the only solution is manual correction.

 

Since you have obviously played with SQLite and understand the XML blob fields, I might offer one suggestion. Something I have done a little bit of with SQLite is to change instances of one source template type to another. That can obviously lose data. But with a little creativity, you can mitigate the loss of data. And changing from one source template type to another doesn't require any parsing of XML - which is a pain in general and which is especially difficult in an SQLite environment.

 

So let's think in terms of merging sources, even though that's not actually what we are going to do. So make a source template of your own design, even if the template of your own design is simply a copy of an existing RM template. This template of your own design will be the template INTO which you merge. Add to the template of your own design any data fields that are in the template FROM which you merge. The fact that the data element is in the template does not mean that the data element has to appear anywhere in your citation sentence. At this point, we are just making sure no actual data is lost.

 

Now either for one particular master source or for all master sources of your FROM type, change the source template type to the INTO type - using SQLite. Even though this is an update, it's very simple and very safe and does not involve any blob fields or parsing of XML. Obviously, do this first in a test database (which can be a copy of your production database), have lots of backups, etc.

 

Do the rest of the work in RM. Go to the Master Source or Master Sources which have the problem and do any needed clean up. You may need to copy and paste data from one field to another in the Master source to get it to the correct place, but the needed data will be there because we made sure that the INTO template included all the data elements that were in the FROM template. This is a manual process, but it means that RM is doing all the XML parsing for you.

 

When you are done with your project, you can leave your new template alone if you wish. That extra data in the source that is not in the citation sentence won't hurt anything. Or you can remove the unused data elements from the template. Your choice.

 

I would emphasize again that this is not a fully automated solution. Rather, it's a solution that can make your manual data entry much easier. In particular, it should mean that you don't have to go back and find all the citations you need to change and delete them and replace them with new citations. The existing citations will be changed in place.

 

Jerry



#4 NEreswearcher

NEreswearcher

    Advanced Member

  • Members
  • PipPipPip
  • 38 posts

Posted 05 March 2018 - 04:25 PM

Jerry Thank you for your response. I will need to read that several times before I attempt anything. It appears that I may be able to at least save some of the manual work.

 

Nettie also made me think of what my goal is with citations. To me there are two goals:

1. Make my work more credible by having source citations for all my facts and relationships.

2. Make my citations useful in finding the original source where I found the data.

    a. Should I rely on using a search engine to find the data or,

    b. make my citation specific enough that I could fill out a NARA form to request the particular page in question.

    c. no decision yet
 

2b would require much more data than 2a since the search engines will use their index to find the record. I saw in another post about the citation data that appears with the results of both Ancestry and Familysearch and I would want to accommodate what they provide.

 

One of my issues with the source templates is that if I don't have data for a particular field then unpleasant things happen in the footnote, short footnote or bibliography.

 

if for instance I have a book and have only the last name of the author then in the bibliography it comes out

 

Last name,,rest of bibliography.

 

If I have both first and last name it would come out

 

Last name,First name,rest of bibliography.

 

I don't think the extra commas make it any clearer. Also in the footnote if for instance I don't enter the page the footnote then has the field name [Page] in the footnote (this does not always occur but I haven't determined where this is happening).

 

Since I cannot edit the built in templates I cannot correct this situation for missing data. 

I was thinking the following might work as a TEMPORARY Fix.

Copy the book template to a new template of mine that I can edit. In SQL the book template has an id of 14 and my new template id is 450

I can then list all the records in the source table with a templateid of 14 and the change the 14 to a 440

Then I would go in and edit the sentence structure for my new template to look the way I want.



#5 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3089 posts

Posted 05 March 2018 - 06:20 PM

Since I cannot edit the built in templates I cannot correct this situation for missing data. 

I was thinking the following might work as a TEMPORARY Fix.

Copy the book template to a new template of mine that I can edit. In SQL the book template has an id of 14 and my new template id is 450

I can then list all the records in the source table with a templateid of 14 and the change the 14 to a 440

Then I would go in and edit the sentence structure for my new template to look the way I want.

 

Essentially, that's what I was talking about, although your explanation is simpler and more straightforward than mine.

 

You don't really have to "list all the records in the source table". All you really have to do is execute the following SQL.

UPDATE SourceTable SET TemplateID = 440
WHERE TemplateID= 14;

However, I'm suspicious of your template numbers. In one sentence you mention 450 and in another sentence you mention 440. The numbers for user defined templates start at 10000, and copies of built-in templates are treated as user defined templates. So your user defined template should have a much larger number than 440 or 450.

 

Jerry



#6 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3089 posts

Posted 05 March 2018 - 07:05 PM

One of my issues with the source templates is that if I don't have data for a particular field then unpleasant things happen in the footnote, short footnote or bibliography.

 

if for instance I have a book and have only the last name of the author then in the bibliography it comes out

 

Last name,,rest of bibliography.

 

If I have both first and last name it would come out

 

Last name,First name,rest of bibliography.

 

I'm absolutely anal about punctuation making logical and mathematical sense, much to the chagrin of every English teacher and librarian in the world with whom I have ever discussed the subject. For example, for a quoted sentence the period or question mark which terminates the sentence must be inside the quote marks, not outside it. For another example, the Oxford comma is absolutely essential.  Without 100% compliance with the use of the Oxford comma, the phrase "Surely, goodness and mercy will follow me ...." could be interpreted as a list of three items - the first of which is surely, the second of which is goodness, and the third of which is mercy. Under these circumstances, I understand the meaning of goodness and mercy, but just what is "surely". (... and don't call me Shirley!)

 

All of which brings us to your problem of the the author with only a last name creating a ludicrous looking bibliography sentence. A human being creating the bibliography sentence might be able to come up with some sort of compromise that doesn't look ludicrous. But when computers are making up the sentences, the underlying punctuation has to make logical and mathematical sense or you can get ludicrous results when unusual conditions or boundary conditions are encountered.

 

If you look at some of RM's built-in templates, they can seem to differ in very minor and almost trivial ways. If you look very closely, some of the differences seem to be the result of attempts to deal with these kinds of subtle punctuation issues. For example, in a bibliography sentence the author's last name is supposed to precede the author's first name unless the author is an organization or group. John Doe is supposed to be listed in a bibliography as "Doe, John" without the quotes rather than as "John Doe" without the quotes, but the Boondock County Genealogical Society as an author is supposed to be listed as "The Boondock County Genealogical Society" without the quotes rather than as "Society, The Boondock County Genealogical" without the quotes. That is why RM has a different book template for an individual author than for an organization or group as an author.

 

I use source templates of my own design to try to get around these kinds of problems in a couple of ways. One is that I use Value Switches to handle things like the ordering of first and last names when the author is a real person and when the author is an organization or a group. Another is that I use multiple levels of punctuation such as both commas and semi-colons. For example, in your case of authors with or without a last name, my structure would be something like

author; title; publishing information

This would work satisfactorily for any of the following situations.

John Doe; The History of the Doe Family; self published, 1974.
Doe, John; The History of the Doe Family; self published, 1974.
Doe; The History of the Doe Family; self published, 1974.

My templates are not perfect, and I'm constantly tinkering with them to address new situations that arise. And I'm aware that my approach doesn't quite follow all the punctuation standards. But I think the punctuation standards are illogical and mathematically incorrect. I would rather be logical and mathematically correct than to follow illogical and mathematically incorrect standards. And having said all that, I live in fear that an English teacher is going to read this message and give it a C- grade (or worse!).  :)

 

Jerry



#7 NEreswearcher

NEreswearcher

    Advanced Member

  • Members
  • PipPipPip
  • 38 posts

Posted 07 March 2018 - 12:40 PM

Jerry

Your comments are well taken. I'm sorry about any typos of the numbers. When I actually do this the numbers will be different anyway.

 

For the time being I am going to put the issue of commas aside as I can live with them.

 

My bigger problem is this. I also use custom fact types but I don't believe they have anything to do with the problem I am going to outline.

 

I have a hard copy real estate deed.

 

I entered the info on a custom fact called real estate because the wording on the Property Fact type didn't suit the situation.

 

I entered a citation with the citation type of deeds Local registers.

 

Because this deed was obtained several years ago I don't have the current volume/page, or item of interest. I left those fields blank. When entering the data on the right portion of the screen it displays what the footnote, short footnote and the bibliography will look like and it all looks ok with no reference to the missing data. At that point I thought I was ok.

 

(I tried providing a screen print but it was rejected)

 

When I generated a narrative report report the footnote did not look the same as what I had seen earlier. It looked like this:

 

        9. Town of Coventry Connecticut, , [VolumePage], [ItemOfInterest], [RecordDate]; Coventry Land Records,, 1712 Main St, Coventry , Connecticut 06238.

 

This also occurs on a family group sheet report. If I place a space in those fields then the field names disappear in the footnote. I am at a loss as to why the display on the citation screen is ok and in the reports they contain the field names.

 

Obviously the correct solution is for me to provide all the data for every citation.

 

I can't see any solution other than to review and edit every fact and citation that I have. Whew - I am already tired.



#8 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3089 posts

Posted 07 March 2018 - 03:40 PM

If I am understanding your situation correctly, the solution in absence of having all the data would be to copy the built-in template you are using, change the footnote sentence in the copied template to put <angle brackets> around the variables that might be missing, and use SQLite to change change all occurrences of the built-in template number in the SourceTable with the number of the new template that you made. This number will be 10000 or greater. Then, moving forward you should use your new template rather than the built-in template so you should never again have to use SQLite to solve the problem for the this particular template.

 

Jerry



#9 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 5876 posts

Posted 08 March 2018 - 01:52 PM

If memory serves correctly, the " [VolumePage], [ItemOfInterest], [RecordDate]" part of the footnote is due to the absence of any corresponding XML tag in the Fields value of the citation, even a null tag, "<VolumePage/>". When you edit the citation, RM will create tags in the Fields field for all the defined variables, whether or not a value has been entered. But when you convert a source from one Source Template to another with different fields, the new fields are missing from the Master Source "Fields" field in the SourceTable and the Source Details "Fields" field for all of its citations in the CitationTable. I think the developers had the names of those missing fields outputted in the sentence to assist with debugging. I've suggested years ago that this debugging feature be removed because I don't see that it does anything for the average user but gets in the way of the power user.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#10 NEreswearcher

NEreswearcher

    Advanced Member

  • Members
  • PipPipPip
  • 38 posts

Posted 11 March 2018 - 09:31 AM

Tom and Jerry,

 

OK I couldn't resist the temptation with the names.

 

It looks like I can use both of your suggestions one to permanently fix a template and second to fix, apparently an anomaly in RM7. I was still getting the field names out and once I saw Tom's explanation it was easy to find and fix my problem. In this case all I needed to do was create a second citation and then delete all references to the first citation. Problem Solved.

 

Jerry - I will be using your solution to permanently correct a few templates and this will take me a bit longer since I may have quite a few but at least I have a solution that will work.

 

Thanks