Jump to content


Photo

Deleting Sources - Incorrect Verification Dialog


  • Please log in to reply
23 replies to this topic

#1 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 08 December 2014 - 07:00 PM

I have been gradually re-doing all my sources, which were all originally imported using the 'Free-form' template as part of a GEDCOM import using a database that I exported from Genbox a while ago.  I create the new source using the appropriate template, then add new citations as required and delete the old citiations tied to the old free-form source.  Sometimes when deleting the old source, I have been erroneously getting a dialog saying that there are still people, families or facts using the source.  I run a source report to see what citations are still attached, and in each case, the report shows a citation attached to the person name, but the citiation was deleted previously and does not show in the list of sources for the person's name (or any other events attached to that person). I've tried cleaning and re-indexing, etc., but has no effect.  Note that this does not prevent me from deleting the source if I answer 'yes' at the confirmation dialog, but wastes time to run the report, etc., and not sure if it's leaving a bunch of orphaned citiations out there. 



#2 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 08 December 2014 - 07:52 PM

I typically instead of deleting the source merge the new source into the old one. This will ensure that all citations are kept where they are. 



#3 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 12:45 AM

Yes, agree.  However, I found that merging works OK for things that only have one field in the source detail (like books with page numbers), but not so well for more complicated citations like census sources.  If the source detail has several fields, i.e. page, ED, subject, etc., all of the source detail included with the free-form source citation is lost during the merge.  (In the free-form citiations, all the detail is combined in a single 'page number' field.)  

 

Unfortunately, I can't see any good way of overcoming this, since there's no way that RM could tell how to parse the combined data into the various fields of the new source detail (at least as long as we continue to have to deal with GEDCOM).  However, the merge dialog should probably warn you if it expects data will be lost during the merge.



#4 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3609 posts

Posted 09 December 2014 - 08:01 AM

Yes, agree.  However, I found that merging works OK for things that only have one field in the source detail (like books with page numbers), but not so well for more complicated citations like census sources.  If the source detail has several fields, i.e. page, ED, subject, etc., all of the source detail included with the free-form source citation is lost during the merge.  (In the free-form citiations, all the detail is combined in a single 'page number' field.)  

 

Unfortunately, I can't see any good way of overcoming this, since there's no way that RM could tell how to parse the combined data into the various fields of the new source detail (at least as long as we continue to have to deal with GEDCOM).  However, the merge dialog should probably warn you if it expects data will be lost during the merge.

 

Strictly speaking, the free form template only has one field, even if that one field contains multiple pieces of data such as page, ED, subject, etc. Namely, it has a field called Page. So here is a trick that might help out a bit.

 

First of all, it's important for the new source to use a copy of the appropriate template, not the built-in RM template itself. That way, the template you are actually using can be edited. Then edit the template you are using to add a text field called Page. The text field called Page will not appear in any of your Footnote, ShortFootnote, or Bibliography sentences. In that sense, it may seem like there's no reason for the Page field even to be there. However, the presence of the Page field will assure that the data is not completely lost when you merge the sources. You will have to move the data manually from the now unused Page field to where it really needs to go, but at least the data you need will be there rather than having been lost.

 

Even more strictly speaking, the free form template really has four fields; Footnote, ShortFootnote, Bibliography, and Page. The secret sentence template that you never see for creating footnotes is [Footnote]<, [Page]>  The secret sentence template that you never see for creating short footnotes is [ShortFootnote]<, Page>  The secret sentence template for creating bibliography sentences is [Bibliography]  Therefore, you may find it advantageous to add text fields called Footnote, ShortFootnote, Bibliography, and Page to the copy of the source template you are using.

 

Jerry



#5 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 09 December 2014 - 08:14 AM

Yes, agree.  However, I found that merging works OK for things that only have one field in the source detail (like books with page numbers), but not so well for more complicated citations like census sources.  If the source detail has several fields, i.e. page, ED, subject, etc., all of the source detail included with the free-form source citation is lost during the merge.  (In the free-form citiations, all the detail is combined in a single 'page number' field.)  

 

Unfortunately, I can't see any good way of overcoming this, since there's no way that RM could tell how to parse the combined data into the various fields of the new source detail (at least as long as we continue to have to deal with GEDCOM).  However, the merge dialog should probably warn you if it expects data will be lost during the merge.

 

It does not carry over because all the source templates do not have a PAGE field. Data only will carry over on a field if it exists on what you are merging it into. 



#6 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 10:31 AM

OK, thanks for the info.  Jerry, I'll give it a try.  That said, I would still recommend that the developer look at the underlying issue I described above.  I suspect that many users would not pursue it even to the level that I did.  Migrating data between different systems is a major pain, and having these sort of issues just makes it more annoying.

 

A few more recommendations based on the prevuius comments:

 

1) If the user community considers it 'best practice' not to use the original templates, and the developers agree with this approach, then I recommend that RM be changed so that the supplied templates are presented as an uneditable reference library separate from the actual template set used to create sources.   When the user creates a new source type, they could either copy one from the reference library, or create a new one from scratch.  This preserves the reference templates, while allowing the user to keep a concise list of working templates limited to the ones they actually use.  

 

2) Each original source template already references the style guide(s) that they model (i.e. EE, AQS, etc.), so take it one step further and tag each template with the source style guide(s) and allow the user to filter the reference list to only the template styles they prefer.

 

3) Allow the user to activate/deactivate their templates (could also be a first step towards implementing item #1).

 

4) Data migration is a huge pain (this is my fourth go at it since the late-80's ... I've done FTM -> TMG -> Genbox -> RM ... I guess once a decade or so is about all I can handle :-) ).  It would be really helpful to have a knowledge base set up for new migrators that provides guidance for handling situations like above that are not obvious to the new user. 



#7 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8492 posts

Posted 09 December 2014 - 10:49 AM

Confirming enhancement requests are in our tracking system. 

 

Sometimes when deleting the old source, I have been erroneously getting a dialog saying that there are still people, families or facts using the source.  I run a source report to see what citations are still attached, and in each case, the report shows a citation attached to the person name, but the citiation was deleted previously and does not show in the list of sources for the person's name (or any other events attached to that person).

 

When you run into this situation again could you please make a backup of your database and send it to us. Let us know which sources you are seeing this issue with.  Then development can figure out why its behaving in such a manner when it shouldn't.

 

You can open the support ticket here - http://support.roots...us_requests/new


Renee
RootsMagic

#8 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3609 posts

Posted 09 December 2014 - 12:40 PM

 

It does not carry over because all the source templates do not have a PAGE field. Data only will carry over on a field if it exists on what you are merging it into. 

 

That's why I suggested that you add a Page field to the template into which you are merging. If you add the Page field to the template into which you will be merging, then the data will carry over.

 

You won't be using the Page field that you added for anything. You will just be making sure the data is preserved so you can still see it. For example, you can copy and paste the data from the otherwise unused Page field to where it really needs to be.

 

Jerry



#9 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 09 December 2014 - 01:05 PM

 

That's why I suggested that you add a Page field to the template into which you are merging. If you add the Page field to the template into which you will be merging, then the data will carry over.

 

You won't be using the Page field that you added for anything. You will just be making sure the data is preserved so you can still see it. For example, you can copy and paste the data from the otherwise unused Page field to where it really needs to be.

 

Jerry

 

 

This is a good tip. This will greatly improve the speed of source cleanup when I have to preform them. I just wish I would have realized this sooner. I vary well might start a new database to where I do just this. 



#10 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 01:05 PM

The Page field addition works great.  The only problem is that it is difficult to tell if I have fixed the citation, i.e. moved the different elements in the Page field into the correct locations in the source, since the source line looks similar in the citation manager, same info, just in a different order.  I was considering adding a citiation element called 'Fixed' and entering 'Yes' when I've repaired it.  Then eventually delete the field from the template once all the citations are updated. Any better ideas?    



#11 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 09 December 2014 - 01:17 PM

The Page field addition works great.  The only problem is that it is difficult to tell if I have fixed the citation, i.e. moved the different elements in the Page field into the correct locations in the source, since the source line looks similar in the citation manager, same info, just in a different order.  I was considering adding a citiation element called 'Fixed' and entering 'Yes' when I've repaired it.  Then eventually delete the field from the template once all the citations are updated. Any better ideas?    

 

That depends on if you ever plan to use the PAGE field for anything else. If not you can run a SQL query to check if the PAGE field is NOT NULL. Then you would know that you still need to update those citations. 



#12 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3609 posts

Posted 09 December 2014 - 01:28 PM

 

If not you can run a SQL query to check if the PAGE field is NOT NULL. Then you would know that you still need to update those citations. 

 

You can do this, but the necessary SQL is not trivial. The reason is that there really isn't a Page field as a database column to test with SQL. Rather, all the source data is stored as a bunch of XML tags in one big text field that's the actual database column. That is, items you have added as "fields" to source templates can't be database columns because RM has no idea what "fields" you might add to templates. So you have to write SQL that can parse the XML - not rocket science, but neither is it trivial.

 

Jerry



#13 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 09 December 2014 - 01:36 PM

 

You can do this, but the necessary SQL is not trivial. The reason is that there really isn't a Page field as a database column to test with SQL. Rather, all the source data is stored as a bunch of XML tags in one big text field that's the actual database column. That is, items you have added as "fields" to source templates can't be database columns because RM has no idea what "fields" you might add to templates. So you have to write SQL that can parse the XML - not rocket science, but neither is it trivial.

 

Jerry

 

Good point. That would be a pain to write. You could also put something in the citation comments field stating {Migrated date}. You could then run for a search query to where that text is missing. 



#14 Laura

Laura

    Advanced Member

  • Members
  • PipPipPip
  • 4276 posts

Posted 09 December 2014 - 01:47 PM

In RM 7, create a Group for the Mastsr source,:

 

Any fact, source, name contains, [Master source name].

 

Since I don't use the Detail text, reference box for Source details, I have started entering a period in the reference box when I correct the source detail.

 

Then I can use the . as a filter to mark or unmark a Group.

 

If I was using that reference box, I could still use the period or some other character.

 

The Source details, Details text, reference box is not printed for reports so it isn't that important what character is used.

 

With the new QuickGroup feature, I can remove the person from the Group when I finish the corrections for that person.

 

Since I am working on census sources whern I have the same source citations with the same source details and Research notes and Source detail notes linked to multiple people, I am still entering the period.

 

Even if f I was correcting a source where the Source detail was just for one person, I would still add the period to the Detail text, reference box for it's value as a search filter rather than just  use QuickGrouo to remove a person from the Group.



#15 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 02:14 PM

Thanks Laura.  I have never used the Source to create a group.  There's a gold mine of good stuff buried in the long list of operators you can use for filters.  Wish I'd seen this a few months ago :-). 

 

I'd love it if they could either 1) separate the system items from the fact types in the 'Select Field' list, or 2) allow us to turn on/off adding fact types to the list.  With them co-mingled alphabetically, the system items get lost in the shuffle, and those are the ones that I tend to use the most.



#16 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6268 posts

Posted 09 December 2014 - 02:14 PM

 

Good point. That would be a pain to write. You could also put something in the citation comments field stating {Migrated date}. You could then run for a search query to where that text is missing. 

The basic test to find rows in the CitationTable that have non-empty Page fields (<Name>Page</Name><Value>something</Value>) would be

 
SELECT CitationID 
FROM CitationTable
WHERE Fields LIKE '%<Name>Page</Name><Value>_%</Value>%'
;
 

The _ wildcard requires a single character; the % wildcard any number of characters from 0 up.

 

The added compexity is to tie that row to its Master Source and to the fact that it supports. But that's helped by: Search - wayfinding from data tables to RootsMagic screens. One could plug the row number from the above query into the filter on the temp.CitationWay view to be guided through RootsMagic to the citation in question. 


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#17 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 03:49 PM

Thanks to all for the diffierent suggestions.  My SQL skills are a bit rusty by a few years, but may be worth a look as well.



#18 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3609 posts

Posted 09 December 2014 - 03:55 PM

I have never used the Source to create a group.  

 

I use Sources to create Groups quite a bit. There are lots of different source fields available for testing. I do have to say that I have found the process to be much more effective now that I have switched to using a different fact type for each census year. Making this change will probably cause me some unforeseen grief on down the road. But in the meantime it means that the Boolean logic I use to test against census data when using RM to create groups works really well. Using only the built-in census fact for each census year essentially meant that I had lots of duplicate census facts for each person, and there is no way to direct RM's boolean logic in such away that each test goes against the same fact when there are duplicate facts of the same type.

 

Jerry



#19 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 09 December 2014 - 06:24 PM

Tom,

 

I'm thinking my best approach going fwd is a combination of your's and Laura's suggestions.  FYI, I have about 700 sources and about 20K citations to wade thru.  I like Laura's approach to manage future updates because I'm updating them to better conform to EE, so I have to manually touch many of them anyway.  The problem is that I've been at it for several months, so I've already fixed a few thousand and don't want to start over.  For her approach to be effective, I'd need to go find all the citations I've already fixed and update the reference number of each so that the Group filter works correctly.  So I'm thinking I have two possible choices:

 

1) Write a query to update all the citation reference numbers to add a short text string to any citations that are linked to a free-form source type.

 

or

 

2) Write a query to update all the citations, as above, that have "<Name>Page</Name>" in the Fields BLOB.  For this to work, I'm presuming that RM has added that code to every free-form citation, regardless if there is any page text.  Based on what I have already read from this thread, I think that assumption is correct.

 

My preference is option 2, since I only need to be concerned with one table.  I'm using SQLite Expert, have loaded the unifuzz DLL, and have my DB backed up.  Do you concur, and are there any gotchas in working with this DB that I need to be concerned with?



#20 quarter

quarter

    Member

  • Members
  • PipPip
  • 13 posts

Posted 10 December 2014 - 01:27 AM

UPDATE:

 

I realized that option 2 above will not work because there are indeed pages in sources like books.  No kidding.  

 

So, resorting to option 1, I ran the following query to set all the free-form citations to 'Fix'.  This will allow me to set up a Group of people that have citations that need to be fixed.  I'll delete the 'Fix' as I update them.  That's the opposite of Laura's approach, but I prefer it as it leaves the citation reference no. empty when I'm done.   

 

UPDATE CitationTable
SET RefNumber = "Fix"
WHERE SourceID IN (SELECT SourceID FROM SourceTable WHERE TemplateID = 0);