Jump to content


Photo

Error in 'My Heritage' lightbulb


  • Please log in to reply
48 replies to this topic

#21 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 15 December 2014 - 05:06 PM

MyHeritage displays the person info (except for the UID) it receives from RootsMagic via the API when you click on one of those Hint links (Pending, Confirmed, ...). That info is displayed under the heading "In my RootsMagic Family Tree" or words to that effect in the left panel. The right panel shows the Record Matches or Tree Matches from their database. The RM person info is discarded by MyHeritage (it is only needed to search for matches and return that page to you) at some point but the UID it received is stored with each Confirmed Match (and I would have thought with each Rejected Match but there seems to be a bug there).

 

BTW, the UID concept was present in PAF. A UID created in PAF is preserved when imported to RootsMagic, even via GEDCOM. It would be interesting to know how widely it is supported.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#22 ennoborg

ennoborg

    Advanced Member

  • Members
  • PipPipPip
  • 53 posts

Posted 16 December 2014 - 08:35 AM

http://www.tamurajon...he_UIDTag.xhtml



#23 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3487 posts

Posted 16 December 2014 - 09:42 AM

Here's another little tidbit. I confirmed a hint for a record match in MyHeritage. It was a very simple match - an SSDI entry where the name, birth date, and death date all matched properly between my database and the SSDI record at MyHeritage. Then I wondered if I could unconfirm the match. I couldn't find an explicit "unconfirm", so as an experiment I rejected the match at MyHeritage and the WebHint went away completely in my RM database. Subsequently, I couldn't find or think of any way to re-confirm or un-reject the match at MyHeritage. The original match simply became invisible to me after it was rejected.

 

So to get the WebHint back in my RM database, I changed the UID in my database for the individual. I did so by creating a dummy person and merging my real person with the dummy person. The result was a person with a new UID and all the data from my original person.

 

The WebHint then reappeared for the person. It seemed to take a very long time for it to do so - like an hour or two. I'm not sure what might have had to have been reset at the MyHeritage site to get the WebHint back, but it finally came back. In the meantime, there is an orphaned rejected hint at the MyHeritage web site that is associated with a UID that doesn't exist anywhere in the world anymore except at the MyHeritage Web site. It seems to me like MyHeritage should deal with this situation of unrejecting a rejection a little more gracefully than it does, but I'm not sure exactly how I would propose to fix it.

 

Finally, it occurs to me that if I were to start confirming a lot of WebHints at MyHeritage, there would be no way to know which WebHints I had confirmed unless I kept my own meticulous records of what I had done . You can tell that there are confirmed hints if you see a clear light bulb, but there is no way to search for or run a report against all the clear light bulbs in your database. You can't even do it with SQLite because the data is not in your database. The same issue sort of exists also at familysearch, but at least at familysearch you can see the WebHint matches (called Record Hint matches) in the community tree at familysearch.

 

Jerry



#24 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 16 December 2014 - 11:00 AM

The URL sent by RootsMagic from WebHints can be modified to see the Rejected matches.

 

Total => http://www.myheritag...h_confidence=10

Pending => http://www.myheritag..._status=pending

Confirmed => http://www.myheritag...tatus=confirmed

and, therefore, logically:

Rejected => http://www.myheritag...status=rejected

 

You can click the checkmark to un-reject a hint and click on the checkmark to confirm/unconfirm a hint.

 

WebHints is evidently not getting or displaying the Rejected Hints count nor generating the hyperlink with this URL as reported in WebHints not tracking Rejected Hints from MyHeritage

 

If you see info under "In my RootsMagic tree" when you click on these links, then it is evident that MyHeritage does not immediately purge that info that came from my database. Then the question is, "how long does it persist?".


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#25 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 16 December 2014 - 09:45 PM

If you see info under "In my RootsMagic tree" when you click on these links, then it is evident that MyHeritage does not immediately purge that info that came from my database. Then the question is, "how long does it persist?".

Now I'm getting suspicious. It has been over ten hours since I accessed MyHeritage via WebHints. I see the "In my RootsMagic tree" info about Martha Alexander when I click on these links from my iPad browser! My iPad cannot send any info about her to MyHeritage because it cannot run the real RootsMagic. So its browser is getting back from somewhere the info that we are assured MyHeritage does not keep. Could it be from a network router cache or is MyHeritage hanging onto it longer than we assume it should.

 

If you click on the links above, do you see any "In my RootsMagic tree" info about Martha Alexander? If you do, then surely it is MyHeritage that is keeping it.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#26 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3487 posts

Posted 16 December 2014 - 09:51 PM

In short, yes.

 

The information from your RootsMagic tree is there with all four of the links.

 

Jerry



#27 ennoborg

ennoborg

    Advanced Member

  • Members
  • PipPipPip
  • 53 posts

Posted 17 December 2014 - 09:08 AM

Same here. I see full details of Martha Alexander and the names of close relatives on the left side of my screen, and 6 matches, when I select the Total link. This makes it look like My Heritage keeps these details as long as it suits them. And details include the person's vitals, and at least the names of close relatives.

 

And that makes me wonder about another thing, that may have been mentioned before. What if you change the given name in RM? Does that show on My Heritage? Will you get new matches then? You can do the same for close relatives, even change the surname of her mother, and in most cases I assume that the UID inside RM stays the same. What if you remove some relatives?



#28 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 17 December 2014 - 10:22 AM

This is getting to be disconcerting; it is becoming increasingly certain that MyHeritage is not living up to its commitments and RootsMagic cannot therefore claim that information from a user's database is not retained online. If I have WebHints enabled in my database, as soon as I navigate one of the main views that supports WebHints lightbulbs to any person(s), including the living, RootsMagic issues a query through the API to MyHeritage conveying those persons' vital information and immediate relatives (living or dead) so that MyHeritage can conduct a search for possible new matches and look up previously pending, confirmed (and rejected) matches. It is now becoming evident that this personal info is not discarded on completion of the search and remains online for an indefinite period, now known to be in excess of 22 hours. Moreover, knowledge of the UUIDs enables anyone to construct URLs to retrieve that information. While obtaining a UUID may be difficult, we also know that systems of companies with even higher security concerns than MyHeritage have been breached.

 

How does this square with the statement published in their news releases?

 

Information sent by RootsMagic to MyHeritage for matching is never collected or shared, and is deleted after matching to ensure the complete privacy of RootsMagic users and their data.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#29 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 17 December 2014 - 10:54 AM

While RootsMagic does send what looks like a UID to MyHeritage, it is not the UniqueID found in its PersonTable. Rather, it is a shorter UID generated on the fly, or, at least, not stored in the RM database. Example:

0D1255A5EB824125BFC60A81B98ED7102AC5 - RootsMagic UniqueID
da48a789e71fbb2dfcb3e5ae592364df - WebHint generated UID-like number in URL, as in

http://www.myheritage.com/matchingresult-da48a789e71fbb2dfcb3e5ae592364df?match_type=all&minimal_match_confidence=10

The WebHints UID appears similar to a UUID lacking a checksum at the end.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#30 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 17 December 2014 - 02:49 PM

I jumped to an erroneous conclusion about the nature of the long hexadecimal string in the URL. It is not a UUID, or not exactly. If I copy a person to another database, that person keeps his RM UniqueID which is a UUID. The WebHints generated for that same person in both databases are identical but the URLs for the hyperlinks to the MyHeritage pages are different. The pages returned do differ in the information presented in the "In my RootsMagic tree" panes, as one would expect - in one case, the person has family while the other no family. They are identical on the matches; confirming on one page has the ultimate effect of updating the counts in both databases. So these persons with identical RM UIDs do get identified in MyHeritage as the same person but RM generates different URLs. Changing the person's name results in a further change in the URL, a reduction in the number of unconfirmed matches but the confirmed matches remains the same. And confirming a match that is common to both the name-changed matches and the name-unchanged matches is reflected in both databases.

 

So, methinks that RM does pass its UniqueID to MyHeritage and gets back another hex number to be used in the URLs. That MyHeritageID may be some kind of a hash of the name and other stuff (maybe the RM UID) that points to the data on which it builds each page. And I would guess it has a table linking the RM UniqueID to all of these MHIDs. Still looks like MH is hanging onto the person's vitals and family...


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#31 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 19 December 2014 - 10:15 AM

This is getting to be disconcerting; it is becoming increasingly certain that MyHeritage is not living up to its commitments and RootsMagic cannot therefore claim that information from a user's database is not retained online. If I have WebHints enabled in my database, as soon as I navigate one of the main views that supports WebHints lightbulbs to any person(s), including the living, RootsMagic issues a query through the API to MyHeritage conveying those persons' vital information and immediate relatives (living or dead) so that MyHeritage can conduct a search for possible new matches and look up previously pending, confirmed (and rejected) matches. It is now becoming evident that this personal info is not discarded on completion of the search and remains online for an indefinite period, now known to be in excess of 22 hours. Moreover, knowledge of the UUIDs enables anyone to construct URLs to retrieve that information. While obtaining a UUID may be difficult, we also know that systems of companies with even higher security concerns than MyHeritage have been breached.
 
How does this square with the statement published in their news releases?

  

The URL sent by RootsMagic from WebHints can be modified to see the Rejected matches.
 
Total => http://www.myheritag...h_confidence=10
Pending => http://www.myheritag..._status=pending
Confirmed => http://www.myheritag...tatus=confirmed
and, therefore, logically:
Rejected => http://www.myheritag...status=rejected
 
You can click the checkmark to un-reject a hint and click on the checkmark to confirm/unconfirm a hint.
 
WebHints is evidently not getting or displaying the Rejected Hints count nor generating the hyperlink with this URL as reported in WebHints not tracking Rejected Hints from MyHeritage
 
If you see info under "In my RootsMagic tree" when you click on these links, then it is evident that MyHeritage does not immediately purge that info that came from my database. Then the question is, "how long does it persist?".

Three days since the above links were generated through WebHints and they still return the info sent from my RootsMagic database. So much for "never collected" and "deleted after matching"; note they don't say how long after...

What this means is that if you are the root person in your database and you have your program set to open on the root person and because WebHints is enabled by default, the very first time you open your database, RootsMagic sends your personal info to MyHeritage for its search for matches and that info stays on a MyHeritage server for days, at least, and permanently, at worst. I see this as a clever stratagem to amass data about people and mine it for leads on family connections (taking the least jaundiced view).

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#32 Renee Zamora

Renee Zamora

    Advanced Member

  • Support
  • PipPipPip
  • 8375 posts

Posted 19 December 2014 - 11:06 AM

This is what the RootsMagician has told me.

 

My heritage is not keeping their data. They create a hash of the data. If we send a request with the exact same data it would generate the same hash so they would know it had already been requested.

 

MyHeritage cache's hints for +30 days possibly up to 4 mo's. Won't see new webhints during that time frame unless you change data on the person.


Renee
RootsMagic

#33 anzenketh

anzenketh

    Advanced Member

  • Members
  • PipPipPip
  • 197 posts

Posted 19 December 2014 - 11:35 AM

This is what the RootsMagician has told me.

Quote

My heritage is not keeping their data. They create a hash of the data. If we send a request with the exact same data it would generate the same hash so they would know it had already been requested.

 

MyHeritage cache's hints for +30 days possibly up to 4 mo's. Won't see new webhints during that time frame unless you change data on the person.

 

 

 

Do you know what type of hash they are using. Some hashes are one-way (MD5, SHA1, SHA2) while others (not common) are invertible. 



#34 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3487 posts

Posted 19 December 2014 - 12:00 PM

MyHeritage cache's hints for +30 days possibly up to 4 mo's. Won't see new webhints during that time frame unless you change data on the person.

 

 

That raises as many questions as it answers. Does that mean that after a WebHint has been cached for "+30 days possibly up to 4 mo's" that a confirmed WebHint will no longer be confirmed?

 

Jerry



#35 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 19 December 2014 - 12:46 PM

Another question (given my lack of knowledge of hashing), is it possible that "935595c0576156b3ca096b209e58fd4d" can possibly represent this:
In my RootsMagic tree
Martha Alexander
Birth: May 20 1784
Death: Oct 27 1863
Parents: John Ferguson and Jane, -(w/John Ferguson)
Partner: John Alexander
Children: Jane, Mary, Elizabeth, Susanna, John, Margaret, Julian, Josiah, Samuel and James
and potentially much more given that these are short names?

I rather doubt it. I think that 32 character hexadecimal-like string is a pointer to the data on the MyHeritage server and maybe a decryption key and it is carried in an URL that is plain, open http, not even https.

Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#36 kbens0n

kbens0n

    Advanced Member

  • Members
  • PipPipPip
  • 3444 posts

Posted 19 December 2014 - 03:14 PM

The Family Graph API is openly available for your perusal:

http://www.familygra...m/documentation

---
--- "GENEALOGY, n. An account of one's descent from an ancestor who did not particularly care to trace his own." - Ambrose Bierce
--- "The trouble ain't what people don't know, it's what they know that ain't so." - Josh Billings
---Ô¿Ô---
K e V i N


#37 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 19 December 2014 - 04:12 PM

The Family Graph API is openly available for your perusal:

http://www.familygra...m/documentation

I assumed that that was for something else but the MatchingRequest definition seems to fit the style of the links from WebHints:

 

Get matches count for a given individual details.

Individual details are sent as a raw POST body. The MD5 hash result of this body is concatenated to the link as shown in the example below.

That would indicate that the RM UID is not involved in the 32 character hex string, only the name, Birth/Death data, family members' names and relationships. The person's data has to be stored for it to be returned with the matches. And it would seem that two users on separate databases and computers having persons in common in their databases with the exact same key info but different RM UIDs should end up with the exact same matches and one's confirmations will be seen by the other. We should test that. 


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#38 TomH

TomH

    Advanced Member

  • Members
  • PipPipPip
  • 6179 posts

Posted 20 December 2014 - 04:43 PM

I assumed that that was for something else but the MatchingRequest definition seems to fit the style of the links from WebHints:

That would indicate that the RM UID is not involved in the 32 character hex string, only the name, Birth/Death data, family members' names and relationships. The person's data has to be stored for it to be returned with the matches. And it would seem that two users on separate databases and computers having persons in common in their databases with the exact same key info but different RM UIDs should end up with the exact same matches and one's confirmations will be seen by the other. We should test that. 

Testing it myself, I am left puzzled. Despite the API documentation, I remain convinced that the 36 character hex string RM UniqueID does figure into the WebHints query results. In my test, I copied a person with her parental family and child family to a database. The WebHints were identical 3-pending, 3-confirmed, 6-total and I know there are 2-rejected. If I changed or deleted her RM UID using SQLite, WebHints came back with 8-pending and total. If I kept the RM UID the same and added a Given name to her and/or her father, the ID in the WebHints URL changed but the results came back as at the beginning 3-pending, 3-confirmed, 6-total. So it seems that the RM UID is a key identifier and that there can be some variation in the person's and their families' data before the WebHints process results in MyHeritage deciding that it is not the same person.


Tom user of RM7550 FTM2017 Ancestry.ca FamilySearch.org FindMyPast.com
SQLite_Tools_For_Roots_Magic_in_PR_Celti wiki, exploiting the database in special ways >>> RMtrix-tiny.png app, a bundle of RootsMagic utilities.


#39 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3487 posts

Posted 20 December 2014 - 07:43 PM

I  have not repeated any of my testing, but my earlier testing certainly indicated that the RM UniqueID has something to do with the way the matching works.I agree that the documentation of the MatchingRequest object suggests otherwise. It's all very puzzling.

 

We are in need to a Webinar about how this works.  The Webinar really needs to address in a very candid way how the matching works, what is sent by RM to MyHeritage, what is stored on the MyHeritage server, for how long, who can see it other than the user whose RM submitted the data, and what else the MyHeritage web site might be doing to harvest the data.

 

I used to be much more of a techie than I am now. If I still had the appropriate skill set and the appropriate tools, I would want to see a trace of the network traffic between RM and the MyHeritage web site as the place to start. That wouldn't tell me what is stored nor for how long, but at least I could see what is being sent. It appears to be https traffic, so I would need to see the traffic before encryption took place.

 

In any case, it does seem very disconcerting that absolutely nothing at all about the matches is stored in the RM database. Therefore, whether matches have been confirmed or not (or rejected or not) has to be stored on the MyHeritage server. That means that every individual who has ever had a MyHeritage related light bulb as I use RM must have data of some kind stored on the MyHeritage server, even though I don't have a tree at MyHeritage. Again, it's very disconcerting.

 

The situation with FamilySearch is very different. It's still a bit of a mystery to me how FamilySearch comes up with its Record Hints in the first place. But having come up with them, it's very transparent how they work because all the Record Hints are associated with the FamilySearch community tree. Anything about the FamilySearch Record Hints that you can see as WebHints in RM is also clearly visible on the FamilySearch web site even if you don't use RM or any other local genealogy software.

 

Jerry



#40 Jerry Bryan

Jerry Bryan

    Advanced Member

  • Members
  • PipPipPip
  • 3487 posts

Posted 20 December 2014 - 09:47 PM

I did some additional testing. My results are very different than before. Perhaps I have fooled myself into seeing what I was expecting to see.  :)

 

My test subject was John H. Peters, born 8 Oct 1843, died 8 Mar 1890. In my production database, there is a light bulb with two WebHints for MyHeritage, a record match for the 1850 census and a tree match.

 

  • I created a new, empty database and hand entered all the data that follows. Nothing was copied over from my production database.
  • I entered John H. Peters. No light bulb.
  • I added a birth date of 1843. No light bulb.
  • I added a birth date of 1843 and a death date of 1890. No light bulb.
  • I deleted the death date and changed the birth date to 8 Oct 1843. Light bulb for the tree. That's not much data, but there was a WebHint anyway.
  • I changed the birth date back to 1890. The light bulb went away.
  • I added a father of John W. Peters with no other data. No light bulb.
  • I added a mother of Ruby (her full name was Ruby M. Smith, but the census doesn't care about that), and the light bulb appeared for the 1850 census. It didn't need a birth date for either John W. Peters or Ruby, But it did need both of them. John W. alone as father or Ruby alone as mother wasn't enough. It needed both of them. And they worked even though the 1850 census doesn't explicitly identify family relationships.

If I copy the MyHeritage URL for the WebHint matches (including the weird hash string) and paste it into a different browser, I do see the limited amount of data that I have entered for John H. Peters into my test database. But unless I post my URL, you all shouldn't be able to see the data from my test RM database. If you all repeat my experiment with the data I've included in this message, you should be able to see the data from your RM database. The data will be the same, simply because I've shared it with you in this message. But you shouldn't be able to see the rest of my data, such as the birth place for John H. Peters or the death place or the burial date and place, etc. And you could even post your URL with its weird hash string so the rest of us could see your data that you just entered for John H. Peters. But it's hard to see how you could see my data unless I were willing to share my URL with my hash string. So the process seems pretty safe to me at this point (well, unless you get the exact same hash string with my data as I get with my data).

 

Nevertheless, it remains the fact that my data is being stored on the MyHeritage server even though I didn't upload it - hopefully in such a way that only I can see it. But I didn't explicitly create a tree or otherwise explicitly upload my data. It just got uploaded and kept automatically as a part of creating my light bulbs for me.

 

Furthermore, if I confirm a match, it can never delete my data after an appropriate timeout period or else the confirmation will go away.

 

Finally, nothing in this last round of testing seems to have anything to do with RM's Unique ID. Since I made a whole new database and didn't drag and drop anything into it, the John H. Peters in my test database had a different Unique ID than in the John H. Peters in my production database.

 

Jerry