Proposal: Change some optional substructures to be required

[mother10]

Allow better tracking of the origin of a GEDCOM file, and changes made to it.

To prevent certain situations like this to be an official header:

0 HEAD
1 GEDC
2 VERS 7.0

Which totally does not say anything useful about where this file comes from, or what software created it.

and also to make changes more traceable:

This proposal adds the following to the specification:

n HEAD                                     {1:1}  g7:HEAD
  +1 GEDC                                  {1:1}  g7:GEDC
     +2 VERS <Special>                     {1:1}  g7:GEDC-VERS
  +1 SCHMA                                 {0:1}  g7:SCHMA
     +2 TAG <Special>                      {0:M}  g7:TAG
  +1 SOUR <Special>                        {1:1}  g7:HEAD-SOUR
     +2 VERS <Special>                     {1:1}  g7:VERS
     +2 NAME <Text>                        {1:1}  g7:NAME
     +2 CORP <Text>                        {1:1}  g7:CORP
        +3 <<ADDRESS_STRUCTURE>>           {0:1}
        +3 PHON <Special>                  {0:M}  g7:PHON
        +3 EMAIL <Special>                 {1:M}  g7:EMAIL
        +3 FAX <Special>                   {0:M}  g7:FAX
        +3 WWW <Special>                   {1:M}  g7:WWW
     +2 DATA <Text>                        {0:1}  g7:HEAD-SOUR-DATA
        +3 DATE <DateExact>                {0:1}  g7:DATE-exact
           +4 TIME <Time>                  {0:1}  g7:TIME
        +3 COPR <Text>                     {0:1}  g7:COPR
  +1 DEST <Special>                        {0:1}  g7:DEST
  +1 DATE <DateExact>                      {1:1}  g7:HEAD-DATE
     +2 TIME <Time>                        {1:1}  g7:TIME
  +1 SUBM @<XREF:SUBM>@                    {1:1}  g7:SUBM
  +1 COPR <Text>                           {0:1}  g7:COPR
  +1 LANG <Language>                       {0:1}  g7:HEAD-LANG
  +1 PLAC                                  {0:1}  g7:HEAD-PLAC
     +2 FORM <List:Text>                   {1:1}  g7:HEAD-PLAC-FORM
  +1 <<NOTE_STRUCTURE>>                    {0:1}
  +1 <<CHANGE_DATE>>                       {1:1}
  +1 <<CREATION_DATE>>                     {1:1}

Some cardinality markers are changed:

• SOUR, the program used to create this GEDCOM, should at a minimum give certain information, so users and other software can track the origin of a specific GEDCOM.
• VERS, NAME, CORP, EMAIL and WWW, are the minimal info about the software this GEDCOM was created by, that should be present in a header.
• DATE and TIME, The datetime this GEDCOM file was created, is now also mandatory.
• SUBM, Submitter is now also mandatory.
• CREA and CHAN, added these too, as the header was the only record that did not have these. Because a conversion from 1 GEDCOM version to another, also changes info in the header. (replacing the name of the software for instance)

PHON and FAX are not included, because of current privacy regulation.

Further:

• All records that have a <<CHANGE_DATE>> and a <<CREATION_DATE>>, should now have the cardinality markers for those set to {1:1} also.

[tychonievich]

Before weighing in on if these requirements should be added, I wanted to review why we require structures now. I see the following five reasons:

1. Metasyntactic convention. The top-level structure(s) of any rule that creates a single structure (most rules them except the four rules with names ending _DETAIL)
2. Syntactic requirement. Structures used to organize or identify datasets:
   • HEAD, TRLR
   • HEAD.GEDC and HEAD.GEDC.VERS
3. Semantic requirement. Structures whose definition directly references a substructure:
   • Generic structures defined by their required substructure:
     • FACT.TYPE
     • EVEN.TYPE
     • IDNO.TYPE
     • ASSO.ROLE
     • TRAN.LANG
     • TRAN.FORM
     • EXID.TYPE (not required by mistake, but omitting is deprecated)
   • A 2D vector in 2 structures: MAP.LATI and MAP.LONG
   • Superstructure+substructure pair giving a single meaning:
     • HUSB.AGE
     • WIFE.AGE
     • HEAD.PLAC.FORM
     • CHAN.DATE
     • CREA.DATE
   • Substructure is the primary (but not only) purpose of the superstructure: OBJE.FILE
4. Operation enabling. Substructures required for common operations:
   • REPO.NAME (for creating some citations)
   • SUBM.NAME (for displaying attributions)
   • FILE.FORM (for knowing what tool to use to display the file)
5. Good research. Structures where "good" research would always produce a substructure too:
   • ord-STAT.DATE (data provided by a single organization that provides dates with this information)

I see all of these proposals, and those in #689, #687, and several other recent issues, as being of types 4 or 5.

[tychonievich]

When drafting 7.0, it became clear that 5.5.1's "good research" requirements of HEAD.SOUR and HEAD.SUBM (every file came from some user and some tool, so it's good to share what that is) led to many files either braking the rules or including meaningless placeholders, like HEAD.SOUR NONE or SUBM.NAME anonymous; or worse, misleading values gained by copying them from other files. That motivated us to remove those requirements, and I oppose adding them back in. HEAD.SOUR should remain optional, as should its substructures. I get why it would be nice if every file had accurate values there, but I know from experience that making them required substructures will not achieve that end.

Requiring change and creation dates could be an operation-enabling requirement, where the operation is showing "Recently changed" lists or the like. However, I also don't trust that making them required would cause them to have always have accurate data, and I'd rather not have those fields at all than have them present but incorrect, so I also oppose making them required, though my opposition of "operation enabling" changes is less strong than for the "good research" proposals.

As a somewhat-related aside: I'd support making REPO.NAME and SUBM.NAME optional in a future version.

[mother10]

Maybe it would help if there were a neutral reference page that tracks how well different programs align with specific GEDCOM versions.

For example, such a page might list something like:

• GEDCOM 7.0 support: 80%
• TAG usage according to spec: 70%

The idea wouldn’t be to “grade” software in an exam sense, but to give users and developers a clear picture of where things are strong and where improvements could be made. That way, if a TAG is not used for its intended purpose, it’s visible in the percentage, and people can make informed choices.

To me, it sometimes feels like saying: “so many drivers ignore this traffic rule, let’s just remove the traffic light.” I think clearer guidance, and visibility on adherence, might be a more effective path.

[dthaler]

Maybe it would help if there were a neutral reference page that tracks how well different programs align with specific GEDCOM versions.

For example, such a page might list something like:

• GEDCOM 7.0 support: 80%
• TAG usage according to spec: 70%

The idea wouldn’t be to “grade” software in an exam sense, but to give users and developers a clear picture of where things are strong and where improvements could be made. That way, if a TAG is not used for its intended purpose, it’s visible in the percentage, and people can make informed choices.

To me, it sometimes feels like saying: “so many drivers ignore this traffic rule, let’s just remove the traffic light.” I think clearer guidance, and visibility on adherence, might be a more effective path.

https://magikeygedcomconverter.azurewebsites.net/Compatibility does provide some measure of alignment. It detects things that are not compliant, but also measures what optional parts are used vs not supported (i.e., discarded on import, and not present on export). Feedback from others was they did NOT want a specific % number listed. A higher % is not necessarily better, if it supports less "important" pieces without supporting more important pieces. And of course different applications with different focuses might consider different things important. So a strict number is not as useful as say feature-specific measures. FamilySearch/GEDCOM.io#48 is relevant to the discussion of tracking feature-specific usage information. But if a user wants to know what programs support, say, "memories" (multimedia objects, etc.), https://magikeygedcomconverter.azurewebsites.net/Compatibility does display that.

[mother10]

I tried the above with Magikey, and used the maximal70.ged as instructed.

Imported in my main software (the one I use now), that could read it and process it, although it reported some minor changes..
I could see the tree there and everything.
So the software showed me the tree inside the file, and as far as i can see, all info from the original file.

But the file it created, when loaded in Magikey, gave me "Original GEDCOM content could not be parsed".
No idea why.

I tried the same with the other software I once used, hmmm, that could not even read the file properly. The original filelength of 14K was reduced to 4K when i asked for a GEDCOM 7 export, so that didnot work well.
But I still loaded the created file in Magikey, and that gave me the same error.

So I think I must be doing something wrong.