Improve training data quality for references#678
Merged
Conversation
part of eLifePathways/ScienceBeam2.0#105 The XPath for the REFERENCE_AUTHOR sub-field targeted <string-name> elements outside <person-group>, which does not exist in standard JATS. Authors live inside <person-group person-group-type="author">, and the not(ancestor::person-group) guard would have excluded them regardless. No reference author tokens were being annotated in generated citation training data. Fixed the XPath and added a unit test.
<etal/> carries no text content, so it was silently dropped from the author needle. The generated training data now includes "et al." in the <author> span, matching the grobid training data convention.
…g data - Sub-fields sorted longest-first so shorter needles (e.g. label "1") cannot displace longer ones (e.g. year "1987") at the same position - Full masking applied to all sub-field types, not only REFERENCE_AUTHOR - Within-gap mid-token guard advances prev_included_end so the gap chain is not broken by overlapping SW noise blocks - Tail extension off-by-one fixed (inclusive gap=3 boundary) - Surname-only fallback extended to also find given-name initials within gap=3 of the matched surname, in both mid-token and regular fallback paths - Gap fill tolerates unlabeled tokens (entry=None) so "et al." and interstitial periods no longer reset the fill state - Per-name author sub-fields: only REFERENCE_AUTHOR ranges are masked across names; other sub-fields are single-occurrence and do not need it
SW character-level matching allowed "200" to match inside "2020" by consuming characters with a gap (2,0,_,0). For digit-only needles (years, page numbers, volumes, issues) the correct match is always an exact token — "200" cannot be a prefix or suffix of a longer number. _is_pure_number + _exact_number_match bypass SW for these needles, requiring the match to start at a token boundary and end at a token boundary. Two new _TokenIndex helper methods (is_in_token, is_token_boundary_after) expose the boundary checks without accessing the protected _token_index_at attribute directly.
When multiple windows produce a hit, the first match was returned regardless of whether it was exact (single contiguous block) or gapped (SW used character-level gaps to cover the needle). This caused "2020b" to be labeled at the first plain "2020" occurrence via a 0.8-quality gap match, while the true "2020b" token later in the text was ignored. _fuzzy_match_field_value now stores the first gap match as a fallback and continues scanning, returning immediately only when an exact contiguous match is found. The existing pure-number path (_exact_number_match) is unaffected; this change covers all other tokens.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/105