Improve JATS aligner quality for reference segmenter training data#681
Merged
Conversation
Two improvements to JATS-guided training data generation for the reference-segmenter and citation models: 1. Line-level gap fill in _split_references_by_jats_instance: unlabeled lines (e.g. DOI continuation lines that the aligner didn't match) inherit the preceding line's REFERENCE instance_id rather than being silently dropped. Restores DOI continuation text that was absent from citation training data. 2. Whole-line expansion in ReferenceSegmenterModelTrainingDataGenerator .get_jats_label_fn: if any token on a PDF line is labeled REFERENCE, all tokens on that line receive the same <reference>/<label> tag. Eliminates the fragmented <bibl> spans caused by unlabeled separator tokens (colons, semicolons, whitespace between sub-fields). Both changes rest on the invariant that no two distinct references share a single PDF line, which holds for all observed training documents.
Each reference in the generated TEI now lands in its own <bibl>, with its label (e.g. "9.") at the start rather than stranded at the end of the previous reference. This means the reference-segmenter model trains on correctly structured data, improving its ability to detect where one reference ends and the next begins — particularly for references whose number appears on the same line as their content. Also fixes block splitting for the citation model path: a layout block that straddles two JATS instances is now split at the boundary rather than assigned wholesale to one reference, and running headers/page numbers inside the references section no longer bleed into adjacent references via gap fill.
Bracket-style reference labels like `[1]`, `[2]` were not appearing in `<label>` elements in the generated TEI training data. The PDF tokeniser splits `[1]` into three tokens `[`, `1`, `]`, producing `[ 1 ]` in the alignment haystack. The Smith-Waterman library's traceback logic terminates early at gap moves, so it matched only the final `]` (quality 0.33, below threshold). A new `_try_bracket_label_match` fallback strips the brackets, locates the inner number via exact token match, and extends the range to include the surrounding bracket tokens. It also searches a small buffer before the nominal segment start, since the parent SW match often begins at `]` — leaving `[` and the number outside the sub-field search range. Separately, "DOI:" prefix lines (no annotated tokens) were causing the label function to return `None`, which backed the TEI writer up to `<listBibl>` level and produced a spurious `<bibl>` for the URL on the next line. Lines with no annotated reference tokens now continue the current bibl as `<reference>` when a reference is already open.
Two fixes reduce the scielo citation score regression introduced when
JATS-guided bibl splitting replaced ML-model splitting:
1. field_extractor.py: strip <pub-id> subtrees from the parent
reference text used for Smith-Waterman alignment. PMID/PMCID values
appear in JATS but not in PDF reference lists; including them pads the
needle length without adding matched characters, pushing quality below
the 0.8 threshold.
2. aligner.py: when a REFERENCE parent match fails at 0.8, retry with a
relaxed 0.65 threshold. JATS may concatenate author initials ("CA")
or order publisher/place differently from the PDF, producing valid
near-threshold alignments that the primary pass rejects.
Both issues caused parent matches to silently fail, skipping a bibl
entirely. Because partial_list scoring is ordered, one skipped bibl
shifts every subsequent field comparison by one position, cascading into
many false mismatches (e.g. PPR489375 dropped from 22/22 to 3/22 correct
issue matches due to a single missing bibl at position 2).
After these fixes the scielo regression shrinks from −0.124 to −0.009
mean while ORE remains at +0.363 mean, for a net +0.177 on citation.
…aining data - Add <chapter-title> to REFERENCE_ARTICLE_TITLE XPath so book-chapter and government-document refs are annotated with titles (recovers ~31 missing titles across scielo_preprints-jats documents) - Post-process <biblScope unit="page"> elements to add @from/@to attributes when the element contains a fpage-lpage range ("388 - 412"), matching the format expected by sciencebeam-judge for correct fpage extraction Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A truncated PDF reference (one where year/volume/URL are absent from the scanned text) produces a JATS needle much longer than the matching haystack fragment, giving quality ≈ 0.60. The previous 0.65 floor rejected these, leaving the bibl unmatched and cascading into position-shifted comparisons for every subsequent ordered field (reference_issue, reference_volume). Lowering to 0.55 captures these cases while still rejecting genuinely absent references. When the parent match now succeeds, the aligner can search for sub-fields beyond the reference-segmenter bibl boundary — recovering the source annotation and any continuation text the segmenter missed — producing a complete, separate citation bibl for the reference. Scielo impact (PPR459390 case): reference_issue/exact: 0.943 → 0.987 (+0.044) reference_volume/exact: 0.979 → 0.998 (+0.019) reference_source/lev: 0.984 → 0.996 (+0.012) scielo mean: 0.908 → 0.919 (+0.011) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…arent match Labels "1."–"9." appear immediately before the JATS parent text in the PDF (the parent text excludes the label), so the Smith-Waterman parent match starts after them. The sub-field search range began at p_start, placing single-digit labels outside it. Two-part fix: - Wire in _SUB_FIELD_PARENT_PRE_BUFFER (was defined but unused) and set it to 20 chars. Applied only for REFERENCE_LABEL sub-fields so other sub-fields (e.g. country) cannot false-match an identical word in an earlier context. - Add _is_punct_suffix_token helper so "1." tokenised as a single token passes _exact_number_match even though is_token_boundary_after fails for the digit alone. Improves citation label annotation (labels 1-9 now appear as <note>N</note>) and reference-segmenter label score (0.870 → 0.964 on scielo_preprints-jats). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
REFERENCE_LABEL sub-fields with a suffix (e.g. "20-" in scielo_preprints
references like "1-", "20-") are included in the JATS parent reference text,
so p_start is already at the label — extending the search backward would let
"20-" false-match the "20-" fragment inside a preceding DOI like "2020-0248".
The fix applies the pre-buffer only when the label text is a pure number
(digits only). Suffixed labels ("1-", "20-") and all other sub-fields keep
pre=0 and are confined to the parent match range.
The pre-buffer value stays at 20 to handle cases where the JATS parent text
starts after a publisher name ("Brasil."), leaving a ~14-char gap between the
label token and p_start.
Adds three unit tests covering the positive case, the DOI false-match
regression, and the long-gap (extra publisher text) scenario.
…tated
When a PDF tokenises "1-" as two tokens "1" and "-", Smith-Waterman
cannot match the 2-char needle "1-" against the spaced haystack "1 - …"
with sufficient quality (matched=1/2=0.5 < threshold=0.8). The parent
SW match compounds the issue: it aligns the full reference text by
skipping the leading digit and starting at "-" (p_start=2), so the
label sub-field search range excluded position 0 even for pure-digit
prefixes of suffixed labels.
Two targeted fixes in aligner.py:
1. _search_range: apply a small pre-buffer (_SUB_FIELD_LABEL_DIGIT_PRE_BUFFER=3)
for any REFERENCE_LABEL whose text starts with digits followed by a
suffix (e.g. "1-", "9-"). This shifts search_start back to 0 so the
digit token is within range.
2. _fuzzy_match_field_value: add _try_numeric_prefix_label_match as a
final fallback for REFERENCE_LABEL sub-fields when SW fails. It
splits the label into a digit prefix and suffix, locates the prefix
via _exact_number_match (which enforces token-start boundaries and
therefore cannot false-match inside "2020" or "0248"), then extends
the match to cover the immediately adjacent suffix token(s).
Two-digit suffixed labels (e.g. "10-") continue to be found via the
normal SW path (quality=1.0) and are unaffected. The existing
regression guard for "20-" not matching "2020-0248" is preserved.
…overlap When _extend_match_for_needle_tail greedily extends a reference's parent match into the first token of the next reference, prev_parent_end can exceed the current reference's parent start (a_start). Without a guard, effective_prev_end is set past the token boundary, causing _label_tokens_for_blocks to skip it as a mid-token start. Apply the same <= a_start two-column guard to prev_parent_end that already existed for prev_id_end.
When the gap between the last author initial and the next SW match point exceeds the fill threshold, the abbreviation period immediately following that initial has no entry in token_label_by_id. The trailing-period pass previously required entry is not None, so it silently skipped the period. Relaxing the condition to also handle entry is None attaches it correctly.
When a PDF orders a reference as author → source → article-title but the JATS element order is author → article-title → source, Smith-Waterman anchors the parent match on the article-title. The resulting p_start lands after the source in the haystack, so the source sub-field search (started at p_start) misses it entirely. The fix adds REFERENCE_SOURCE to the elif branch that sets the 200-char backward pre-buffer, already used by REFERENCE_AUTHOR. Both sub-fields share the same pre_parent_ref_floor guard that prevents the extended search from reaching into a preceding reference's matched range.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/110
Expand bibl boundary detection to whole-line level and fix a series of
alignment issues that caused sub-fields to be missed or mis-labelled in
the generated training data:
(e.g. "1-"), including a pre-buffer restriction to pure-number labels only
cover REFERENCE_SOURCE, so sources that appear before the parent SW anchor
in the PDF (e.g. when PDF order is author → source → title but JATS order
is author → title → source) are not missed