Generate JATS-guided training data without model cascade dependency#675
Merged
Conversation
Introduces a new JATS XML → LayoutDocument alignment pipeline that labels PDF tokens directly from JATS ground truth, removing the cascade dependency on upstream model predictions. New modules under sciencebeam_parser/training/jats/: - field_extractor: parses JATS XML into (text, field_name) pairs using the xml-mapping vocabulary; emits author names in Given-Surname order to match typical PDF byline layout - aligner: fuzzy Smith-Waterman alignment (threshold 0.8) with sliding windows; handles line-break hyphens, reference_floor separation from body_content_end, and body-floor soft fallback for out-of-order paragraphs; no exact-match fast path (quality > speed) - segmentation: derives per-line segmentation labels from token-level field annotations via majority vote - annotated_document: token → field label store and coverage_ratio() - coverage, text_normalizer, field_vocab: supporting utilities generate_data.py extended with --source-xml-path, --num-workers, --required-fields, --require-matching-fields; the JATS path feeds the existing TeiTrainingDataGenerators without changing their interface.
… header model Per GROBID header annotation guidelines: - Authors: emit one AUTHOR field value per <contrib-group>, merging all author names in Given-Surname order with their affiliation/fn/corresp xref markers appended. This covers the full byline span (including separating commas and connectors) with a single aligner pass. - Keywords title: remove KEYWORDS_TITLE from HEADER_LABEL_BY_FIELD so the header model leaves the "Keywords" heading token unlabelled. KEYWORDS_TITLE is still emitted and mapped to <header> in SEGMENTATION_LABEL_BY_FIELD so the segmentation model keeps the heading within the header region.
Each <aff> element now produces its own <byline><affiliation>...</affiliation></byline> block in the header TEI, followed by a separate <address> element where the JATS provides geographic content. - Assign a monotonically increasing instance_id per main JATS field value in the aligner; the header label fn emits B- on instance change rather than label change, so consecutive affiliations with the same label each start a new <byline> block - Address content is collected in document order via _aff_addr_parts: <addr-line> and <country> element text, plus the tail text of <institution> elements (which is where city/postcode sit in semi-structured JATS that omits <addr-line>) - AUTHOR_AFF_ADDR sub-field tokens are mapped to <address>; individual city, postcode, region and country sub-fields narrow-label within that span
Institution tail text (text after </institution>) is only collected as address content when the same <aff> also has a <country> or <addr-line> element. Without that anchor the tail may be a department-name continuation split across two <institution> tags rather than a geographic address.
Two bugs prevented the timeout from firing: 1. signal.alarm (SIGALRM) only fires between Python bytecodes and cannot interrupt a tight loop in a compiled C extension. LocalSequenceMatcher runs in align_fast_utils.cpython-311-x86_64.so, so the alarm was delivered but never processed. 2. concurrent.futures.as_completed() blocks until a future completes before yielding it. The future.result(timeout=...) call came after as_completed() had already yielded — the future was always done by then, so the timeout could never fire. A hung future blocked as_completed() indefinitely. Replace both paths with multiprocessing.Pool.apply_async().get(timeout=). Pool.terminate() sends SIGTERM to the worker OS process, killing the C extension regardless of its internal state. For the serial path the pool is recreated after each timeout so subsequent documents can run. For the parallel path all documents are submitted upfront; stuck workers are cleaned up by pool.terminate() in the finally block.
ORE papers have two blocks that were incorrectly labeled <body>: 1. Page 2 front-matter metadata (competing interests, grant information, copyright / licence): the corresponding JATS elements (<funding-group/funding-statement>, <permissions/copyright-statement>, <permissions/license/license-p>) were not extracted, so those tokens fell through to the <body> default. Added FUNDING and COPYRIGHT fields mapped to <header>; extracted via a new _iter_front_publication_values helper. 2. Peer-review reports (sub-articles, pages 13-18 in a typical ORE paper): <sub-article> content was not extracted at all. Added SUB_ARTICLE field mapped to <other> and a new _iter_sub_article_values method that yields paragraphs and titles from every <sub-article> in document order. Also increased _DEFAULT_FRONT_MAX_START_LINE_INDEX from 40 to 80. ORE papers have a second front-matter page whose JATS-matched lines (author notes, funding, copyright) were being cleared by the threshold because they start at line ~60+ after a long abstract.
On macOS, Python uses the 'spawn' start method for new processes rather than 'fork'. When _run_serial always spawned a multiprocessing.Pool, the worker process reimported the module from scratch and did not inherit test monkey-patches on ScienceBeamParser, causing _worker_init to try to load real models and producing no output files. The pool only exists to let pool.terminate() kill workers stuck in C extensions (signal.alarm cannot interrupt them). Without a timeout that mechanism is never used, so there is no benefit to spawning a subprocess. _run_serial now takes two paths: - document_timeout == 0: calls _worker_init() and _worker_process() inline in the current process. No subprocess overhead; test patches remain active on all platforms. - document_timeout > 0: keeps the multiprocessing.Pool path unchanged so hung C-extension workers can still be terminated.
Fix 1 (_POST_BODY_FIELDS): sub-article fields (e.g. ORE peer-review boxes) are now searched from last_match_end rather than from the front-matter window, preventing author-response text that quotes the paper verbatim from overwriting figure/table tokens — fixes Figure 10 missing in 1-123_v2. Fix 2 (anchor+chain): Smith-Waterman produces many tiny (1–4 char) scatter blocks while traversing interleaved sidebar content; the previous gap-fill loop chained them via single-space gaps and labelled entire sidebar words. Replace with an anchor+chain strategy: a block is only labelled if it is ≥ 5 chars (anchor) or starts within 3 chars of the previous included block; fields whose entire text is shorter than the anchor threshold fall back to labelling all blocks. Verified on 1-114_v2: the Open Peer Review sidebar is now fully clean while both abstract segments remain correctly tagged. Add test_abstract_does_not_label_sidebar_content to pin the new behaviour.
Adds a pipeline for fetching PDF + JATS XML pairs that are safe to use as training data based on licence. - benchmarks/training-source.yml: separate config listing the two confirmed CC-BY corpora (ore, scielo_preprints-jats) with smoke/small/full sampling modes; null sample size means fetch all records - benchmarks/fetch.py: handle None sample size in fetch_data/fetch_gold (null in YAML → fetch entire corpus); add fetch_training_source which filters to cc_by_corpora before delegating to fetch_data - benchmarks/fetch_training_source_cli.py: CLI entry point for the new fetch function (python -m benchmarks.fetch_training_source_cli) - Makefile: SOURCE_TRAINING_* variables, dev-fetch-training-source target, dev-generate-training-data updated to source from SOURCE_TRAINING_DATA (default: data/source-training-data) with a preflight check that fails clearly if TRAINING_DATA_OUTPUT does not exist
… data generation Replaces a flat generate_data call in the Makefile with a Python CLI (benchmarks/generate_training_data_cli.py) that reads cc_by_corpora from the training-source config and invokes generate_data once per corpus, writing output to <output-path>/<split>/<corpus>/. Unknown flags are forwarded to generate_data, missing corpus source directories are skipped gracefully, and failures are collected so all corpora are attempted before exiting 1.
Previously the aligner only labeled tokens from the first long matching
block onward, so short segments before it — DOI components ("10", ".",
"3233", "/") and hyphenated word prefixes — were silently left unlabeled
in the generated training data.
Adds a backward pre-anchor pass that includes tightly adjacent short
blocks before the first anchor. A token-boundary guard prevents a
spurious SW match on the tail of a preceding token from overwriting an
already-correct heading label with a paragraph label.
DOI tokens were being written to <ptr type="web"> in generated citation training data, but the evaluation reads reference_doi exclusively from <idno[@type="DOI"]>. Scores were structurally zero regardless of how many DOI tokens were correctly labeled.
Reference DOIs were written to <ptr type="web"> but the evaluation reads from <idno[@type="DOI"]>, so DOI scores were structurally zero. Bare DOIs from JATS pub-id[@pub-id-type="doi"] now emit <idno type="DOI">. Adds a REFERENCE_WEB sub-field that extracts <ext-link> elements whose text is a URL (filters out "Reference Source" hyperlink labels), emitting <ptr type="web"> for the full URL — matching grobid's convention for doi.org and other reference links.
de-code
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/103
Previously, producing training data for any downstream model (header,
citation, fulltext) required first running all upstream models in
sequence — bad segmentation corrupted every label downstream.
This PR adds a direct path: given paired (PDF, JATS XML) inputs, field
values are extracted from JATS and aligned to the PDF layout tokens,
producing correctly-labelled TEI/DeLFT training files for all cascade
model levels without invoking any model.
What this enables:
data from their existing JATS deposits
convention and evaluation config), so DOI recall scores reflect actual
model performance
, enabling a new reference_url evaluation field
single-token identifiers) that were previously dropped by the pre-anchor gap
Scope of changes:
annotated document, coverage, segmentation, text normalizer
--require-matching-fields, --required-fields flags
new label -> ptr[@type="web"] element
and reference_url field definitions