Add grobid_custom_hybrid profile with SciELO-preprints reference segmenter#680
Draft
de-code wants to merge 9 commits into
Draft
Add grobid_custom_hybrid profile with SciELO-preprints reference segmenter#680de-code wants to merge 9 commits into
de-code wants to merge 9 commits into
Conversation
…enter Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in.
ScienceBeam Parser EvaluationOverall (280 docs across 6 corpora)grobid 0.9.0-crf (default): 280 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 280 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 280 docs
biorxiv (47 docs)grobid 0.9.0-crf (default): 47 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 47 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 47 docs
ore (49 docs)grobid 0.9.0-crf (default): 49 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 49 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 49 docs
pkp (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 50 docs
scielo_br (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 50 docs
scielo_mx (34 docs)grobid 0.9.0-crf (default): 34 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 34 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 34 docs
scielo_preprints-jats (50 docs)grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:main-0baf0824-20260701.1904 (grobid_crf): 50 docs | sciencebeam-parser:pr-680-c09ca2e8-20260703.1147 (grobid_custom_hybrid): 50 docs
|
…enter Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in. Also wires up the profile:grobid_custom_hybrid PR label in the benchmark workflow so it selects this profile.
Replaces the hardcoded per-profile contains() checks with a step that extracts any profile:<name> label directly, validated against a safe character set, so new profiles no longer require a benchmark.yml edit.
Authenticate CI to GCP via Workload Identity Federation so the grobid_custom_hybrid profile can pull the reference-segmenter model from gs://sciencebeam-v2-models while the trained model's release is not yet public.
TensorFlow's built-in gs:// filesystem support only recognizes plain service-account JSON keys, not the Workload Identity Federation external_account credential files google-github-actions/auth produces - those reference a credential source that can't be resolved from inside the sciencebeam-parser container, causing anonymous (unauthenticated) requests. Decode a service account key from the GCS_MODEL_READER_SA_KEY_B64 secret instead and mount it into the container.
Points the citation model at the new CustomBidLSTM_CRF model in the private sciencebeam-v2-models GCS bucket, replacing the grobid 0.9.0 wapiti model, while its release is not yet public.
Collaborator
Author
|
This doesn't look that good unfortunately |
Custom hybrid profiles are expected to grow beyond this one, so inherit from a base profile via a code-resolved `extends` key rather than a YAML anchor - it deep-merges instead of replacing whole model entries, gives clear errors for unknown/circular references, and keeps working if profiles ever get split across files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/113
Introduces a new sequence-model profile that overlays the grobid 0.9.0 CRF/wapiti baseline with our own retrained DELFT models, starting with a CustomBidLSTM_CRF reference segmenter trained on SciELO preprints. Intended to grow as more custom models are retrained and swapped in.