Skip to content

Bump pkp and scielo_mx variant to v2 to refresh benchmark predictions#684

Merged
HazalCiplak merged 2 commits into
mainfrom
dummy-branch
Jul 8, 2026
Merged

Bump pkp and scielo_mx variant to v2 to refresh benchmark predictions#684
HazalCiplak merged 2 commits into
mainfrom
dummy-branch

Conversation

@HazalCiplak

Copy link
Copy Markdown
Member

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

ScienceBeam Parser Evaluation

Overall (280 docs across 6 corpora)

grobid 0.9.0-crf (default): 280 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 280 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.532 0.528 -0.004
title (levenshtein) string 0.665 0.669 +0.004
title (edit_sim) string 0.677 0.677 +0.001
abstract (levenshtein) string 0.639 0.636 -0.004
abstract (edit_sim) string 0.664 0.665 +0.000
author_full_names (levenshtein) partial_ulist 0.731 0.724 -0.007
author_full_names (edit_sim) partial_ulist 0.742 0.740 -0.002
affiliation_text (levenshtein) partial_ulist 0.495 0.483 -0.012
affiliation_text (edit_sim) partial_ulist 0.600 0.580 -0.020
keywords (levenshtein) partial_ulist 0.641 0.000 -0.641
keywords (edit_sim) partial_ulist 0.603 0.000 -0.603
body_section_titles (levenshtein) partial_list 0.338 0.338 -0.000
body_section_titles (edit_sim) partial_list 0.328 0.332 +0.004
acknowledgement (levenshtein) string 0.584 0.504 -0.080
acknowledgement (edit_sim) string 0.595 0.516 -0.080
first_reference_text (levenshtein) string 0.533 0.541 +0.008
first_reference_text (edit_sim) string 0.663 0.670 +0.006
reference_title (levenshtein) partial_list 0.446 0.429 -0.017
reference_title (edit_sim) partial_list 0.454 0.448 -0.007
reference_doi (levenshtein) partial_ulist 0.720 0.602 -0.117
reference_doi (edit_sim) partial_ulist 0.665 0.568 -0.097
biorxiv (47 docs)

grobid 0.9.0-crf (default): 47 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 47 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.867 0.840 -0.028
title (levenshtein) string 0.956 0.956 +0.000
title (edit_sim) string 0.942 0.942 +0.000
abstract (levenshtein) string 0.779 0.779 +0.000
abstract (edit_sim) string 0.773 0.780 +0.007
author_full_names (levenshtein) partial_ulist 0.977 0.975 -0.002
author_full_names (edit_sim) partial_ulist 0.946 0.951 +0.005
affiliation_text (levenshtein) partial_ulist 0.877 0.816 -0.060
affiliation_text (edit_sim) partial_ulist 0.877 0.821 -0.056
keywords (levenshtein) partial_ulist 0.835 0.000 -0.835
keywords (edit_sim) partial_ulist 0.834 0.000 -0.834
body_section_titles (levenshtein) partial_list 0.456 0.376 -0.080
body_section_titles (edit_sim) partial_list 0.432 0.373 -0.060
acknowledgement (levenshtein) string 0.732 0.548 -0.184
acknowledgement (edit_sim) string 0.731 0.586 -0.146
first_reference_text (levenshtein) string 0.905 0.905 +0.000
first_reference_text (edit_sim) string 0.869 0.869 +0.000
reference_title (levenshtein) partial_list 0.694 0.730 +0.035
reference_title (edit_sim) partial_list 0.653 0.690 +0.036
reference_doi (levenshtein) partial_ulist 0.865 0.858 -0.007
reference_doi (edit_sim) partial_ulist 0.824 0.821 -0.003
ore (49 docs)

grobid 0.9.0-crf (default): 49 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 49 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.537 0.515 -0.022
title (levenshtein) string 0.600 0.639 +0.039
title (edit_sim) string 0.717 0.742 +0.025
abstract (levenshtein) string 0.693 0.693 +0.000
abstract (edit_sim) string 0.738 0.740 +0.002
author_full_names (levenshtein) partial_ulist 0.802 0.798 -0.004
author_full_names (edit_sim) partial_ulist 0.792 0.794 +0.002
affiliation_text (levenshtein) partial_ulist 0.564 0.507 -0.057
affiliation_text (edit_sim) partial_ulist 0.577 0.514 -0.063
keywords (levenshtein) partial_ulist 0.297 0.000 -0.297
keywords (edit_sim) partial_ulist 0.292 0.000 -0.292
body_section_titles (levenshtein) partial_list 0.198 0.228 +0.030
body_section_titles (edit_sim) partial_list 0.232 0.271 +0.038
acknowledgement (levenshtein) string 0.921 0.885 -0.035
acknowledgement (edit_sim) string 0.935 0.881 -0.054
first_reference_text (levenshtein) string 0.444 0.469 +0.024
first_reference_text (edit_sim) string 0.620 0.606 -0.014
reference_title (levenshtein) partial_list 0.229 0.283 +0.054
reference_title (edit_sim) partial_list 0.262 0.312 +0.049
reference_doi (levenshtein) partial_ulist 0.641 0.033 -0.609
reference_doi (edit_sim) partial_ulist 0.494 0.010 -0.485
pkp (50 docs)

grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 50 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.676 0.676 +0.000
title (levenshtein) string 0.810 0.795 -0.015
title (edit_sim) string 0.781 0.759 -0.022
abstract (levenshtein) string 0.746 0.746 +0.000
abstract (edit_sim) string 0.735 0.737 +0.002
author_full_names (levenshtein) partial_ulist 0.785 0.777 -0.008
author_full_names (edit_sim) partial_ulist 0.780 0.769 -0.011
affiliation_text (levenshtein) partial_ulist 0.618 0.617 -0.002
affiliation_text (edit_sim) partial_ulist 0.680 0.651 -0.029
keywords (levenshtein) partial_ulist 0.887 0.000 -0.887
keywords (edit_sim) partial_ulist 0.876 0.000 -0.876
body_section_titles (levenshtein) partial_list 0.617 0.647 +0.030
body_section_titles (edit_sim) partial_list 0.560 0.587 +0.027
acknowledgement (levenshtein) string 0.727 0.667 -0.061
acknowledgement (edit_sim) string 0.739 0.656 -0.083
first_reference_text (levenshtein) string 0.907 0.920 +0.013
first_reference_text (edit_sim) string 0.876 0.896 +0.020
reference_title (levenshtein) partial_list 0.706 0.619 -0.087
reference_title (edit_sim) partial_list 0.697 0.620 -0.077
reference_doi (levenshtein) partial_ulist 0.984 0.984 +0.000
reference_doi (edit_sim) partial_ulist 0.984 0.952 -0.032
scielo_br (50 docs)

grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 50 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.462 0.485 +0.023
title (levenshtein) string 0.485 0.485 +0.000
title (edit_sim) string 0.512 0.512 +0.000
abstract (levenshtein) string 0.697 0.677 -0.020
abstract (edit_sim) string 0.666 0.652 -0.013
author_full_names (levenshtein) partial_ulist 0.607 0.592 -0.015
author_full_names (edit_sim) partial_ulist 0.658 0.653 -0.005
affiliation_text (levenshtein) partial_ulist 0.000 0.000 +0.000
affiliation_text (edit_sim) partial_ulist 0.298 0.295 -0.003
keywords (levenshtein) partial_ulist 0.538 0.000 -0.538
keywords (edit_sim) partial_ulist 0.515 0.000 -0.515
body_section_titles (levenshtein) partial_list 0.265 0.291 +0.026
body_section_titles (edit_sim) partial_list 0.275 0.295 +0.020
acknowledgement (levenshtein) string 0.667 0.571 -0.095
acknowledgement (edit_sim) string 0.704 0.618 -0.086
first_reference_text (levenshtein) string 0.083 0.043 -0.041
first_reference_text (edit_sim) string 0.399 0.394 -0.004
reference_title (levenshtein) partial_list 0.388 0.318 -0.070
reference_title (edit_sim) partial_list 0.407 0.365 -0.042
reference_doi (levenshtein) partial_ulist 0.934 0.897 -0.038
reference_doi (edit_sim) partial_ulist 0.890 0.842 -0.048
scielo_mx (34 docs)

grobid 0.9.0-crf (default): 34 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 34 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.057 0.057 +0.000
title (levenshtein) string 0.489 0.489 +0.000
title (edit_sim) string 0.497 0.497 +0.000
abstract (levenshtein) string 0.111 0.111 +0.000
abstract (edit_sim) string 0.402 0.404 +0.001
author_full_names (levenshtein) partial_ulist 0.720 0.720 +0.000
author_full_names (edit_sim) partial_ulist 0.730 0.730 +0.000
affiliation_text (levenshtein) partial_ulist 0.209 0.207 -0.002
affiliation_text (edit_sim) partial_ulist 0.461 0.462 +0.001
keywords (levenshtein) partial_ulist 0.627 0.000 -0.627
keywords (edit_sim) partial_ulist 0.579 0.000 -0.579
body_section_titles (levenshtein) partial_list 0.000 0.000 +0.000
body_section_titles (edit_sim) partial_list 0.000 0.000 +0.000
acknowledgement (levenshtein) string 0.000 0.000 +0.000
acknowledgement (edit_sim) string 0.000 0.000 +0.000
first_reference_text (levenshtein) string 0.474 0.474 +0.000
first_reference_text (edit_sim) string 0.560 0.561 +0.001
reference_title (levenshtein) partial_list 0.327 0.326 -0.001
reference_title (edit_sim) partial_list 0.348 0.351 +0.003
reference_doi (levenshtein) partial_ulist 0.000 0.000 +0.000
reference_doi (edit_sim) partial_ulist 0.000 0.000 +0.000
scielo_preprints-jats (50 docs)

grobid 0.9.0-crf (default): 50 docs | sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf): 50 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:pr-684-97e8eea3-20260707.1526 (grobid_crf) Δ grobid 0.9.0-crf (default)
title (exact) string 0.462 0.462 +0.000
title (levenshtein) string 0.611 0.611 +0.000
title (edit_sim) string 0.571 0.572 +0.001
abstract (levenshtein) string 0.649 0.649 +0.000
abstract (edit_sim) string 0.595 0.600 +0.004
author_full_names (levenshtein) partial_ulist 0.509 0.499 -0.010
author_full_names (edit_sim) partial_ulist 0.557 0.553 -0.004
affiliation_text (levenshtein) partial_ulist 0.636 0.686 +0.051
affiliation_text (edit_sim) partial_ulist 0.677 0.712 +0.035
keywords (levenshtein) partial_ulist 0.662 0.000 -0.662
keywords (edit_sim) partial_ulist 0.522 0.000 -0.522
body_section_titles (levenshtein) partial_list 0.388 0.375 -0.013
body_section_titles (edit_sim) partial_list 0.367 0.361 -0.006
acknowledgement (levenshtein) string 0.286 0.200 -0.086
acknowledgement (edit_sim) string 0.286 0.200 -0.086
first_reference_text (levenshtein) string 0.387 0.438 +0.050
first_reference_text (edit_sim) string 0.635 0.667 +0.031
reference_title (levenshtein) partial_list 0.304 0.283 -0.020
reference_title (edit_sim) partial_list 0.333 0.331 -0.002
reference_doi (levenshtein) partial_ulist 0.671 0.654 -0.017
reference_doi (edit_sim) partial_ulist 0.592 0.606 +0.015

@HazalCiplak HazalCiplak merged commit 0368976 into main Jul 8, 2026
7 checks passed
@HazalCiplak HazalCiplak deleted the dummy-branch branch July 8, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant