Skip to content

Add DeBERTa training and ONNX export pipeline for required phrases extraction#5222

Draft
Kaushik-Kumar-CEG wants to merge 12 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/train-deberta-model
Draft

Add DeBERTa training and ONNX export pipeline for required phrases extraction#5222
Kaushik-Kumar-CEG wants to merge 12 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/train-deberta-model

Conversation

@Kaushik-Kumar-CEG

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG commented Jul 5, 2026

Copy link
Copy Markdown

part of #5137. Adds the training and export pipeline for the required phrase
NER model, building on the dataset from #5105

the model tags which words in a rule are required phrases, so we can predict
{{ }} markers for license expressions that have no coverage yet

this PR:

  • train_model.py - loads the BIOES JSONL, aligns word labels to subwords,
    runs training with per epoch eval and entity F1. Small CLI (epochs, lrs,
    --limit, --resume)
  • phrase_model.py - DeBERTa-v3-large with a word level CRF head and layer
    wise lr decay
  • export_onnx.py - exports the emissions to ONNX and the CRF transitions
    beside it, with a numpy viterbi decoder for CPU inference
  • test_train_model.py - unit tests for the alignment, metrics and viterbi
  • etc/requirements-ml.txt - training only deps, separate from the runtime

update: first full training run is done

Test results on unseen licenses:

  • F1: 0.822
  • precision: 0.811
  • recall: 0.833

This is a first run and already improves on my earlier BIO prototype (F1 ~0.79)
I expect later iterations (checkpoint averaging, more tuning, maybe adding negative examples) to do better

model: https://huggingface.co/Kaushik-Kumar-CEG/scancode-required-phrases-deberta-bioes-crf

disclosure : used Claude to help review and clean up bugs in script

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG marked this pull request as draft July 5, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant