Skip to content

chore: model card #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TeunB6 merged 2 commits into from
Apr 3, 2026
Merged

chore: model card #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72a6a72
chore: model card
squidistaken Apr 3, 2026
4e6965b
remove the todo
squidistaken Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions MODELCARD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# DistilBERTClassifer on 4-way News Topic Classification
## Model Details
* **Model Name:** DistilBERTClassifer.
* **Base Model:** `distilbert-base-uncased` from [HuggingFace](https://huggingface.co/distilbert/distilbert-base-uncased).
* **Developers:** Teun Boersma (s5195179), Julian Sprietsma (s5096219), Marcus Harald Olof Persson (s5343798).
* **Model Type:** Transformer-based sequence classifier.
* **Task:** Multi-class text classification (4 classes).
* **Language:** English.
## Intended Usage and Limitations
This model's primary usage is for the classification of news articles into four specific classes: World, Sports, Business, and Sci/Tech. Regarding scope, the model is specifically finetuned on this dataset and may not generalise to non-news domains or different language contexts without further finetuning.
## Training Data
The dataset originates from the [AGNews dataset](https://huggingface.co/datasets/sh0416/ag_news). For preprocessing, the model utilises a combined "text" field created by concatenating the "title" and "description" columns from the dataset. With labelling of data:
* 1: World.
* 2: Sports.
* 3: Business.
* 4: Sci/Tech.
## Training Procedure
The model was trained using a custom `Trainer` class on the University of Groningen's high-performance computing cluster Hábrók.
* **Hardware:** 1x NVIDIA A100 GPU, 4 CPUs, 16GB RAM.
* **Optimiser:** AdamW with a learning rate of $2 \times 10^{-5}$.
* **Loss Function:** `CrossEntropyLoss`.
* **Batch Size:** 32.
* **Early Stopping:** Triggered after 4 epochs without improvements, where the best model weights being restored from epoch 3.

| Epoch | Train Loss | Eval/Valid Loss | Note |
| - | - | - | - |
| 1 | 0.2352 | 0.2636 | |
| 2 | 0.1410 | 0.2690 | |
| 3 | 0.1011 | 0.2356 | Best Model |
| 4 | 0.0695 | 0.2955 | |
| 5 | 0.0478 | 0.3328 | |
| 6 | 0.0331 | 0.3674 | Early stopping triggered |
## Results
| | Dev Set | Test Set |
| - | - | - |
| Accuracy | 0.9226 | 0.9424 |
F1 Score (Weighted) | 0.9224 | 0.9424 |

| Predicted \ Actual | World | Sports | Business | Sci/Tech |
| - | - | - | - | - |
World | 1785 | 14 | 57 | 44 |
Sports | 10 | 1880 | 6 | 4 |
Business | 32 | 8 | 1724 | 136 |
Sci/Tech | 26 | 8 | 93 | 1773 |
## Biases and Risks
* The model inherits potential biases from the `distilbert-base-uncased` pretraining data.
* Performance may drop significantly if applied to news data significantly newer than the training set.
* The model most frequently confuses "Business" news with "Sci/Tech".
## Environmental Impact
* **Data Centre:** Hábrók is maintained at the Coenraad Bron Center, which is [reported to operate at a Power Usage Effective (PUE) of 1.25 or below](https://www.rug.nl/groundbreakingwork/projects/datacenter/design-en-duurzaamheid).
* **Location:** Groningen, the Netherlands.
* **Total Runtime:** Approximately 51 minutes and 52 seconds, rounded up to an hour.
* **Carbon Emitted:** From the [Machine Learning Emissions Calculator](https://mlco2.github.io/impact), assuming the carbon effiency is OECD's 2014 yearly average ($0.432 \text{ kg}/\text{kWh}$), then $0.11 \times 1.25 = 0.1375 \text{ kg CO}_2$ was approximately emitted.
5 changes: 1 addition & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,7 @@

This project tackles 4-way news topic classification on the [AGNews](https://huggingface.co/datasets/sh0416/ag_news) on the classes World, Sports, Business, Sci/Tech. It contains the three deliverables required for Natural Language Processing (WBAI059-05).

## Team
* Teun Boersma (s5195179)
* Julian Sprietsma (s5096219)
* Marcus Harald Olof Persson (s5343798)
The model card (from Assignment 3) can be found in this respository as well: [MODELCARD.md](MODELCARD.md)

## Developing
This project uses [uv](https://docs.astral.sh/uv/) for dependency and environment management.
Expand Down
1 change: 0 additions & 1 deletion src/showcase/assignment3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ class Assignment3Showcase:

ds = AGNews2Trans()

# TODO: Implement robustness/slice evaluations.
def __call__(self, choice: Optional[int] = None) -> None:
"""Call the showcase.

Expand Down