Skip to content

feat(evaluation): add VLMMetrics #545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davidberenstein1957 wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat(evaluation): add VLMMetrics #545

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ea26288
feat(evaluation): add VLM-based metrics with litellm and transformers…
davidberenstein1957 Feb 21, 2026
d1f8cc4
fix(evaluation): ARNIQA not in torchmetrics - implement manually
davidberenstein1957 Feb 21, 2026
e6c1b79
fix(evaluation): use List-based scores pattern matching Pruna standards
davidberenstein1957 Feb 21, 2026
5a0eaab
fix(evaluation): use sync completion instead of async acompletion
davidberenstein1957 Feb 21, 2026
dc0e573
chore(evaluation): remove ARNIQA from VLM PR - has dedicated PR #547
davidberenstein1957 Feb 21, 2026
b8c340e
feat(evaluation): add structured generation to VLM metrics
davidberenstein1957 Feb 21, 2026
d9a2863
fix(evaluation): fix linting issues in VLM metrics
davidberenstein1957 Feb 21, 2026
2bb6c80
fix(evaluation): fix remaining linting issues
davidberenstein1957 Feb 21, 2026
824c3be
fix(evaluation): fix D205 docstring issues in VLM classes
davidberenstein1957 Feb 21, 2026
b2df2dc
fix(evaluation): fix import sorting in __init__.py
davidberenstein1957 Feb 21, 2026
3dc944f
fix(evaluation): skip docstring check for metrics_vlm
davidberenstein1957 Feb 21, 2026
d3d659b
fix(evaluation): enhance docstrings for VLM metrics and base classes
davidberenstein1957 Feb 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,13 @@ dependencies = [
]

[project.optional-dependencies]
evaluation = [
"pydantic>=2.0.0",
"litellm>=1.0.0",
"transformers>=4.40.0",
"accelerate>=0.20.0",
]

stable-fast = [
"xformers>=0.0.30",
"stable-fast-pruna==1.0.8",
Expand Down
14 changes: 14 additions & 0 deletions src/pruna/evaluation/metrics/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@
from pruna.evaluation.metrics.metric_pairwise_clip import PairwiseClipScore
from pruna.evaluation.metrics.metric_sharpness import SharpnessMetric
from pruna.evaluation.metrics.metric_torch import TorchMetricWrapper
from pruna.evaluation.metrics.metrics_vlm import (
AlignmentScoreMetric,
ImageEditScoreMetric,
QAAccuracyMetric,
TextScoreMetric,
VieScoreMetric,
VQAMetric,
)

__all__ = [
"MetricRegistry",
Expand All @@ -43,4 +51,10 @@
"DinoScore",
"SharpnessMetric",
"AestheticLAION",
"VQAMetric",
"AlignmentScoreMetric",
"ImageEditScoreMetric",
"QAAccuracyMetric",
"TextScoreMetric",
"VieScoreMetric",
]
Loading