Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,17 @@ make check-safety

The SDK has a minimal structure centered on the `Client` class:

- **`qualifire/client.py`**: Main `Client` class with two public methods:
- **`qualifire/client.py`**: Main `Client` class with three public methods:
- `evaluate()` - Run ad-hoc evaluations with various checks (hallucinations, grounding, PII, content moderation, etc.)
- `invoke_evaluation()` - Run pre-configured evaluations from the Qualifire dashboard
- `compile_prompt()` - Compile a prompt with topic scoping and policy targets

- **`qualifire/types.py`**: All data classes and enums:
- `EvaluationRequest`/`EvaluationResponse` - API request/response structures
- `LLMMessage`, `LLMToolCall`, `LLMToolDefinition` - Message and tool types for conversation-based evaluation
- `ModelMode` (SPEED/BALANCED/QUALITY) - Quality vs speed trade-off for checks
- `SyntaxCheckArgs` - Configuration for syntax validation
- `PolicyTarget`, `CompilePromptResponse` - Types for prompt compilation with topic scoping

- **`qualifire/utils.py`**: Helper functions for API key and base URL resolution from environment variables

Expand Down
4 changes: 2 additions & 2 deletions assets/images/coverage.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "qualifire"
version = "0.13.0"
version = "0.13.1"
description = "Qualifire Python SDK"
authors = [{ name = "qualifire-dev", email = "dror@qualifire.ai" }]
requires-python = ">=3.8,<4"
Expand Down
12 changes: 12 additions & 0 deletions qualifire/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ def evaluate(
grounding_multi_turn_mode: bool = False,
policy_multi_turn_mode: bool = False,
policy_target: PolicyTarget = PolicyTarget.BOTH,
topic_scoping_mode: Optional[ModelMode] = None,
topic_scoping_multi_turn_mode: bool = False,
topic_scoping_target: PolicyTarget = PolicyTarget.BOTH,
allowed_topics: Optional[List[str]] = None,
) -> Union[EvaluationResponse, None]:
"""
Evaluates the given input and output pairs.
Expand Down Expand Up @@ -105,6 +109,10 @@ def evaluate(
:param grounding_multi_turn_mode: Enable multi-turn mode for grounding check.
:param policy_multi_turn_mode: Enable multi-turn mode for policy check.
:param policy_target: Target for policy checks (input/output/both).
:param topic_scoping_mode: Model mode for topic scoping check (speed/balanced/quality).
:param topic_scoping_multi_turn_mode: Enable multi-turn mode for topic scoping check.
:param topic_scoping_target: Target topic for topic scoping check.
:param allowed_topics: List of allowed topics for topic scoping check.

:return: An EvaluationResponse object containing the evaluation results.
:raises Exception: If an error occurs during the evaluation.
Expand Down Expand Up @@ -224,6 +232,10 @@ def evaluate(
grounding_multi_turn_mode=grounding_multi_turn_mode,
policy_multi_turn_mode=policy_multi_turn_mode,
policy_target=policy_target,
topic_scoping_mode=topic_scoping_mode,
topic_scoping_multi_turn_mode=topic_scoping_multi_turn_mode,
topic_scoping_target=topic_scoping_target,
allowed_topics=allowed_topics,
)

response = requests.post(
Expand Down
4 changes: 4 additions & 0 deletions qualifire/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ class EvaluationRequest(BaseModel):
grounding_multi_turn_mode: bool = False
policy_multi_turn_mode: bool = False
policy_target: PolicyTarget = PolicyTarget.BOTH
topic_scoping_mode: Optional[ModelMode] = None
topic_scoping_multi_turn_mode: bool = False
topic_scoping_target: Optional[str] = None
allowed_topics: Optional[List[str]] = None

@model_validator(mode="after")
def validate_model(self) -> "EvaluationRequest":
Expand Down
2 changes: 1 addition & 1 deletion uv.lock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to bump the version in pyproject.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.