Skip to content

Conversation

@AmeenP
Copy link
Collaborator

@AmeenP AmeenP commented Sep 30, 2025

Summary

This PR adds Python entry point support to load_environment(), enabling namespace-aware environment loading with owner/name format.

Changes

Core Functionality

  • New priority-based loading system:

    1. Explicit target spec (module:attr) - highest priority
    2. Entry point exact match - preferred for owner/name format
    3. Module import fallback - backward compatibility
  • New helper functions:

    • _call_loader(): Common loader with parameter inspection and logging
    • _load_from_target_spec(): Handles explicit module:attr targets
    • _load_via_entry_point_exact(): Entry point discovery and loading via importlib.metadata

Improvements

  • Cleaner, more modular code structure
  • Better error messages with contextual information
  • Support for namespaced environment IDs (owner/name)
  • Backward compatible with existing environments

Testing

All tests passed:

  • ✓ Entry point discovery and registration
  • ✓ Load via entry point (exact match ameenp/wordle)
  • ✓ Load via fallback (module name wordle)
  • ✓ Load via explicit target spec (wordle:load_environment)
  • ✓ Priority order working correctly

Example Usage

from verifiers import load_environment

# Via entry point (preferred for owner/name)
env = load_environment("ameenp/wordle")

# Via explicit target
env = load_environment("wordle:load_environment")

# Via fallback (backward compatible)
env = load_environment("wordle")

Related

- Add --header CLI argument with action="append" for multiple headers
- Parse headers with "Name: Value" format, splitting on first colon
- Thread headers through eval_environment() and setup_client()
- Update httpx AsyncClient to use custom headers for API requests
- Fix function signature consistency (always Dict[str, str], not Optional)
- Simplify headers assignment (remove unnecessary 'or None')

Usage:
  vf-eval gsm8k -- --header "X-Team-ID: abc" --header "X-Trace: 123"
- Remove redundant comment line
- Improve help text: 'Extra HTTP header to pass to inference API'
- Implement priority-based loading: explicit target > entry point > fallback
- Add entry point exact match loading via importlib.metadata
- Support explicit target spec with 'module:attr' syntax
- Maintain backward compatibility with module name fallback
- Extract common loading logic into _call_loader helper
- Improve logging and error messages
@AmeenP AmeenP closed this Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant