CONTRIBUTING.md

Contributing

Thank you for your interest in contributing to Inspect Eval Convertor!

Adding New Examples

To add a new example converter:

1. Create example directory:

   mkdir examples/my_format

2. Add input data (input.json or other format):

   • Use synthetic data only (no private content)
   • Make it clear and simple (3-5 samples is fine)
   • Include edge cases if relevant

3. Add task script (task.py):

   • Follow the pattern from existing examples
   • Use @task decorator to create your task function
   • Use task_main() from inspect_convertor.utils to run the task
   • Include docstrings explaining the format
   • Handle errors gracefully
   • Store messages and scores in sample metadata

4. Test it:

   python examples/my_format/task.py examples/my_format/input.json
   # Creates: examples/my_format/input.eval
   
   inspect-convert-validate examples/my_format/input.eval

5. Update documentation:

   • Add to examples/README.md if it demonstrates a unique pattern
   • Document in docs/CONVERSION_GUIDE.md if it adds insights

Code Style

• Use black for formatting (line length 100)
• Use ruff for linting
• Use type hints
• Add docstrings to public functions
• Use structlog for logging

Testing

Before submitting:

# Run all tests
make test

# Check linting
ruff check src/

# Format code
black src/ examples/

Documentation

• Keep examples clear and educational
• Update relevant docs when adding features
• Use the same style as existing docs

Commit Messages

Use clear, descriptive commit messages:

• Add example: CSV format converter
• Fix: Handle missing tool_call_id in tool messages
• Update docs: Add troubleshooting for score extraction

Thank you for contributing!