|
1 | 1 | # Evaluation Function Toolkit for Python |
| 2 | + |
| 3 | +A Python toolkit for building and serving evaluation functions. Supports multiple transport mechanisms (stdio, IPC, file) and provides result types for evaluation, chat, and preview use cases. |
| 4 | + |
| 5 | +## Installation |
| 6 | + |
| 7 | +```bash |
| 8 | +pip install lf_toolkit |
| 9 | +``` |
| 10 | + |
| 11 | +Install with optional extras: |
| 12 | + |
| 13 | +```bash |
| 14 | +# Set parsing (antlr4, lark, latex2sympy) |
| 15 | +pip install "lf_toolkit[parsing]" |
| 16 | + |
| 17 | +# IPC support on Windows (named pipes) |
| 18 | +pip install "lf_toolkit[ipc]" |
| 19 | + |
| 20 | +# HTTP server support |
| 21 | +pip install "lf_toolkit[http]" |
| 22 | +``` |
| 23 | + |
| 24 | +## Quick Start |
| 25 | + |
| 26 | +```python |
| 27 | +from lf_toolkit import create_server, run |
| 28 | +from lf_toolkit.evaluation import Result |
| 29 | +from lf_toolkit.shared import Params |
| 30 | + |
| 31 | +server = create_server() |
| 32 | + |
| 33 | +@server.eval |
| 34 | +def evaluate(response, answer, params: Params) -> Result: |
| 35 | + is_correct = response.strip() == answer.strip() |
| 36 | + result = Result(is_correct=is_correct) |
| 37 | + if not is_correct: |
| 38 | + result.add_feedback("hint", "Check your answer again.") |
| 39 | + return result |
| 40 | + |
| 41 | +run(server) |
| 42 | +``` |
| 43 | + |
| 44 | +## Servers |
| 45 | + |
| 46 | +The toolkit provides three server types: |
| 47 | + |
| 48 | +| Class | Transport | Use case | |
| 49 | +|---|---|---| |
| 50 | +| `StdioServer` | stdin/stdout | Default; subprocess communication | |
| 51 | +| `IPCServer` | Unix socket / named pipe | Local IPC | |
| 52 | +| `FileServer` | Files on disk | File-based request/response | |
| 53 | + |
| 54 | +### Manual instantiation |
| 55 | + |
| 56 | +```python |
| 57 | +from lf_toolkit import StdioServer, IPCServer, FileServer |
| 58 | + |
| 59 | +server = StdioServer() |
| 60 | +server = IPCServer(endpoint="/tmp/eval.sock") |
| 61 | +server = FileServer(request_file_path="request.json", response_file_path="response.json") |
| 62 | +``` |
| 63 | + |
| 64 | +### `create_server()` with environment variables |
| 65 | + |
| 66 | +`create_server()` selects the server type from environment variables: |
| 67 | + |
| 68 | +| Variable | Values | Default | |
| 69 | +|---|---|---| |
| 70 | +| `EVAL_IO` | `rpc`, `file` | `rpc` | |
| 71 | +| `EVAL_RPC_TRANSPORT` | `stdio`, `ipc` | `stdio` | |
| 72 | +| `EVAL_RPC_IPC_ENDPOINT` | socket/pipe path | — | |
| 73 | +| `EVAL_FILE_NAME_REQUEST` | file path | — | |
| 74 | +| `EVAL_FILE_NAME_RESPONSE` | file path | — | |
| 75 | + |
| 76 | +## Handlers |
| 77 | + |
| 78 | +Register handlers using decorators on the server instance: |
| 79 | + |
| 80 | +```python |
| 81 | +@server.eval |
| 82 | +def evaluate(response, answer, params: Params) -> Result: |
| 83 | + ... |
| 84 | + |
| 85 | +@server.preview |
| 86 | +def preview(response, params: Params): |
| 87 | + ... |
| 88 | + |
| 89 | +@server.health |
| 90 | +def healthcheck(): |
| 91 | + ... |
| 92 | +``` |
| 93 | + |
| 94 | +Handlers can be `async`: |
| 95 | + |
| 96 | +```python |
| 97 | +@server.eval |
| 98 | +async def evaluate(response, answer, params: Params) -> Result: |
| 99 | + ... |
| 100 | +``` |
| 101 | + |
| 102 | +## Result Types |
| 103 | + |
| 104 | +### `evaluation.Result` |
| 105 | + |
| 106 | +```python |
| 107 | +from lf_toolkit.evaluation import Result |
| 108 | + |
| 109 | +result = Result( |
| 110 | + is_correct=True, |
| 111 | + latex=r"\frac{1}{2}", |
| 112 | + simplified="1/2", |
| 113 | +) |
| 114 | + |
| 115 | +result.add_feedback("hint", "Well done!") |
| 116 | + |
| 117 | +result.to_dict() |
| 118 | +# {"is_correct": True, "feedback": "Well done!", "response_latex": "...", ...} |
| 119 | +``` |
| 120 | + |
| 121 | +### `chat.ChatResult` |
| 122 | + |
| 123 | +```python |
| 124 | +from lf_toolkit.chat import ChatResult |
| 125 | + |
| 126 | +result = ChatResult() |
| 127 | +result.add_response("main", "Here is the explanation...") |
| 128 | +result.add_metadata("model", "gpt-4") |
| 129 | +result.add_processing_time(1.23) |
| 130 | + |
| 131 | +result.to_dict() |
| 132 | +# {"chatbot_response": "Here is the explanation..."} |
| 133 | +``` |
| 134 | + |
| 135 | +### `chat.ChatParams` |
| 136 | + |
| 137 | +```python |
| 138 | +from lf_toolkit.chat import ChatParams |
| 139 | + |
| 140 | +params: ChatParams = { |
| 141 | + "include_test_data": False, |
| 142 | + "conversation_history": ["Hello", "Hi there"], |
| 143 | + "summary": "Previous summary", |
| 144 | + "conversational_style": "formal", |
| 145 | + "question_response_details": "...", |
| 146 | + "conversation_id": "abc-123", |
| 147 | +} |
| 148 | +``` |
| 149 | + |
| 150 | +### `chat` API types |
| 151 | + |
| 152 | +`lf_toolkit.chat` also exports auto-generated types for the MuEd API: |
| 153 | + |
| 154 | +```python |
| 155 | +from lf_toolkit.chat import ChatRequest, ChatResponse, Message |
| 156 | +``` |
| 157 | + |
| 158 | +## Image Upload |
| 159 | + |
| 160 | +Upload PIL images to S3 using AWS SigV4 authentication: |
| 161 | + |
| 162 | +```python |
| 163 | +from PIL import Image |
| 164 | +from lf_toolkit.evaluation.image_upload import upload_image |
| 165 | + |
| 166 | +img = Image.open("diagram.png") |
| 167 | +url = upload_image(img, folder_name="my-eval-function") |
| 168 | +``` |
| 169 | + |
| 170 | +Required environment variables: |
| 171 | + |
| 172 | +| Variable | Description | |
| 173 | +|---|---| |
| 174 | +| `S3_BUCKET_URI` | Base S3 bucket URI | |
| 175 | +| `AWS_ACCESS_KEY_ID` | AWS access key | |
| 176 | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | |
| 177 | +| `AWS_SESSION_TOKEN` | (optional) Session token | |
| 178 | +| `AWS_REGION` | AWS region (default: `eu-west-2`) | |
| 179 | + |
| 180 | +## Set Notation Parser |
| 181 | + |
| 182 | +Parse and evaluate set expressions (requires `parsing` extra): |
| 183 | + |
| 184 | +```python |
| 185 | +from lf_toolkit.parse.set import parse, evaluate |
| 186 | +``` |
| 187 | + |
| 188 | +## Development |
| 189 | + |
| 190 | +```bash |
| 191 | +# Install dependencies |
| 192 | +poetry install |
| 193 | + |
| 194 | +# Run tests |
| 195 | +pytest |
| 196 | + |
| 197 | +# Lint |
| 198 | +make lint |
| 199 | +``` |
0 commit comments