Skip to content

Commit 312746e

Browse files
committed
Expanded README.md with detailed usage instructions and examples for installation, server types, handlers, result types, image uploads, and development setup.
1 parent 367aeaa commit 312746e

1 file changed

Lines changed: 198 additions & 0 deletions

File tree

README.md

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,199 @@
11
# Evaluation Function Toolkit for Python
2+
3+
A Python toolkit for building and serving evaluation functions. Supports multiple transport mechanisms (stdio, IPC, file) and provides result types for evaluation, chat, and preview use cases.
4+
5+
## Installation
6+
7+
```bash
8+
pip install lf_toolkit
9+
```
10+
11+
Install with optional extras:
12+
13+
```bash
14+
# Set parsing (antlr4, lark, latex2sympy)
15+
pip install "lf_toolkit[parsing]"
16+
17+
# IPC support on Windows (named pipes)
18+
pip install "lf_toolkit[ipc]"
19+
20+
# HTTP server support
21+
pip install "lf_toolkit[http]"
22+
```
23+
24+
## Quick Start
25+
26+
```python
27+
from lf_toolkit import create_server, run
28+
from lf_toolkit.evaluation import Result
29+
from lf_toolkit.shared import Params
30+
31+
server = create_server()
32+
33+
@server.eval
34+
def evaluate(response, answer, params: Params) -> Result:
35+
is_correct = response.strip() == answer.strip()
36+
result = Result(is_correct=is_correct)
37+
if not is_correct:
38+
result.add_feedback("hint", "Check your answer again.")
39+
return result
40+
41+
run(server)
42+
```
43+
44+
## Servers
45+
46+
The toolkit provides three server types:
47+
48+
| Class | Transport | Use case |
49+
|---|---|---|
50+
| `StdioServer` | stdin/stdout | Default; subprocess communication |
51+
| `IPCServer` | Unix socket / named pipe | Local IPC |
52+
| `FileServer` | Files on disk | File-based request/response |
53+
54+
### Manual instantiation
55+
56+
```python
57+
from lf_toolkit import StdioServer, IPCServer, FileServer
58+
59+
server = StdioServer()
60+
server = IPCServer(endpoint="/tmp/eval.sock")
61+
server = FileServer(request_file_path="request.json", response_file_path="response.json")
62+
```
63+
64+
### `create_server()` with environment variables
65+
66+
`create_server()` selects the server type from environment variables:
67+
68+
| Variable | Values | Default |
69+
|---|---|---|
70+
| `EVAL_IO` | `rpc`, `file` | `rpc` |
71+
| `EVAL_RPC_TRANSPORT` | `stdio`, `ipc` | `stdio` |
72+
| `EVAL_RPC_IPC_ENDPOINT` | socket/pipe path ||
73+
| `EVAL_FILE_NAME_REQUEST` | file path ||
74+
| `EVAL_FILE_NAME_RESPONSE` | file path ||
75+
76+
## Handlers
77+
78+
Register handlers using decorators on the server instance:
79+
80+
```python
81+
@server.eval
82+
def evaluate(response, answer, params: Params) -> Result:
83+
...
84+
85+
@server.preview
86+
def preview(response, params: Params):
87+
...
88+
89+
@server.health
90+
def healthcheck():
91+
...
92+
```
93+
94+
Handlers can be `async`:
95+
96+
```python
97+
@server.eval
98+
async def evaluate(response, answer, params: Params) -> Result:
99+
...
100+
```
101+
102+
## Result Types
103+
104+
### `evaluation.Result`
105+
106+
```python
107+
from lf_toolkit.evaluation import Result
108+
109+
result = Result(
110+
is_correct=True,
111+
latex=r"\frac{1}{2}",
112+
simplified="1/2",
113+
)
114+
115+
result.add_feedback("hint", "Well done!")
116+
117+
result.to_dict()
118+
# {"is_correct": True, "feedback": "Well done!", "response_latex": "...", ...}
119+
```
120+
121+
### `chat.ChatResult`
122+
123+
```python
124+
from lf_toolkit.chat import ChatResult
125+
126+
result = ChatResult()
127+
result.add_response("main", "Here is the explanation...")
128+
result.add_metadata("model", "gpt-4")
129+
result.add_processing_time(1.23)
130+
131+
result.to_dict()
132+
# {"chatbot_response": "Here is the explanation..."}
133+
```
134+
135+
### `chat.ChatParams`
136+
137+
```python
138+
from lf_toolkit.chat import ChatParams
139+
140+
params: ChatParams = {
141+
"include_test_data": False,
142+
"conversation_history": ["Hello", "Hi there"],
143+
"summary": "Previous summary",
144+
"conversational_style": "formal",
145+
"question_response_details": "...",
146+
"conversation_id": "abc-123",
147+
}
148+
```
149+
150+
### `chat` API types
151+
152+
`lf_toolkit.chat` also exports auto-generated types for the MuEd API:
153+
154+
```python
155+
from lf_toolkit.chat import ChatRequest, ChatResponse, Message
156+
```
157+
158+
## Image Upload
159+
160+
Upload PIL images to S3 using AWS SigV4 authentication:
161+
162+
```python
163+
from PIL import Image
164+
from lf_toolkit.evaluation.image_upload import upload_image
165+
166+
img = Image.open("diagram.png")
167+
url = upload_image(img, folder_name="my-eval-function")
168+
```
169+
170+
Required environment variables:
171+
172+
| Variable | Description |
173+
|---|---|
174+
| `S3_BUCKET_URI` | Base S3 bucket URI |
175+
| `AWS_ACCESS_KEY_ID` | AWS access key |
176+
| `AWS_SECRET_ACCESS_KEY` | AWS secret key |
177+
| `AWS_SESSION_TOKEN` | (optional) Session token |
178+
| `AWS_REGION` | AWS region (default: `eu-west-2`) |
179+
180+
## Set Notation Parser
181+
182+
Parse and evaluate set expressions (requires `parsing` extra):
183+
184+
```python
185+
from lf_toolkit.parse.set import parse, evaluate
186+
```
187+
188+
## Development
189+
190+
```bash
191+
# Install dependencies
192+
poetry install
193+
194+
# Run tests
195+
pytest
196+
197+
# Lint
198+
make lint
199+
```

0 commit comments

Comments
 (0)