REFINE

Rule Enhancement via Feedback and Iterative Neural Editing

A browser-based tool for refining MedTagger symbolic NLP rules using LLM-assisted error analysis.

Overview

REFINE closes the loop between symbolic NLP systems and large language models. It loads your MedTagger rule files, evaluates system output against gold annotations, surfaces false positives and false negatives, and uses a locally hosted LLM to suggest targeted rule edits — all without sending data to external services.

Requirements

• Python 3.8+
• Ollama with a model pulled (e.g. ollama pull llama3)
• Java (for MedTagger)

pip install flask flask-cors

Quickstart

# Start the local LLM
ollama serve

# Start the bridge server
python server.py

# Open in browser
open http://localhost:5050

Project Structure

LLM4Sym/
├── refine.html              # Single-page app (no build step)
├── server.py                # Flask bridge server
├── models/
│   └── <TaskName>/
│       ├── context/
│       │   └── contextRule.txt
│       ├── rules/
│       │   └── resources_rules_matchrules.txt
│       ├── regexp/
│       │   └── resources_regexp_re*.txt
│       └── used_resources.txt
├── input/                   # MedTagger input .txt files
├── output/                  # MedTagger .ann output files
└── *.sh                     # MedTagger run scripts

Workflow

1. Configure — select a task, load rule files and annotation guidelines
2. Evaluate — upload gold XML + system .ann files, or run sentence-level CSV evaluation
3. Analyze — review false positives and false negatives; generate LLM rule suggestions
4. Refine — accept edits and write updated rules back to disk
5. Re-evaluate — run MedTagger again and compare metrics

Adding a New Task

Drop a folder under models/<TaskName>/ following the structure above. REFINE discovers regexp files automatically from used_resources.txt and resolves normalization values from resources_rules_matchrules.txt — no configuration changes needed.

Server API

Endpoint	Method	Description
/api/health	GET	Lists available tasks and scripts
/api/read-rules?task=<name>	GET	Returns all rule file contents
/api/write-rules	POST	Writes updated rule files to disk
/api/run-script	POST	Executes a .sh MedTagger script
/api/read-ann?dir=<dir>	GET	Returns .ann output files
/api/read-gold?dir=<dir>	GET	Returns gold .xml files

Options

python server.py --port 5050 --base /path/to/project