Skip to content

Commit f4117b1

Browse files
JuliaEdomclaude
andcommitted
script: run_selfcon_retry.py — add --gen-model + --sleep-between args
Расширил `scripts/run_selfcon_retry.py` для прогона любой Mistral модели (не только codestral-latest) с настраиваемым sleep между pipeline вызовами. Использование осталось обратно-совместимым (default = codestral-latest, sleep=0). Новые опции: - `--gen-model <id>` (например `mistral-large-latest`, `mistral-medium-latest`) - `--sleep-between <sec>` (rate-limit throttle между pipeline iterations) Контекст: v17-extended попытка прогнать mistral-large-latest на v16 29-residue показала, что Mistral free-tier rate-limit на large model очень жёсткий (HTTP 429 даже с 3s sleep, потому что pipeline делает 2-3 LLM calls/case). Скрипт-arg всё равно полезен для будущих экспериментов с другими Mistral моделями или для paid-tier upgrade. Findings зафиксированы в ~/.claude/projects/D--/memory/reference_mistral_free_tier_limits.md. Smoke tests: 25/25 passed (tests/eval/test_metrics.py + tests/test_db_connection.py). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent dde4d76 commit f4117b1

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

scripts/run_selfcon_retry.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,17 +40,19 @@ def main() -> int:
4040
p.add_argument("--baseline", type=Path, required=True)
4141
p.add_argument("--bird-root", type=Path, default=Path("data/bird_mini_dev/MINIDEV"))
4242
p.add_argument("--temperatures", nargs="+", type=float, default=[0.2, 0.4, 0.6, 0.8])
43+
p.add_argument("--gen-model", default="codestral-latest", help="Mistral model id")
44+
p.add_argument("--sleep-between", type=float, default=0.0, help="seconds between pipeline calls (use for mistral-large rate limits)")
4345
p.add_argument("--out", type=Path, required=True)
4446
args = p.parse_args()
4547

4648
settings = get_settings()
4749
baseline = json.loads(args.baseline.read_text(encoding="utf-8"))
4850
fails = [r for r in baseline["records"] if not r.get("match")]
49-
print(f"[info] {len(fails)} failures, temps={args.temperatures}", file=sys.stderr)
51+
print(f"[info] {len(fails)} failures, temps={args.temperatures}, model={args.gen_model}", file=sys.stderr)
5052

5153
examples = {e.question_id: e for e in load_bird_mini_dev(args.bird_root)}
5254
registry = get_default_registry()
53-
mistral = MistralProvider(api_key=settings.mistral_api_key, gen_model="codestral-latest")
55+
mistral = MistralProvider(api_key=settings.mistral_api_key, gen_model=args.gen_model)
5456
sql_prov = CachingLLMProvider(mistral, cache_dir=settings.llm_cache_dir)
5557
emb = CachingEmbeddingProvider(
5658
MistralProvider(api_key=settings.mistral_api_key), cache_dir=settings.llm_cache_dir
@@ -99,6 +101,8 @@ def main() -> int:
99101
candidates.append(Candidate(result=r, temperature=temp))
100102
except Exception as exc:
101103
print(f"[{i:3d}/{len(fails)}] qid={qid} T={temp} EXC: {exc}", file=sys.stderr)
104+
if args.sleep_between > 0:
105+
time.sleep(args.sleep_between)
102106
if not candidates:
103107
continue
104108

0 commit comments

Comments
 (0)