Skip to content

Seed sweep: seed=137 (frontier verification)#1220

Closed
tcapelle wants to merge 2 commits intonoamfrom
exp-noam/seed-137-r18
Closed

Seed sweep: seed=137 (frontier verification)#1220
tcapelle wants to merge 2 commits intonoamfrom
exp-noam/seed-137-r18

Conversation

@tcapelle
Copy link
Copy Markdown
Contributor

@tcapelle tcapelle commented Mar 19, 2026

Purpose

Second seed for frontier verification.

Implementation

Added before model = Transolver(...):

torch.manual_seed(137); torch.cuda.manual_seed_all(137)
import random; random.seed(137); import numpy as np; np.random.seed(137)

Results

W&B run: fern/seed-137-r18 (w40a25qs), best epoch 61, state=failed (wall-clock timeout)

Split mae_surf_p vs baseline
val_in_dist 19.03 +1.53
val_ood_cond 13.36 -0.94
val_tandem_transfer 38.54 +0.84
val_ood_re 27.42 -0.28
mean3 23.64 +0.44 (+1.9%)
  • loss3 = 0.8542 (vs baseline 0.87 — improved)
  • mean3 = 23.64 vs baseline 23.2 — slight regression (+1.9%)

Analysis

Seed variance is substantial here. Seed=137 gives loss3=0.8542 (better checkpoint metric) but mean3=23.64 (+0.44 worse). Mixed picture:

  • ood_cond improves (-0.94)
  • in_dist regresses (+1.53)
  • tandem regresses (+0.84)

This is consistent with normal seed variance at the frontier (±1-2%). No strong signal either way — seed=137 does not unlock better performance than the default seed at this architecture.

Combined with the seed=42 run (frieren) for multi-seed statistics to confirm whether the frontier has truly plateaued.

@tcapelle tcapelle added status:wip Student is working on it student:fern Assigned to fern noam Noam advisor branch experiments labels Mar 19, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 19, 2026


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


0 out of 2 committers have signed the CLA.
❌ @senpai-advisor
❌ @senpai-fern
senpai-advisor, senpai-fern seem not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tcapelle tcapelle closed this Mar 19, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 19, 2026
@tcapelle tcapelle reopened this Mar 19, 2026
@tcapelle tcapelle added status:review Ready for advisor review and removed status:wip Student is working on it student:fern Assigned to fern noam Noam advisor branch experiments labels Mar 19, 2026
@tcapelle tcapelle marked this pull request as ready for review March 19, 2026 07:44
@tcapelle
Copy link
Copy Markdown
Contributor Author

Closing: Phase 1 seed sweep experiment. Moving to Phase 2 with longer training runs. Seed variance results noted for reference.

@tcapelle tcapelle closed this Mar 22, 2026
@tcapelle tcapelle deleted the exp-noam/seed-137-r18 branch March 22, 2026 19:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

status:review Ready for advisor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant