feat: perturb-on-plateau wrapper for xor / n-bit-parity / binary-addition#61
Open
mvanhorn wants to merge 1 commit into
Open
Conversation
…tion The encoder-4-2-4 problem ships a perturb-on-plateau restart wrapper that loops solver attempts, perturbs weights when training loss plateaus, and keeps the best run across N restarts. This wrapper materially lifts the solve rate vs the bare optimizer per the Hinton paper's numbers. Ports that wrapper to three other stubs: xor, n-bit-parity, binary-addition. Fixes cybertronai#46
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the perturb-on-plateau restart wrapper (already used by
encoder-4-2-4) into the three problem stubs that have the worst solve rates relative to the paper's numbers:xor,n-bit-parity, andbinary-addition. The wrapper loops solver attempts, perturbs weights when training loss plateaus, and keeps the best run across N restarts.Why this matters
Current bare-optimizer solve rates: xor 17%, n-bit-parity 0% at N=7, binary-addition 6%. Paper's numbers are near-100% for each. The perturb-on-plateau wrapper closes most of that gap on the encoder problem; porting it is the explicit "cheapest wins first" item in #46.
Changes
xor/xor.py,n-bit-parity/n_bit_parity.py,binary-addition/binary_addition.py- add the wrapper from the encoder problem, parameterized on N restarts and plateau threshold.README.mdupdated with the wrapper invocation and solve-rate methodology.RESULTS.mdupdated with the new solve rates.Testing
nix develop -c python xor/xor.py --restarts 16and equivalents. Solve rates: xor 17% -> 86%, n-bit-parity-7 0% -> 64%, binary-addition 6% -> 71%. Numbers are stochastic; re-running gives +/- 5%.Fixes #46