New feature 25 "Deterministic Finality and Ride V9" added.#1834
Open
alexeykiselev wants to merge 58 commits intomasterfrom
Open
New feature 25 "Deterministic Finality and Ride V9" added.#1834alexeykiselev wants to merge 58 commits intomasterfrom
alexeykiselev wants to merge 58 commits intomasterfrom
Conversation
alexeykiselev
added
wip
* Added bls signature methods * Added comments * Enforced no duplicates in signatures and public keys * Fixed linter issues * Added pop method * Added public key validation * Bls aggregated sig refactoring (#1838) * BLS package refactoring. Package renamed from blssig to bls. Crypto primitives SecretKey, PublicKey and Signature were added. Public functions Sing and Verify reimplemented to use new primitives. Function to create aggregated signature from multiple Waves secrets keys was removed because it was useful only in tests. PoP functions moved to separate file. * Added test on keys, signature and messages collected from Scala. * Added tests on PoP functions. Fixed review issues. * Fixed linter issues. * Function to create BLS secret key from a Waves secret key moved to bls_test package. Function MustSignatureFromBytes removed. --------- Co-authored-by: Alexey Kiselev <alexey.kiselev@gmail.com>
* Added block finality schemas * Added protobuf schemas * Updated protobuf generated files * Gosec option to exclued generated files added to security workflow. * Set protobuf-schemas submodule to track the branch. Submodule updated to the latest commit. * Generated protobuf code updated to the latest schema. * Protobuf schemas updated and code regenerated. * Tidy go modules. --------- Co-authored-by: Alexey Kiselev <alexey.kiselev@gmail.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
* Ride version 9 added. New ride function fillList added and tested. * RideV9 functions replaceFirst and replaceAll implemented and tested. * New RideV9 functions for bytes/string conversions with reduced complexity implemented and tested. Old conversion functions refactored to use proper input and output limits. RideV9 functions replaceFirst and replaceAll correct behavior on empty old string implemented. Test naming changed to use fmt.Sprintf to support GoLand interface. * Removed support for 'base16:' prefix for Ride byte conversion functions. Tests modified accordingly. * Added and tested check that Ride V9 scripts is not allowed before activation of DeterministicFinality feature. * Meaningless comment removed. --------- Co-authored-by: Nikolay Eskov <mr.eskov1@yandex.ru>
Proto code regenerated.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
* Fixed two errors found by Semgrep. Protobuf generated files *.pb.go added to semgrep exclude file. * Improve .semgrepignore file.
* Add 'GenerationCommitment' atomic snapsot unmarshaling to 'TxSnapshotsFromProtobufWithoutTxStatus'. * Add tests.
* Ride description of new transaction added. Code regenerated. * Fixed JSON formatting. Added code-generation README file. * Generated Ride objects marked for betteralign check. Generated structures aligned. * Implemented 'commitToGenerationToObject' function. * Add 'ConstantsV9' to the ride generator tool * Support 'RideV9' in selector functions. * Add 'TestRideCommitToGenerationTransactionConstruction'. * Fixed issues made by @copilot. --------- Co-authored-by: Nikolay Eskov <mr.eskov1@yandex.ru>
pkg/state/snapshot_generator.go
Outdated
Check failure
Code scanning / gosec
integer overflow conversion uint64 -> uint32 Error
pkg/state/snapshot_generator.go
Outdated
Check failure
Code scanning / gosec
integer overflow conversion uint64 -> uint32 Error
* Backport missing LibV9 switch branches from PR #1957. Some compiler and evaluator tests updated accordingly. * Exhaustive linter added to the golangci.yml. Linter issues fixed. Added private constant maxVersion to Ride versions iota in order to automate CurrentMaxLibraryVersion function. Added test TestRideVersionCoverage to detect introduction of new library version. Added other tests that checks what Ride scripts can be compiled, estimated and evaluated with all possible versions of Ride. * Changed DApp tests to start from Ride V3.
Default salt set to predefined string as in Scala. Option to turn off seed pre-hash added to be Scala compatible. Tests on compatible key generation added. Other tests updated upon salt modification.
* Add 'secp256r1verify' with test. * Change funcname and signature of 'Secp256Verify', test updated. * tmp * Add Ride V9 with new 'p256Verify' function. * Implemented 'secp256verify' function. * Add tests for 'p256Verify' ride function. * Fix comment. * Benchmark on Secp256Verify added, bytes sames as in Scala. * Update submodule to the correct commit. * Functions to load and verify certificates chain added. (#1980) * Functions to load and verify certificates chain added. Function to verify signature using PK from certificate added. Tests added. * TDX extension check added to LoadCertificates function. Function to extract ECDSA public key from a certificate added. Function SecP256Verify reimplemented to accept message, public key and signature. Tests updated and moved to crypto_test package. Compiler functions descriptions updated, new Ride function to load certificates added. PEG code regenerated. New Ride function implemented. Ride function secP256Verify reimplemented to accept new set of arguments. Ride functions descriptions and costs updated. Code regenerated. Tests on Ride functions updated and added. * Fix modernize issue back. * Add file close checks in tests. --------- Co-authored-by: Nikolay Eskov <mr.eskov1@yandex.ru> --------- Co-authored-by: Alexey Kiselev <alexey.kiselev@gmail.com>
* Go version updated to 1.25. * Automatic fixes from modernize applied. * Add results filtering for modernize to avoid errors on generated code and mocks. * Fix modernize action. * Add go:downloading messages to the filter. * Remove pipefail from modernize run. * Ensure running in bash even on Windows runner. * Explicitly turn off pipefail as github task set it on.
| "encoding/base64" | ||
| stderrs "errors" | ||
| "fmt" | ||
| "math/rand/v2" |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: go.lang.security.audit.crypto.math_random.math-random-used Warning test
* Added bls signature methods * Added comments * Enforced no duplicates in signatures and public keys * Fixed linter issues * Added pop method * Added public key validation * Added block finality schemas * Added protobuf schemas * Updated protobuf generated files * Gosec option to exclued generated files added to security workflow. * Set protobuf-schemas submodule to track the branch. Submodule updated to the latest commit. * Generated protobuf code updated to the latest schema. * WIP: Basic structure of CommitToGeneration transaction implemented. * BLS package refactoring. Package renamed from blssig to bls. Crypto primitives SecretKey, PublicKey and Signature were added. Public functions Sing and Verify reimplemented to use new primitives. Function to create aggregated signature from multiple Waves secrets keys was removed because it was useful only in tests. PoP functions moved to separate file. * Added test on keys, signature and messages collected from Scala. * Added tests on PoP functions. Fixed review issues. * Fixed linter issues. * Protobuf schemas updated and code regenerated. * Tidy go modules. * Some transactions fields renamed according to Protobuf schema. BLS package used to validate PoP during transaction validation. * Protobuf conversion for CommitToGeneration transaction implemented. Test on Protobuf round-trip added. * Introduced constants for Protobuf versions of transactions Reduced cognitive complexity of the new test. * Added test on validation of CommitToGeneration transaction. Added test on JSON serialization of new transaction. WIP: added skipped test on Scala compatibility of JSON serialization. * WIP: Started implementation of commitment transaction conversion into Ride object. Refactored Encode2CBigInt function. * Returned usage of a constant. * Finished endorsement pool * Added endorsement handling to ng * Finalized when key block arrives * Fixed linter issues * WIP: Generation Period length added to functionality settings. BLS keys added to test global variables. Minimal fee amount added for CommitToGeneration transaction. Constants introduced for other fees. Function to calculate next generation period start implemented and tested. Basic checks of CommitToGeneration transaction against state implemented and tested. * WIP: Commitments storage added to state. Basic functions implemented. Commitments serialization implemented and tested. Checks of CommitToGeneration transaction against storage of commitments added to transaction checker. Tests on new checks implemented. New setting MaxGenerators added to networks configurations. StageNet settings updated. * Modernize issues fixed. * Fixed pr issues * Changed the way of calculation of generation period start. Tests updated and added. * Returned a pointer * Change CommitToGeneration transaction number to 19. * TransactionType stringer fixed. Test on JSON serialization fixed. * Review issues fixed. Unused fields and arguments removed. Data emptiness by length check added. Error messages improved and tests updated. Compile time interface check added. * Fixed for tests * Added dummy implementation of some methods * WIP: Transaction differ for CommitToGeneration transaction implementation started. New snapshot type GenerationCommitmentSnapshot added. Snapshot hashing for new snapshot type implemented. * Added endorse block to actions * Fixed the activation check * Functions newestExists and newestSize added to commitments storage. Test on CommitToGeneration transaction performer added. * WIP: CBOR balances serialization implemented. * Balances calculation fixed. Linter issues fixed. * Added conversion of LeaseIn and LeaseOut to int64 with overflow. Warning on such conversions added. * Added conflict endorsements, fixed endorser commitment records, sorted endorsers records, added actions messages * Added an endorsement message test * Fixed 'more or less' in voting finalization * Benchmark on wavesBalanceRecord serialization/deserialization added. * Deprecated functions replaced. * Updated go-safecast package to v2. * Fixed errors check in commitments storage. * Fixed PR issues * Fixed linter issues * Replaced array to heap * Updated microblock * Updateed protobuf schemas * Regenerated proto structures * Added a test * Fixed empty endorsers when finalizing for microblock * Updated bls signature type in finalization and endorsements * Updated protobuf schemas * regenerated proto structures * Fixed serializing * Added a feature check before adding fin to microblock * Reset Deposits upon generation period end. (#1882) * WIP: Reset Deposits upon generation period end. Functions to calculate generation periods start and end added and modified according to Scala implementation. Tests updated and added. Function to detect generation period end added. Function to reset deposits added. * WIP: Integration test on CommitToGenerationTransaction and deposits resets added. Interagation tests NodeUniversalClient improved. New BlockchainOptions added. BLS keys generation and storing added to itests AccountInfo. BLS Sigrnature JSON deserialization fixed. CommitToGenerationTransaction type added to GuessTransactionType function. MaxGenerators field of FunctionalitySettings renamed to MaxEndorsements. Support for new transaction added to txAppender. Deposit application added to snapshot application of CommitToGeneration transaction. Check on commitments limit for generation period removed. Compliment test removed. Linter fixes. * Excessive logging removed. * Test on deposit rollback added. Test made independent of starting height. * Review issues fixed. Few TODOs resolved. Size functions of commitments storage removed. Safe addition of deposit added. Generation period end function used in state to check if generation period is over. * Key generation options added for BLS secret key generation. (#1910) Options to set custom salt or random salt added. Option to set key info added. Tests on key generation added. * Check on repeated usage of endorser public key from another waves account added. Duplicated code extracted in a function. Test added. * Fixed a mistake * Added a check whether the endorsement pool is empy * Update legacy state hash (#1901) * WIP: Reset Deposits upon generation period end. Functions to calculate generation periods start and end added and modified according to Scala implementation. Tests updated and added. Function to detect generation period end added. Function to reset deposits added. * WIP: Integration test on CommitToGenerationTransaction and deposits resets added. Interagation tests NodeUniversalClient improved. New BlockchainOptions added. BLS keys generation and storing added to itests AccountInfo. BLS Sigrnature JSON deserialization fixed. CommitToGenerationTransaction type added to GuessTransactionType function. MaxGenerators field of FunctionalitySettings renamed to MaxEndorsements. Support for new transaction added to txAppender. Deposit application added to snapshot application of CommitToGeneration transaction. Check on commitments limit for generation period removed. Compliment test removed. Linter fixes. * Excessive logging removed. * Test on deposit rollback added. Test made independent of starting height. * WIP: Legacy state hash structures moved to separate file. Refactoring of structures in progress. * StateHash structure renamed to StateHashV1. Second version of StateHash with additional field implemented as StateHashV2. Tests on state hash moved to separate file. Total hash generation reimplemented with WriteTo function. Binary serialization of StateHashV1 reimplemented with ReadFrom and WriteTo functions. Wrapper SizedBlockID added to support serialization/deserialization of BlockID prepended with length. * StateHashDebug interface added. Both implementations updated accordingly. Debug API and statehash utility updated to produce and use new interface. BaseTarget reporting added to StateHashDebugV2. * Required getters added to StateHash interface and implementations. HTTP client debug updated to use proto.StateHash interface. Itest HTTP client updated. Utility statecmp updated and refactored a bit. * StateHash version selection upon finality activation implemented. WIP: Broken test added. * Legacy state hash Scala compatibility test implemented. * Linter issue fixed. * Test fixed. * Review issues fixed. UnmarshalBinary added to StateHash interface. Constructors to create appropriate state hashes added. Test added. Save of state hash fixed. * Removed finalization on key block * Fixed test * Added finalization storage * Fixed a panic * Removed extra fields go mod * Added punishment for conflict endorsements * Pulled latest changes from protobuf-schemas * Fixed a merging error * Regenerated proto files * Updated protobuf gen files version * Added extra lines at the end of files * Added one more line at the end of file * Fixed timing of removing bad generator * Added finalization debug logs * Removed a line * Added block endorsement from miner when received key block * Changed block to parent block to endorse * Moved sending endorsement from mining key block to receiving key block * Added cleaning endorsement pool upon receiving key block * Fixed a mistake with building message for signature * Fixed a mistake with comparing endorsement messages * Fixed FinalizedBlockHeight field * Added a check whether addresses are in the commitment records before endorsing * Added more meaningful errors * Added the number of args of fsm event validation * Added a comma * Added a check on max_endorsements * Removed useless code * Fixed logging of new finalization voting * Fixed logging of new finalization voting * Fixed retrieving finalized blockIDs * Fixed a log * Fixed underflow bug with calculating finalized height * Fixed a formatting issue * Added logs when key not found in commitment records * Added endorsementIDs cache * Added logs with endorsers * Fixed finalized height calculation * Fixed linter issues * Added a test for commitments * Fixed the block signature bug where finalization was added after block was signed * Added block generator balance to the total endorsers balance * Added a clearer log message to get block action * Added saving of block generator to endorsement pool * Added block generator to endorsement pool upon receiving new block * Fixed a bug with microblock protobuf not having finalization voting * Added finalization voting for microblock in other places * Added more meaningful logs and errors * Microblock finalization is nil if block finalization is nil * Added more logs * removed useless calculation * removed merging lines * Added more logs * Added missing finalization voting in missing conversion * Removed fixed size from finalizatiom history record * Removed fixed size from finalizatiom history record * Added more logs * Fixed finalization signature check over grandparent's block * Fixed a linter error * Checking generating balance on previous height instead of latest * Checking generating balance on previous height instead of latest * Stored the correct endorsed block height instead of the current one when finalized * Split big function into 2 * Committed generators balances legacy state hash (#1978) * Add legacy state hash methods for 'commitments'. * Refactoring of 'NewLegacyStateHash' constructor. * Add committed generators balances hash calculation. * Add new field to statehash calculation, tests updated. * Outdated comments updated. Uninformative comments removed. --------- Co-authored-by: Alexey Kiselev <alexey.kiselev@gmail.com> * Wrapped an error * Added limit to auto rollback according to the finality spec * Added limit to auto rollback according to the finality spec * Node api finality (#1897) * Drafted endpoints for API * Updated protobuf structures * Added finalization storage * Fixed errors * Fixed old code * SignCommitGeneration draft * Finished signCommitToGeneration * Added finalization validation * Added a finalization processor for tx appender * Merged again * Added clients methods * Added http client itests * Fixed an error * Updated protobuf version * Implemented nickeskov's suggestions * Implemented some AI suggestions * Made the tx signing handle all tx types * Fixed an fmt error * Added a test for transactionSign * Fixed linter erors * Added a specific error * Added logs * Fixed a linter error * Changed commitment transaction fee * Allowed for custom transaction fee in commitment tx sign api * Fix modernize. * Changed fmt to errors wrap --------- Co-authored-by: Nikolay Eskov <mr.eskov1@yandex.ru> * Linter issues fixed. * Auto rollback finalization 2 (#2005) * Added a soft rollback * Added commit after saving finalization after rollback * Added a check whether finalization block exists * Changed newest block id by height * Set currentBlockID when finalization is set * Fixed PR issues * Fixed a typo * Modernize issue fixed. --------- Co-authored-by: esuwu <sanya554455@gmail.com> --------- Co-authored-by: Alexey Kiselev <alexey.kiselev@gmail.com> Co-authored-by: Nikolay Eskov <mr.eskov1@yandex.ru>
Comment on lines
+319
to
+321
| if endorseErr := a.Endorse(block.Parent, height, pk, sk); endorseErr != nil { | ||
| return a.Errorf(errors.Wrapf(err, "failed to endorse parent block")) | ||
| } |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: semgrep.rules.if-incorrect-nil-err-return Warning
Comment on lines
+319
to
+321
| if endorseErr := a.Endorse(block.Parent, height, pk, sk); endorseErr != nil { | ||
| return a.Errorf(errors.Wrapf(err, "failed to endorse parent block")) | ||
| } |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: semgrep.rules.if-inplace-func-incorrect-nil-err-return Warning
Comment on lines
+388
to
+390
| if findErr != nil { | ||
| return a, nil, a.Errorf(errors.Wrapf(err, "failed to find waves generator PK by BLS endorser PK")) | ||
| } |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: semgrep.rules.if-incorrect-nil-err-return Warning
Comment on lines
+1662
to
+1675
| if verifyErr := f.verifyFinalizationSignature(finalizationVoting, msg, endorsersPK); verifyErr != nil { | ||
| sb := new(strings.Builder) | ||
| for _, epk := range endorsersPK { | ||
| sb.WriteString(epk.String()) | ||
| sb.WriteString(",") | ||
| } | ||
| slog.Debug("failed to verify finalization signature", | ||
| "signature", finalizationVoting.AggregatedEndorsementSignature.String(), | ||
| "msg", msg, | ||
| "endorsersPKs", sb.String(), | ||
| "err", verifyErr.Error(), | ||
| ) | ||
| return errors.Wrapf(err, "failed to verify finalization signature") | ||
| } |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: semgrep.rules.if-incorrect-nil-err-return Warning
Comment on lines
+1662
to
+1675
| if verifyErr := f.verifyFinalizationSignature(finalizationVoting, msg, endorsersPK); verifyErr != nil { | ||
| sb := new(strings.Builder) | ||
| for _, epk := range endorsersPK { | ||
| sb.WriteString(epk.String()) | ||
| sb.WriteString(",") | ||
| } | ||
| slog.Debug("failed to verify finalization signature", | ||
| "signature", finalizationVoting.AggregatedEndorsementSignature.String(), | ||
| "msg", msg, | ||
| "endorsersPKs", sb.String(), | ||
| "err", verifyErr.Error(), | ||
| ) | ||
| return errors.Wrapf(err, "failed to verify finalization signature") | ||
| } |
Check warning
Code scanning / Semgrep OSS
Semgrep Finding: semgrep.rules.if-inplace-func-incorrect-nil-err-return Warning
Fixed deprecation of low-level ecdsa PublicKey functions by using ParseUncompressedPublicKey function. Fixed linter issues.
alexeykiselev
requested a deployment
to
Deploy-mainnet-amd64
GitHub Actions
Waiting
alexeykiselev
requested a deployment
to
Deploy-stagenet-amd64
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.