Sync from upstream 2026-06-08#51
Open
forge33 wants to merge 8 commits into
Open
Conversation
…github#1684) * Fix resume data loss: route heartbeat coords through applyEventsQueue onChangelogHeartbeatEvent was mutating applier.CurrentCoordinates directly from the streamer goroutine, before any DML that preceded the heartbeat was applied to the ghost table. The checkpoint loop reads CurrentCoordinates as "applied through this GTID" and could persist a checkpoint whose LastTrxCoords was ahead of what was actually applied. If gh-ost crashed before applyEventsQueue drained, --resume read that checkpoint and called StartSyncGTID with the persisted set; MySQL treated the un-applied GTIDs as already-seen and never re-streamed them. The ghost table silently lost those DMLs and cut-over produced a stale table. Fix: enqueue a tableWriteFunc onto applyEventsQueue that performs the coords bump. The apply goroutine executes it in order, after the DMLs the streamer enqueued before the heartbeat, restoring the invariant. Adds TestMigratorHeartbeatDoesNotAdvancePastUnappliedDML, which fails at the previous HEAD and passes after the fix; also asserts queue ordering to guard against future changes that wrap the heartbeat enqueue in a goroutine. Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com> * Replace direct channel write with SendWithContext Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add Datadog/statsd with simple client emitting startup * Add go runtime metrics to statsd reporting --------- Co-authored-by: meiji163 <meiji163@github.com>
Refactor newTestMigrationContext to set ServeSocketFile via os.TempDir() and remove runtime.Caller-based path derivation.
* Disable CGO for release builds * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…60608 # Conflicts: # go/base/context.go # go/metrics/client.go # go/sql/builder.go
forge33
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Sync
masterfrom upstreamgithub/gh-ostmaster.Upstream commits brought in:
83aaa708Disable CGO for release builds (Disable CGO for release builds github/gh-ost#1697)835f5379Improve performance for tables with composite primary keys (Improve performance for tables with composite primary keys github/gh-ost#1686)251b08daUse os.TempDir() for test socket path (Useos.TempDir()for test socket path github/gh-ost#1694)c6363475Add GET_LOCK to prevent concurrent migrations of the same table (Add GET_LOCK to prevent concurrent migrations of the same table github/gh-ost#1693)59f400f5Add Go runtime metrics to statsd reporting (Add Go runtime metrics to statsd reporting github/gh-ost#1690) — our upstreamed PRe59af3a4Add Datadog/statsd with simple client emitting startup (Add Datadog/statsd with simple client emitting startup github/gh-ost#1689) — our upstreamed PR72cf2297Fix resume data loss: route heartbeat coords through applyEventsQueue (Fix resume data loss: route heartbeat coords through applyEventsQueue github/gh-ost#1684)Conflict resolutions
go/base/context.go: kept Shopify'sMetrics metrics.Emitterinterface (Shopify is ahead of upstream's*metrics.Client).go/metrics/client.go: kept Shopify's superset —Emitterinterface +Histogrammethod.go/sql/builder.go: accepted upstream's additional defensive guards (additive, complementary to Shopify'sedd6e2b8guards).go/metrics/go_runtime.go+go_runtime_test.go: removed — upstream added these as separate files (from cherry-picking our PR Add Go runtime metrics to statsd reporting github/gh-ost#1690), but Shopify has since consolidated their contents intoemit.gousing the more specificMemStatsGaugeEmitterinterface.Verification
go build ./...— successgo test ./...— 212 passed in 7 packages