Skip to content

Add Git4Data Part 8: ML Continuous Learning — Train Only What Changed (EN + ZH)#55

Open
dengn wants to merge 3 commits into
mainfrom
add-git4data-part8-ml-incremental
Open

Add Git4Data Part 8: ML Continuous Learning — Train Only What Changed (EN + ZH)#55
dengn wants to merge 3 commits into
mainfrom
add-git4data-part8-ml-incremental

Conversation

@dengn

@dengn dengn commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

What

Adds Part 8 — ML Continuous Learning: Train Only What Changed (EN + ZH), opening the AI-training arc.

  • EN: matrixorigin/git4data-part8-ml-incremental/
  • ZH: matrixorigin/git4data-part8-ml-incremental-zh/

The data changes every day — why retrain on everything? Pin the training set with a SNAPSHOT; next round, one DATA BRANCH DIFF gives the exact changed rows, so you train only the delta.

Content (detailed, scenario-driven, strong comparison)

  • When you do continuous learning — risk/fraud, recommendation/CTR, moderation, any growing labeled set.
  • The three-step loop — pin → train + register → DIFF the delta; a model ↔ data chain that unlocks exact reproduce / attribute / rollback.
  • Cost experiment — 6 rounds: incremental 6,004 rows vs full retrain 21,000; the gap grows quadratically.
  • "How the alternatives do it — and where they break"updated_at watermark, CDC/Debezium, two full copies + EXCEPT, DVC / lakeFS, Delta Lake CDF — each with concrete failure modes and a system-by-system table.

Figures (2 per language)

  • fig_incremental-loop — the pin → train → DIFF-delta cycle.
  • fig_cost-curve — full-retrain (quadratic) vs incremental (linear) over 6 rounds.

Verification

Every statement verified on MatrixOne 4.0.0-rc3: DIFF → INSERTED 3000 / UPDATED 200; net delta (value anti-join) = 3200 vs full table 103000; snapshot reproduces m1's set = 100000; 6-round experiment per-round delta stays 1000 + a couple even as the table grows 6×. pnpm validate369/369.

🤖 Generated with Claude Code

helgarock and others added 3 commits July 3, 2026 10:45
… (EN + ZH)

Opens the AI-training arc. Pin the training set with a SNAPSHOT; next round, one DATA BRANCH DIFF gives the exact changed rows so you train only the delta. Includes when-to-use scenarios, the three-step loop (pin -> train+register -> diff delta) with a model<->data chain enabling reproduce/attribute/rollback, a 6-round cost experiment (incremental 6,004 vs full 21,000 rows; gap grows quadratically), and a point-by-point comparison against updated_at watermarks / CDC / two-copies EXCEPT / DVC / lakeFS / Delta CDF with a system-by-system table. EN + ZH, 2 figures each. All SQL verified end-to-end on MatrixOne 4.0.0-rc3. Comparison peer is MatrixOne (git4data is its capability).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… on knowing what changed; spell out machine learning

Add an explicit section on when incremental training fits vs when a full retrain is genuinely needed (catastrophic forgetting, concept drift, deletes/corrections = machine unlearning, non-incremental tree models, feature/hyperparameter/architecture changes). Reframe the value: snapshot+DIFF gives the precise 'what changed' answer that powers BOTH incremental training (when it fits) and the retrain decision + reproducibility/attribution/rollback (always). Update title/description accordingly. Replace 'ML' with 'machine learning' throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ovided

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants