Feature: Enable Just-In-Time (JIT) Git LFS Conversion #347

Lesvek · 2025-12-12T16:30:54Z

This PR introduces core support and a new plugin (git_lfs_importer) to perform Git LFS conversion concurrently with the history export.

This approach significantly speeds up migration for large Mercurial repositories (100GiB+) by eliminating the need for the second, time-consuming full history rewrite step (git lfs import --everything).

Key Changes:

New Plugin (git_lfs_importer): Identifies matching files and immediately writes LFS pointers during the initial export, enabling efficient incremental conversion.
- Requires Python pathspec dependency
- Seeplugins/git_lfs_importer/README.md for full setup and usage details.
Core Feature: Introduces the --first-commit-hash option to allow exporting history onto a pre-existing Git commit. This is crucial for configuring LFS (.gitattributes) before the import begins.

Usage Example:

# Export onto an existing commit (e.g., one with .gitattributes)
hg-fast-export.sh -r /path/to/hgrepo -M master \
    --plugin "git_lfs_importer=/path/to/lfs-patterns.txt" \
    --first-commit-hash "$FIRST_GIT_HASH"

Lesvek · 2025-12-12T16:33:41Z

plugins/git_lfs_importer/README.md

+
+### Dependencies
+
+This plugin requires the `pathspec` package:


Should I add it to the requirements-*.txt files in the .github folder?

If so, what version should I put?

You're the author and fast-export didn't use pathspec before, so pick a version/versions that are compatible with the earliest/latest versions we support for fast-export.

I have approved the workflow, so we'll soon see if/what breaks if its missing.

I put pathspec==11.2 in requirements earliest as this is the last version that supports Python 3.7.

frej

Solid work with testcases and well-written documentation. I just have some formatting nits as review comments.

Would you consider adding a pointer to the new plugin to the "Mercurial Largefiles Extension" section of the top level README?

frej · 2025-12-20T15:16:42Z

t/first_commit_hash_option.t

+	git config core.ignoreCase false &&
+	git lfs install --local &&
+	git switch --create master &&
+


Empty trailing whitespace on this line.

Should be fixed.

frej · 2025-12-20T15:16:57Z

t/first_commit_hash_option.t

+	cat > .gitattributes <<-EOF &&
+	* -text
+	EOF
+


Empty trailing whitespace on this line.

Should be fixed.

frej · 2025-12-20T15:17:09Z

t/first_commit_hash_option.t

+	git config core.ignoreCase false &&
+	git lfs install --local &&
+	git switch --create master &&
+


Empty trailing whitespace on this line.

Should be fixed.

frej · 2025-12-20T15:17:53Z

t/first_commit_hash_option.t

+	git add readme.txt &&
+	git commit -q -m "Initialize Git readme file"
+	) &&
+


Empty trailing whitespace on this line.

Should be fixed.

frej · 2025-12-20T15:20:45Z

t/first_commit_hash_option.t

+
+	FIRST_HASH=$(git -C gitrepo rev-parse HEAD) &&
+
+	# 3. Run hg-fast-export with git_lfs_importer plugin


I can't see that you're using the plugin, did you copy this file from the plugin test?

Oh, sorry, I did copy it and didn't fix the comment. Will do.

frej · 2025-12-20T15:28:42Z

plugins/git_lfs_importer/README.md

+
+### Large Files and Largefiles
+
+If the Mercurial repository uses Mercurial's largefiles extension, those files are already converted to their original content before reaching this plugin, allowing the plugin to apply LFS conversion if they match the patterns.


> 80 cols, please line break.

frej · 2025-12-20T15:29:10Z

plugins/git_lfs_importer/README.md

+
+### Force Requirement
+
+You only need to pass the `--force` option when converting the *first* Mercurial commit into a non-empty Git repository. By default, `hg-fast-export` prevents importing Mercurial commits onto a non-empty Git repo to avoid creating conflicting histories. Passing `--force` overrides that safety check and allows the exporter to write the LFS pointer objects and integrate the converted data with the existing Git history.


> 80 cols, please line break.

frej · 2025-12-20T15:29:16Z

plugins/git_lfs_importer/README.md

+
+You only need to pass the `--force` option when converting the *first* Mercurial commit into a non-empty Git repository. By default, `hg-fast-export` prevents importing Mercurial commits onto a non-empty Git repo to avoid creating conflicting histories. Passing `--force` overrides that safety check and allows the exporter to write the LFS pointer objects and integrate the converted data with the existing Git history.
+
+If you are doing an incremental conversion (i.e., running the script a second time to import new changesets into an already converted repository), the --force flag is not required.


> 80 cols, please line break.

frej · 2025-12-20T15:29:22Z

plugins/git_lfs_importer/README.md

+
+If you are doing an incremental conversion (i.e., running the script a second time to import new changesets into an already converted repository), the --force flag is not required.
+
+Omitting `--force` when attempting to import the first Mercurial commit into a non-empty repository will cause the importer to refuse the operation.


> 80 cols, please line break.

frej · 2025-12-20T15:34:27Z

plugins/git_lfs_importer/README.md

+
+### Dependencies
+
+This plugin requires the `pathspec` package:


You're the author and fast-export didn't use pathspec before, so pick a version/versions that are compatible with the earliest/latest versions we support for fast-export.

I have approved the workflow, so we'll soon see if/what breaks if its missing.

The current conversion process mandates an empty repository for a clean start. This presents a barrier to performance optimization strategies. This change introduces the ability to pass a repository root commit hash. This is necessary to support the immediate next commit (Incremental LFS conversion), which uses a `.gitattributes` file and LFS pointers to bypass the slow, full-history rewriting often required on large non-empty monorepos (100GiB+, 1M+ files). The immediate benefit is allowing conversion to start when a non-empty repo already contains an orphan commit, laying the groundwork for the optimized LFS conversion feature.

Converts large Mercurial repositories to Git/LFS significantly faster by integrating the LFS conversion into the history export process. Currently, converting large repositories requires two sequential, long-running steps: 1. Full history conversion (`hg` to `git`). 2. Full history rewrite/import (`git lfs import`). For huge monorepos (100GiB+, 1M+ files), this sequence can take hours or days. This commit introduces a new plugin that allows the repository to be converted *incrementally* (JIT: Just-In-Time). The plugin identifies large files during the initial `hg` to `git` conversion and immediately writes LFS pointers, eliminating the need for the second, time-consuming history rewrite step.

The data example was previously a single line, making it difficult to read. Replaced the one-liner with a multi-line format to improve clarity.

Lesvek · 2025-12-22T19:11:48Z

Solid work with testcases and well-written documentation. I just have some formatting nits as review comments.

Would you consider adding a pointer to the new plugin to the "Mercurial Largefiles Extension" section of the top level README?

I updated the largefile documentation in the top level README.

Lesvek commented Dec 12, 2025

View reviewed changes

frej reviewed Dec 20, 2025

View reviewed changes

Lesvek force-pushed the task/non-empty-repo branch from 065d026 to 237e25c Compare December 22, 2025 19:00

Lesvek added 2 commits December 22, 2025 14:07

Split plugin data example into multiple lines

255d49c

The data example was previously a single line, making it difficult to read. Replaced the one-liner with a multi-line format to improve clarity.

Lesvek force-pushed the task/non-empty-repo branch from 237e25c to 255d49c Compare December 22, 2025 19:07


		### Dependencies

		This plugin requires the `pathspec` package:


		FIRST_HASH=$(git -C gitrepo rev-parse HEAD) &&

		# 3. Run hg-fast-export with git_lfs_importer plugin


		### Large Files and Largefiles

		If the Mercurial repository uses Mercurial's largefiles extension, those files are already converted to their original content before reaching this plugin, allowing the plugin to apply LFS conversion if they match the patterns.


		### Force Requirement

		You only need to pass the `--force` option when converting the first Mercurial commit into a non-empty Git repository. By default, `hg-fast-export` prevents importing Mercurial commits onto a non-empty Git repo to avoid creating conflicting histories. Passing `--force` overrides that safety check and allows the exporter to write the LFS pointer objects and integrate the converted data with the existing Git history.


		You only need to pass the `--force` option when converting the first Mercurial commit into a non-empty Git repository. By default, `hg-fast-export` prevents importing Mercurial commits onto a non-empty Git repo to avoid creating conflicting histories. Passing `--force` overrides that safety check and allows the exporter to write the LFS pointer objects and integrate the converted data with the existing Git history.

		If you are doing an incremental conversion (i.e., running the script a second time to import new changesets into an already converted repository), the --force flag is not required.


		If you are doing an incremental conversion (i.e., running the script a second time to import new changesets into an already converted repository), the --force flag is not required.

		Omitting `--force` when attempting to import the first Mercurial commit into a non-empty repository will cause the importer to refuse the operation.

Feature: Enable Just-In-Time (JIT) Git LFS Conversion #347

Are you sure you want to change the base?

Feature: Enable Just-In-Time (JIT) Git LFS Conversion #347

Uh oh!

Conversation

Lesvek commented Dec 12, 2025

Key Changes:

Usage Example:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frej left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lesvek commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

frej left a comment •

edited

Loading