Skip to content

Parallelize file uploads in fs cp command. #4132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
renaudhartert-db wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Parallelize file uploads in fs cp command. #4132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
5896ae4
Fs cp now upload in parallel
renaudhartert-db Dec 11, 2025
b933398
Improve performance
renaudhartert-db Dec 11, 2025
2fa03e1
Use error value
renaudhartert-db Dec 11, 2025
d393388
Cancel go routines
renaudhartert-db Dec 11, 2025
14cfad9
Properly propagate context
renaudhartert-db Dec 11, 2025
2a9ec1b
Linter
renaudhartert-db Dec 11, 2025
179a944
Test context cancellation
renaudhartert-db Dec 12, 2025
79dd600
Clarified rationale
renaudhartert-db Dec 12, 2025
2d1f19d
Gracefully wait for cancellation to complete
renaudhartert-db Dec 12, 2025
caaff68
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 13, 2025
12dec5c
Add acceptance tests for fs cp
renaudhartert-db Dec 15, 2025
2be2174
Merge branch 'renaud-hartert_data/fs-cp-fast' of github.com-ghec:data…
renaudhartert-db Dec 15, 2025
7db079d
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 16, 2025
e720670
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 16, 2025
4029571
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
### CLI
* Skip non-exportable objects (e.g., `MLFLOW_EXPERIMENT`) during `workspace export-dir` instead of failing ([#4081](https://github.com/databricks/cli/issues/4081))

* Improve performance of `databricks fs cp` command by parallelizing file uploads when
copying directories with the `--recursive` flag.

### Bundles

### Dependency updates
Expand Down
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
localdir/file1.txt -> dbfs:/Volumes/main/default/data/uploaded-dir/file1.txt
localdir/file2.txt -> dbfs:/Volumes/main/default/data/uploaded-dir/file2.txt
6 changes: 6 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
mkdir -p localdir
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of creating these files at test runtime, you can create and commit them here. That way, the test can directly read them without having to write them first.

I recommend commiting input files directly unless the file contents are dynamic.

echo -n "file1 content" > localdir/file1.txt
echo -n "file2 content" > localdir/file2.txt

# Recursive directory copy (output sorted for deterministic ordering).
$CLI fs cp -r localdir dbfs:/Volumes/main/default/data/uploaded-dir 2>&1 | sort
20 changes: 20 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
Local = true
Cloud = false
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend setting Cloud = true for this test as well. That validates the implementation against a real server. You can use fs cat to confirm that the files were uploaded.

Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also nice to ensure that the local test server implementation matches the remote behaviour, atleast for the interfaces we own (like the fs commands)

Ignore = ["localdir"]

# Recursive copy: localdir/ -> uploaded-dir/.
[[Server]]
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a local sserver supporing these operations (see: libs/testserver/handlers.go). This should work even without the overrides:

	server.Handle("HEAD", "/api/2.0/fs/directories/{path:.*}", func(req Request) any {
		return Response{
			Body: "dir path: " + req.Vars["dir_path"],
		}
	})

	server.Handle("PUT", "/api/2.0/fs/files/{path:.*}", func(req Request) any {
		path := req.Vars["path"]
		overwrite := req.URL.Query().Get("overwrite") == "true"
		return req.Workspace.WorkspaceFilesImportFile(path, req.Body, overwrite)
	})

Pattern = "PUT /api/2.0/fs/directories/Volumes/main/default/data/uploaded-dir"
Response.StatusCode = 200

[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/uploaded-dir"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded-dir/file1.txt"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded-dir/file2.txt"
Response.StatusCode = 200
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

>>> [CLI] fs cp local.txt dbfs:/Volumes/main/default/data/mydir/
local.txt -> dbfs:/Volumes/main/default/data/mydir/local.txt
4 changes: 4 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
echo -n "hello world!" > local.txt
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above, local.txt can be commited.


# Copy file into a directory (trailing slash indicates directory target).
trace $CLI fs cp local.txt dbfs:/Volumes/main/default/data/mydir/
12 changes: 12 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Local = true
Cloud = false
Ignore = ["local.txt"]
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you commit local.txt, you can remove this config. Same for the test above.


# Copy file into existing directory: local.txt -> mydir/local.txt.
[[Server]]
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as earlier, we should not need these overrides.

Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/mydir"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/mydir/local.txt"
Response.StatusCode = 200
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

>>> [CLI] fs cp local.txt dbfs:/Volumes/main/default/data/uploaded.txt
local.txt -> dbfs:/Volumes/main/default/data/uploaded.txt

>>> [CLI] fs cp dbfs:/Volumes/main/default/data/remote.txt downloaded.txt
dbfs:/Volumes/main/default/data/remote.txt -> downloaded.txt
content from volume
9 changes: 9 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
echo -n "hello world!" > local.txt

# Upload local file to volume.
trace $CLI fs cp local.txt dbfs:/Volumes/main/default/data/uploaded.txt

# Download file from volume to local.
trace $CLI fs cp dbfs:/Volumes/main/default/data/remote.txt downloaded.txt

cat downloaded.txt
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(optional)

Suggested change
cat downloaded.txt
trace cat downloaded.txt
rm download.txt

By removing it here, you do not need to Ignore it in test.toml.

34 changes: 34 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
Local = true
Cloud = false
Ignore = ["local.txt", "downloaded.txt"]

# Upload: local.txt -> dbfs:/Volumes/.../uploaded.txt.
[[Server]]
Copy link
Contributor

@shreyas-goenka shreyas-goenka Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, remove overrides.

Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/files/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 200

# Download: dbfs:/Volumes/.../remote.txt -> downloaded.txt.
[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/remote.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/files/Volumes/main/default/data/remote.txt"
Response.StatusCode = 200

[[Server]]
Pattern = "GET /api/2.0/fs/files/Volumes/main/default/data/remote.txt"
Response.StatusCode = 200
Response.Body = "content from volume"
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@

>>> errcode [CLI] fs cp src dst --concurrency -1
Error: --concurrency must be at least 1

Exit code: 1

>>> errcode [CLI] fs cp src dst --concurrency 0
Error: --concurrency must be at least 1

Exit code: 1
3 changes: 3 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Invalid concurrency values should fail.
trace errcode $CLI fs cp src dst --concurrency -1
trace errcode $CLI fs cp src dst --concurrency 0
2 changes: 2 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Local = true
Cloud = false
Loading