From cf63e5770988a659d44b08c98315f9ac97b1a8e3 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 16:57:53 -0400 Subject: [PATCH 01/20] Update rest-api.md --- docs/REST API/rest-api.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index 04c50e2564d..775a558038d 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1311,13 +1311,15 @@ None. - [Get carve](#get-carve) - [Get carve block](#get-carve-block) -Fleet supports osquery's file carving functionality as of Fleet 3.3.0. This allows the Fleet server to request files (and sets of files) from Fleet's agent (fleetd), returning the full contents to Fleet. +> TODO: Warning on size of carve -To initiate a file carve using the Fleet API, you can use the [live query](#run-live-query) endpoint to run a query against the `carves` table. +File carving allows you to request files (and sets of files) and their full contents. -Keep in mind that any failure when uploading a file block (like a network error) will result on a failed carved file. Starting in osquery v5.22.1, block uploads will be retried up to three times before failing. +To initiate a file carve using the Fleet API, use the [live query](#run-live-query) endpoint to run a query against the `carves` table. -For more information on executing a file carve in Fleet, go to the [File carving with Fleet docs](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/product-groups/orchestration/file-carving.md). +Any failure when uploading a file block (like a network error) will result on a failed carved file. Block uploads are retried up to three times before failing. + +To learn more about executing a file carve in Fleet, head to the [file carving guide](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/product-groups/orchestration/file-carving.md). ### List carves From 53d0afa29757857471b5c5b0e65ab4c8b5b131b4 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:01:07 -0400 Subject: [PATCH 02/20] Create file-carving.md --- articles/file-carving.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 articles/file-carving.md diff --git a/articles/file-carving.md b/articles/file-carving.md new file mode 100644 index 00000000000..f1166f13db9 --- /dev/null +++ b/articles/file-carving.md @@ -0,0 +1,9 @@ +# File carving + +TODO + + + + + + From 1d05837bf93f4305c211025ca57578f8f179a70e Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:03:03 -0400 Subject: [PATCH 03/20] Apply suggestion from @noahtalerman --- docs/REST API/rest-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index 775a558038d..fe72b36cc55 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1313,7 +1313,7 @@ None. > TODO: Warning on size of carve -File carving allows you to request files (and sets of files) and their full contents. +File carving allows you to request files (and sets of files) and their full contents from hosts. To initiate a file carve using the Fleet API, use the [live query](#run-live-query) endpoint to run a query against the `carves` table. From 04f97685b5c4e7d93ac1c2d16b2fab7a02adcb74 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:10:05 -0400 Subject: [PATCH 04/20] Update file-carving.md --- articles/file-carving.md | 115 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 114 insertions(+), 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index f1166f13db9..c787fb38425 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -1,6 +1,119 @@ # File carving -TODO +Fleet supports file carving which allows you to request files (and sets of files) and their full contents from hosts. + +File carving data can be either stored in Fleet's database or to an external S3 bucket. For information on how to configure the latter, consult the [configuration docs](https://fleetdm.com/docs/deploying/configuration#s-3-file-carving-backend). + +## Setup + +In your agent configuration, add the following [command line flags](https://fleetdm.com/docs/configuration/agent-configuration#options-and-command-line-flags) to enable carving: + +```yaml +command_line_flags: + disable_carver=false + carver_disable_function=false + carver_start_endpoint=/api/v1/osquery/carve/begin + carver_continue_endpoint=/api/v1/osquery/carve/block + carver_block_size=8000000 +``` + +For the (default) MySQL Backend, the configured `carver_block_size` must be less than the value of +`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB. + +For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the +[constraints of S3's multipart +uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html). + +Compression of the carve contents can be enabled with the `carver_compression` flag. When used, the carve results will be compressed with [Zstandard](https://facebook.github.io/zstd/) compression. + +## Create carves + +File carves are initiated with live reports. Run live report using the `carves` table, providing `carve = 1` along with the desired path(s) as constraints. + +For example, to extract the `/etc/hosts` file on a host with hostname `mac-workstation`: + +```sh +fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"' +``` + +The globbing syntax is also supported to carve entire directories or more: + +```sh +fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"' +``` + +## Retrieve carves + +List the non-expired (see below) carves with `fleetctl get carves`. Note that carves will not be available through this command until Fleet's agent (fleetd) checks in to the Fleet server with the first of the carve contents. This can take some time from initiation of the carve. + +To also retrieve expired carves, use `fleetctl get carves --expired`. + +Contents of carves are returned as .tar archives, and compressed if that option is configured. + +To download the contents of a carve with ID 3, use + +```sh +fleetctl get carve --outfile carve.tar 3 +``` + +It can also be useful to pipe the results directly into the tar command for unarchiving: + +```sh +fleetctl get carve --stdout 3 | tar -x +``` + +## Expiration + +Carve contents remain available for 24 hours after the first data is provided from the Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database and the carve is marked as "expired". + +The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to setup a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. + +## Alternative carving backends + +#### RustFS + +Configure the following: +- `FLEET_S3_ENDPOINT_URL=rustfs_host:port` +- `FLEET_S3_BUCKET=bucket_name` +- `FLEET_S3_SECRET_ACCESS_KEY=your_secret_access_key` +- `FLEET_S3_ACCESS_KEY_ID=access_key_id` +- `FLEET_S3_FORCE_S3_PATH_STYLE=true` +- `FLEET_S3_REGION=localhost` or any non-empty string otherwise Fleet will attempt to derive the region. + +If you're testing file carving locally with the docker-compose environment, the `--dev` flag on Fleet server will automatically point carves to the local RustFS container and write to the `carves-dev` bucket (created automatically) without needing to set additional configuration. + +## Troubleshooting + +### Check carve status + +You can report on the status of carves through queries to the `carves` table. + +The details provided by + +```sh +fleetctl query --labels 'All Hosts' --query 'SELECT * FROM carves' +``` + +can be helpful to debug carving problems. + +### Ensure `carver_block_size` is set appropriately + +`carver_block_size` is an option that sets the size of each part of a file carve that Fleet's agent (fleetd) +sends to the Fleet server. + +When using the MySQL backend (default), this value must be less than the `max_allowed_packet` +setting in MySQL. If it is too large, MySQL will reject the writes. + +When using S3, the value must be at least 5MiB (5242880 bytes), as smaller multipart upload +sizes are rejected. Additionally, [S3 +limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) the maximum number of +parts to 10,000. + +The value must be small enough that HTTP requests do not time out. + +Start with a default of 2MiB for MySQL (2097152 bytes), and 5MiB for S3 (5242880 bytes). + + From fc9a50d1984ad34725f0afac812e24d0b01f91be Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:11:11 -0400 Subject: [PATCH 05/20] Apply suggestion from @noahtalerman --- docs/REST API/rest-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index fe72b36cc55..9f7ecdb63f8 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1315,7 +1315,7 @@ None. File carving allows you to request files (and sets of files) and their full contents from hosts. -To initiate a file carve using the Fleet API, use the [live query](#run-live-query) endpoint to run a query against the `carves` table. +To initiate a file carve using the Fleet API, use the [live report](#run-live-report) endpoint to run a query against the `carves` table. Any failure when uploading a file block (like a network error) will result on a failed carved file. Block uploads are retried up to three times before failing. From 4e6a711d3fcceb17f77dc6864f2fb810af485335 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:12:10 -0400 Subject: [PATCH 06/20] Apply suggestion from @noahtalerman --- docs/REST API/rest-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index 9f7ecdb63f8..25d64382cea 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1319,7 +1319,7 @@ To initiate a file carve using the Fleet API, use the [live report](#run-live-re Any failure when uploading a file block (like a network error) will result on a failed carved file. Block uploads are retried up to three times before failing. -To learn more about executing a file carve in Fleet, head to the [file carving guide](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/product-groups/orchestration/file-carving.md). +To learn more about executing a file carve in Fleet, head to the [file carving guide](https://fleetdm.com/guides/file-carving). ### List carves From ad975630f9ba5daf8dd27840cc713628fdc54c2e Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:13:19 -0400 Subject: [PATCH 07/20] Delete docs/Contributing/product-groups/orchestration/file-carving.md --- .../orchestration/file-carving.md | 123 ------------------ 1 file changed, 123 deletions(-) delete mode 100644 docs/Contributing/product-groups/orchestration/file-carving.md diff --git a/docs/Contributing/product-groups/orchestration/file-carving.md b/docs/Contributing/product-groups/orchestration/file-carving.md deleted file mode 100644 index 5b136f36b34..00000000000 --- a/docs/Contributing/product-groups/orchestration/file-carving.md +++ /dev/null @@ -1,123 +0,0 @@ -## File carving - -Fleet supports osquery's file carving functionality as of Fleet 3.3.0. This allows the Fleet server to request files (and sets of files) from Fleet's agent (fleetd) returning the full contents to Fleet. - -File carving data can be either stored in Fleet's database or to an external S3 bucket. For information on how to configure the latter, consult the [configuration docs](https://fleetdm.com/docs/deploying/configuration#s-3-file-carving-backend). - -### Configuration - -Given a working flagfile for connecting fleetd to Fleet, add the following flags to enable carving: - -```sh ---disable_carver=false ---carver_disable_function=false ---carver_start_endpoint=/api/v1/osquery/carve/begin ---carver_continue_endpoint=/api/v1/osquery/carve/block ---carver_block_size=8000000 -``` - -The default flagfile provided in the "Add new host" dialog also includes this configuration. - -#### Carver block size - -The `carver_block_size` flag should be configured in osquery. - -For the (default) MySQL Backend, the configured value must be less than the value of -`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB. - -For the S3-compatible backend, this value must be set to at least 5MiB (`5242880`) due to the -[constraints of S3's multipart -uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html). - -#### Compression - -Compression of the carve contents can be enabled with the `carver_compression` flag in osquery. When used, the carve results will be compressed with [Zstandard](https://facebook.github.io/zstd/) compression. - -### Usage - -File carves are initiated with osquery queries. Issue a query to the `carves` table, providing `carve = 1` along with the desired path(s) as constraints. - -For example, to extract the `/etc/hosts` file on a host with hostname `mac-workstation`: - -```sh -fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"' -``` - -The standard osquery file globbing syntax is also supported to carve entire directories or more: - -```sh -fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"' -``` - -#### Retrieving carves - -List the non-expired (see below) carves with `fleetctl get carves`. Note that carves will not be available through this command until osquery checks in to the Fleet server with the first of the carve contents. This can take some time from initiation of the carve. - -To also retrieve expired carves, use `fleetctl get carves --expired`. - -Contents of carves are returned as .tar archives, and compressed if that option is configured. - -To download the contents of a carve with ID 3, use - -```sh -fleetctl get carve --outfile carve.tar 3 -``` - -It can also be useful to pipe the results directly into the tar command for unarchiving: - -```sh -fleetctl get carve --stdout 3 | tar -x -``` - -#### Expiration - -Carve contents remain available for 24 hours after the first data is provided from the osquery client. After this time, the carve contents are cleaned from the database and the carve is marked as "expired". - -The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to setup a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. - -### Alternative carving backends - -#### RustFS - -Configure the following: -- `FLEET_S3_ENDPOINT_URL=rustfs_host:port` -- `FLEET_S3_BUCKET=bucket_name` -- `FLEET_S3_SECRET_ACCESS_KEY=your_secret_access_key` -- `FLEET_S3_ACCESS_KEY_ID=access_key_id` -- `FLEET_S3_FORCE_S3_PATH_STYLE=true` -- `FLEET_S3_REGION=localhost` or any non-empty string otherwise Fleet will attempt to derive the region. - -If you're testing file carving locally with the docker-compose environment, the `--dev` flag on Fleet server will automatically point carves to the local RustFS container and write to the `carves-dev` bucket (created automatically) without needing to set additional configuration. - -### Troubleshooting - -#### Check carve status in osquery - -Osquery can report on the status of carves through queries to the `carves` table. - -The details provided by - -```sh -fleetctl query --labels 'All Hosts' --query 'SELECT * FROM carves' -``` - -can be helpful to debug carving problems. - -#### Ensure `carver_block_size` is set appropriately - -`carver_block_size` is an osquery flag that sets the size of each part of a file carve that osquery -sends to the Fleet server. - -When using the MySQL backend (default), this value must be less than the `max_allowed_packet` -setting in MySQL. If it is too large, MySQL will reject the writes. - -When using S3, the value must be at least 5MiB (5242880 bytes), as smaller multipart upload -sizes are rejected. Additionally, [S3 -limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) the maximum number of -parts to 10,000. - -The value must be small enough that HTTP requests do not time out. - -Start with a default of 2MiB for MySQL (2097152 bytes), and 5MiB for S3 (5242880 bytes). - - From 08178fce75a14a2756e83fd11e570f2bdf05fb7d Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:26:04 -0400 Subject: [PATCH 08/20] Apply suggestion from @noahtalerman --- docs/REST API/rest-api.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index 25d64382cea..e20ba5192dd 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1311,7 +1311,6 @@ None. - [Get carve](#get-carve) - [Get carve block](#get-carve-block) -> TODO: Warning on size of carve File carving allows you to request files (and sets of files) and their full contents from hosts. From 7003ef2d4e170b02ecede1dea0b98b24f09fb6b0 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 10 Jun 2026 17:38:00 -0400 Subject: [PATCH 09/20] Update file-carving.md --- articles/file-carving.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index c787fb38425..5971b355afd 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -113,10 +113,9 @@ The value must be small enough that HTTP requests do not time out. Start with a default of 2MiB for MySQL (2097152 bytes), and 5MiB for S3 (5242880 bytes). - - - + + From 055e0a7d97714ed266eab0db7ecbeefa5e640c94 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Mon, 15 Jun 2026 10:37:38 -0400 Subject: [PATCH 10/20] Update articles/file-carving.md --- articles/file-carving.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/articles/file-carving.md b/articles/file-carving.md index 5971b355afd..f41d5409432 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -28,6 +28,13 @@ Compression of the carve contents can be enabled with the `carver_compression` f ## Create carves +> File carving can cause significant performance impact if multiple factors are scaled up simultaneously. To avoid overloading your Fleet instance: +> - Target a narrow host set. Avoid running carves against all hosts. +> - Use specific paths. Avoid wildcard paths (e.g. /tmp/* or user home directories) that may match many or large files. +> - Mind the limits. Individual files must be under 8 GB; each file is transferred in 256 MB blocks (one database row and S3 object per block). +> - Avoid scheduled carves on broad targets. Automations that repeat carves against large host sets compound the load over time. +> The total load scales as the product of: number of hosts × number of paths × number of matching files × average file size. Any one of these can be large in isolation, but all four at once can result in millions of database writes and terabytes of S3 data simultaneously. + File carves are initiated with live reports. Run live report using the `carves` table, providing `carve = 1` along with the desired path(s) as constraints. For example, to extract the `/etc/hosts` file on a host with hostname `mac-workstation`: From bb18a783e2e71d44175e80306c6e7f4142523e00 Mon Sep 17 00:00:00 2001 From: Rachael Shaw Date: Tue, 16 Jun 2026 16:27:59 -0500 Subject: [PATCH 11/20] fix typo --- docs/REST API/rest-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/REST API/rest-api.md b/docs/REST API/rest-api.md index f504623473c..2f17e255b93 100644 --- a/docs/REST API/rest-api.md +++ b/docs/REST API/rest-api.md @@ -1316,7 +1316,7 @@ File carving allows you to request files (and sets of files) and their full cont To initiate a file carve using the Fleet API, use the [live report](#run-live-report) endpoint to run a query against the `carves` table. -Any failure when uploading a file block (like a network error) will result on a failed carved file. Block uploads are retried up to three times before failing. +Any failure when uploading a file block (like a network error) will result in a failed carved file. Block uploads are retried up to three times before failing. To learn more about executing a file carve in Fleet, head to the [file carving guide](https://fleetdm.com/guides/file-carving). From 8b67e77c523b0990fe6ccfa9225f87642e32f402 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 17 Jun 2026 10:41:46 -0400 Subject: [PATCH 12/20] Apply suggestion from @sharon-fdm Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com> --- articles/file-carving.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index f41d5409432..28ffd0c3837 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -9,11 +9,11 @@ File carving data can be either stored in Fleet's database or to an external S3 In your agent configuration, add the following [command line flags](https://fleetdm.com/docs/configuration/agent-configuration#options-and-command-line-flags) to enable carving: ```yaml -command_line_flags: - disable_carver=false - carver_disable_function=false - carver_start_endpoint=/api/v1/osquery/carve/begin - carver_continue_endpoint=/api/v1/osquery/carve/block + disable_carver: false + carver_disable_function: false + carver_start_endpoint: /api/v1/osquery/carve/begin + carver_continue_endpoint: /api/v1/osquery/carve/block + carver_block_size: 8000000 carver_block_size=8000000 ``` From 98f215a90db9a9df14f24148e512767f3bcfa924 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Wed, 17 Jun 2026 17:30:05 -0400 Subject: [PATCH 13/20] Apply suggestion from @noahtalerman --- articles/file-carving.md | 1 - 1 file changed, 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index 28ffd0c3837..50f9c29628c 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -14,7 +14,6 @@ In your agent configuration, add the following [command line flags](https://flee carver_start_endpoint: /api/v1/osquery/carve/begin carver_continue_endpoint: /api/v1/osquery/carve/block carver_block_size: 8000000 - carver_block_size=8000000 ``` For the (default) MySQL Backend, the configured `carver_block_size` must be less than the value of From fa27c1ffb88550790cd5885866e34aa1cd092599 Mon Sep 17 00:00:00 2001 From: Steven Palmesano <3100993+spalmesano0@users.noreply.github.com> Date: Thu, 18 Jun 2026 07:00:18 -0500 Subject: [PATCH 14/20] `fleetctl query` -> `fleetctl report` --- articles/file-carving.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index 50f9c29628c..081d0641f6d 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -17,7 +17,7 @@ In your agent configuration, add the following [command line flags](https://flee ``` For the (default) MySQL Backend, the configured `carver_block_size` must be less than the value of -`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB. +`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB (`67108864`). For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the [constraints of S3's multipart @@ -39,13 +39,13 @@ File carves are initiated with live reports. Run live report using the `carves` For example, to extract the `/etc/hosts` file on a host with hostname `mac-workstation`: ```sh -fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"' +fleetctl report --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"' ``` The globbing syntax is also supported to carve entire directories or more: ```sh -fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"' +fleetctl report --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"' ``` ## Retrieve carves @@ -97,7 +97,7 @@ You can report on the status of carves through queries to the `carves` table. The details provided by ```sh -fleetctl query --labels 'All Hosts' --query 'SELECT * FROM carves' +fleetctl report --labels 'All Hosts' --query 'SELECT * FROM carves' ``` can be helpful to debug carving problems. From 2540abff69512081df73ed9a7c2c03c9a947e7d3 Mon Sep 17 00:00:00 2001 From: Rachael Shaw Date: Thu, 18 Jun 2026 16:37:46 -0500 Subject: [PATCH 15/20] A few small wording changes in the guide Co-authored-by: Rachael Shaw --- articles/file-carving.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index 081d0641f6d..fe931c195fa 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -42,7 +42,7 @@ For example, to extract the `/etc/hosts` file on a host with hostname `mac-works fleetctl report --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"' ``` -The globbing syntax is also supported to carve entire directories or more: +Glob syntax is also supported to carve entire directories or more: ```sh fleetctl report --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"' @@ -70,9 +70,9 @@ fleetctl get carve --stdout 3 | tar -x ## Expiration -Carve contents remain available for 24 hours after the first data is provided from the Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database and the carve is marked as "expired". +Carve contents remain available for 24 hours after the first data is provided from Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database and the carve is marked as "expired". -The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to setup a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. +The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to set up a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. ## Alternative carving backends @@ -94,13 +94,12 @@ If you're testing file carving locally with the docker-compose environment, the You can report on the status of carves through queries to the `carves` table. -The details provided by +You can debug carving problems with: ```sh fleetctl report --labels 'All Hosts' --query 'SELECT * FROM carves' ``` -can be helpful to debug carving problems. ### Ensure `carver_block_size` is set appropriately From efa3f1facd59a0db2f68af7d40d198b1c34aaef5 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Fri, 19 Jun 2026 09:37:14 -0400 Subject: [PATCH 16/20] Update articles/file-carving.md Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com> --- articles/file-carving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index fe931c195fa..a9dbd79881a 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -1,6 +1,6 @@ # File carving -Fleet supports file carving which allows you to request files (and sets of files) and their full contents from hosts. +Fleet supports file carving, which allows you to request files (and sets of files) and their full contents from hosts. File carving data can be either stored in Fleet's database or to an external S3 bucket. For information on how to configure the latter, consult the [configuration docs](https://fleetdm.com/docs/deploying/configuration#s-3-file-carving-backend). From d6513dee23fb530403fe08774f347798b4183020 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Fri, 19 Jun 2026 09:37:27 -0400 Subject: [PATCH 17/20] Update articles/file-carving.md Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com> --- articles/file-carving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index a9dbd79881a..d2a8d66c818 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -70,7 +70,7 @@ fleetctl get carve --stdout 3 | tar -x ## Expiration -Carve contents remain available for 24 hours after the first data is provided from Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database and the carve is marked as "expired". +Carve contents remain available for 24 hours after the first data is provided from Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database, and the carve is marked as "expired". The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to set up a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. From 95817396ba19e1391bd0c30f65b7983210eee91c Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Fri, 19 Jun 2026 09:37:36 -0400 Subject: [PATCH 18/20] Update articles/file-carving.md Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com> --- articles/file-carving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index d2a8d66c818..fc0a59c72e3 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -72,7 +72,7 @@ fleetctl get carve --stdout 3 | tar -x Carve contents remain available for 24 hours after the first data is provided from Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database, and the carve is marked as "expired". -The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to set up a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket. +The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to set up a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e., by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what is actually available in the bucket. ## Alternative carving backends From fccbf7de4601ebac9a0e9fb5d380a737eae4a4f3 Mon Sep 17 00:00:00 2001 From: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> Date: Fri, 19 Jun 2026 09:40:41 -0400 Subject: [PATCH 19/20] Update file-carving.md --- articles/file-carving.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index fc0a59c72e3..f2e91574ecf 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -16,11 +16,9 @@ In your agent configuration, add the following [command line flags](https://flee carver_block_size: 8000000 ``` -For the (default) MySQL Backend, the configured `carver_block_size` must be less than the value of -`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB (`67108864`). +The configured `carver_block_size` must be less than the value of `max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB (`67108864`). -For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the -[constraints of S3's multipart +For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the [constraints of S3's multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html). Compression of the carve contents can be enabled with the `carver_compression` flag. When used, the carve results will be compressed with [Zstandard](https://facebook.github.io/zstd/) compression. @@ -86,7 +84,7 @@ Configure the following: - `FLEET_S3_FORCE_S3_PATH_STYLE=true` - `FLEET_S3_REGION=localhost` or any non-empty string otherwise Fleet will attempt to derive the region. -If you're testing file carving locally with the docker-compose environment, the `--dev` flag on Fleet server will automatically point carves to the local RustFS container and write to the `carves-dev` bucket (created automatically) without needing to set additional configuration. +If you're testing file carving locally, the `--dev` flag on Fleet server will automatically point carves to the local RustFS container and write to the `carves-dev` bucket (created automatically) without needing to set additional configuration. ## Troubleshooting @@ -103,15 +101,12 @@ fleetctl report --labels 'All Hosts' --query 'SELECT * FROM carves' ### Ensure `carver_block_size` is set appropriately -`carver_block_size` is an option that sets the size of each part of a file carve that Fleet's agent (fleetd) -sends to the Fleet server. +`carver_block_size` is an option that sets the size of each part of a file carve that Fleet's agent (fleetd) sends to the Fleet server. -When using the MySQL backend (default), this value must be less than the `max_allowed_packet` -setting in MySQL. If it is too large, MySQL will reject the writes. +When using the MySQL backend (default), this value must be less than the `max_allowed_packet` setting in MySQL. If it is too large, MySQL will reject the writes. When using S3, the value must be at least 5MiB (5242880 bytes), as smaller multipart upload -sizes are rejected. Additionally, [S3 -limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) the maximum number of +sizes are rejected. Additionally, [S3 limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) the maximum number of parts to 10,000. The value must be small enough that HTTP requests do not time out. From 146365ad382758e781802d7d41201cfe34cd5bd9 Mon Sep 17 00:00:00 2001 From: Rachael Shaw Date: Fri, 19 Jun 2026 13:06:35 -0500 Subject: [PATCH 20/20] Small grammatical fix --- articles/file-carving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/file-carving.md b/articles/file-carving.md index f2e91574ecf..09a1df50adb 100644 --- a/articles/file-carving.md +++ b/articles/file-carving.md @@ -16,7 +16,7 @@ In your agent configuration, add the following [command line flags](https://flee carver_block_size: 8000000 ``` -The configured `carver_block_size` must be less than the value of `max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB (`67108864`). +The configured `carver_block_size` must be less than the value of `max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) is 64MB (`67108864`). For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the [constraints of S3's multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html).