Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
172 commits
Select commit Hold shift + click to select a range
c1d1646
Change the way new layout assignations are computed.
Alexis211 May 1, 2022
2aeaddd
Apply cargo fmt
Alexis211 May 1, 2022
3ba2c5b
updated cargo.lock
Alexis211 May 1, 2022
948ff93
Corrected the warnings and errors issued by cargo clippy
May 1, 2022
617f28b
Correct small formatting issue
Alexis211 May 5, 2022
03e3a1b
Added the latex report on the optimal layout algorithm
Jul 18, 2022
7b2c065
Merge branch 'optimal-layout' of https://git.deuxfleurs.fr/Deuxfleurs…
Jul 19, 2022
81083dd
Added a first draft version of the algorithm and analysis for the non…
Aug 19, 2022
d38fb6c
ignore log files in commit
Sep 8, 2022
c4adbee
Added the section with description proofs of the parametric assignmen…
Sep 10, 2022
7f3249a
New version of the algorithm that calculate the layout.
Sep 21, 2022
bd842e1
Correction of a few bugs in the tests, modification of ClusterLayout:…
Sep 22, 2022
99f96b9
deleted zone_redundancy from System struct
Oct 4, 2022
829f815
Merge remote-tracking branch 'origin/main' into optimal-layout
Oct 4, 2022
ceac371
modifications in several files to :
Oct 5, 2022
a951b6c
Added a CLI command to update the parameters for the layout computati…
Oct 5, 2022
9407df6
Corrected two bugs:
Oct 6, 2022
911eb17
corrected warnings of cargo clippy
Oct 6, 2022
fcf9ac6
Tests written in layout.rs
Oct 10, 2022
4abab24
cargo fmt
Oct 10, 2022
e5664c9
Improved the statistics displayed in layout show
Oct 11, 2022
bcdd1e0
Added some comment
Oct 11, 2022
3039bb5
rm .gitattributes
Alexis211 Oct 13, 2022
28d7a49
Merge branch 'main' into optimal-layout
Alexis211 Nov 7, 2022
ea5afc2
Style improvements
Alexis211 Nov 7, 2022
fd5bc14
Ensure .sort() is called before counting unique items
Alexis211 Nov 7, 2022
73a4ca8
Use bytes as capacity units
Alexis211 Nov 7, 2022
d75b37b
Return more info when layout's .check() fails, fix compilation, fix test
Alexis211 Nov 8, 2022
fc2729c
Fix integration test
Alexis211 Nov 8, 2022
217abdc
Fix HTTP return code
Alexis211 Nov 8, 2022
ec12d6c
Slightly simplify code at places
Alexis211 Nov 8, 2022
9d83364
itertools .unique() doesn't require sorted items
Alexis211 Dec 11, 2022
2c2e65a
Merge commit 'ec12d6c' into next
Alexis211 Dec 11, 2022
6e44369
Merge pull request 'Optimal layout assignation algorithm' (#296) from…
Alexis211 Dec 11, 2022
8be862a
Changed all instances of 'key new' to 'key create' to make it consist…
jpds Jan 2, 2023
570e5e5
Merge branch 'main' into next
Alexis211 Jan 4, 2023
80e4abb
Merge pull request 'Changed all instances of 'key new' to 'key create…
Alexis211 Jan 4, 2023
cb07e61
Changed all instances of assignation to assignment.
jpds Jan 5, 2023
e3cc7a8
First draft of t a preprint describing the layout computation algorithm
Jan 9, 2023
597d64b
change in gitignore
Jan 9, 2023
4f409f7
Merge pull request 'Changed all instances of assignation to assignmen…
Alexis211 Jan 11, 2023
84b4a86
Migration of cluster layout from v0.8 to v0.9
Alexis211 Jan 11, 2023
12a4e1f
Merge branch 'optimal-layout' into next
Alexis211 Jan 11, 2023
db56d46
util/Cargo.toml: Updated rmp-serde from 0.15 to 1.1.
jpds Jan 23, 2023
654999e
Update Cargo.nix
Alexis211 Jan 26, 2023
fa78d80
Merge branch 'main' into next
Alexis211 Apr 25, 2023
a1fcf1b
Merge branch 'main' into next
Alexis211 Apr 25, 2023
351d734
Merge branch 'main' into next
Alexis211 May 9, 2023
1963970
Mark sled as deprecated, make lmdb default, and improve sqlite and lm…
Alexis211 May 17, 2023
8a74e1c
Split garage/admin.rs into smaller files
Alexis211 Jun 6, 2023
906fe78
Integration tests: print logs when fails
Alexis211 Jun 9, 2023
865f0c7
Add LMDB to debug builds
Alexis211 Jun 9, 2023
1e466b1
Revert integration tests to using Sled as LMDB causes failures
Alexis211 Jun 9, 2023
e7e164a
Make fsync an option for meta and data
Alexis211 Jun 9, 2023
ea3bfd2
Minio tests for multipart upload behaviour:
Alexis211 Apr 25, 2023
6005491
Use Cow<[u8]> for sort keys
Alexis211 Apr 27, 2023
38d6ac4
New multipart upload table layout
Alexis211 Apr 27, 2023
82e75c0
Adapt S3 API code to use new multipart upload models
Alexis211 May 3, 2023
87be8ee
updaet block admin for new multipartupload models
Alexis211 May 3, 2023
c1e1764
move git-version dependency to main crate to reduce rebuilds
Alexis211 May 3, 2023
bb176eb
cargo fmt
Alexis211 May 3, 2023
75a0e01
fix online repair
Alexis211 May 3, 2023
7ad7dae
fix s3 list test
Alexis211 May 3, 2023
8644376
fix test; simplify code
Alexis211 May 4, 2023
058518c
refactor repair workers with a trait
Alexis211 May 4, 2023
4ea53dc
Add multipart upload repair
Alexis211 May 4, 2023
511e07e
fix mpu counter (add missing workers) and report info at appropriate …
Alexis211 May 4, 2023
412ab77
comments and clippy lint fixes
Alexis211 May 4, 2023
53bf2f0
undo sort_key() returning Cow
Alexis211 May 9, 2023
c14d373
Add test for multipart uploads and fix part renumbering
Alexis211 May 9, 2023
a6cc563
UploadPart: automatic cleanup of version (and reference blocked) when…
Alexis211 Jun 6, 2023
58563ed
Add multipart upload using aws s3api
Alexis211 Jun 9, 2023
e645bbd
smoke test: add multipart upload test with part re-upload
Alexis211 Jun 9, 2023
3d47790
properly delete multipart uploads after completion
Alexis211 Jun 9, 2023
0a06fda
Merge pull request 'Fix #204 (full Multipart Uploads semantics)' (#55…
Alexis211 Jun 9, 2023
942c1f1
multipart uploads: save timestamp
Alexis211 Jun 13, 2023
7126f3e
garage key import: add checks and `--yes` CLI flag (fix #278)
Alexis211 Jun 13, 2023
bf19a44
admin API: add missing camelCase conversions (fix #381)
Alexis211 Jun 13, 2023
90b2d43
Merge branch 'main' into next
Alexis211 Jun 13, 2023
cda957b
update netapp's rmp-serde dependency to v1.1
Alexis211 Jun 13, 2023
5670367
multipartupload in test: add forgotten timestamp
Alexis211 Jun 13, 2023
187240e
Merge branch 'main' into next
Alexis211 Jun 14, 2023
52376d4
admin api: change cluster status/layout to use lists and not maps (fi…
Alexis211 Jun 14, 2023
35c108b
admin api: switch GetClusterHealth to camelcase (fix #381 again)
Alexis211 Jun 14, 2023
2c83006
admin api: fix doc in drafts
Alexis211 Jun 14, 2023
28cc9f1
admin api: make name optionnal for CreateKey
Alexis211 Jun 14, 2023
4a82f63
admin api: move all endpoints to v1/ by default (v0/ still supported)
Alexis211 Jun 14, 2023
7895f99
admin and cli: hide secret keys unless asked
Alexis211 Jun 14, 2023
a83a092
admin: uniformize layout api and improve code
Alexis211 Jun 14, 2023
8ef42c9
admin docs: reformatting, key admin: add check
Alexis211 Jun 14, 2023
2e90e1c
Merge branch 'main' into next
Alexis211 Aug 29, 2023
0b83e05
bucket_table: data model for lifecycle configuration
Alexis211 Aug 29, 2023
8041d9a
s3: add xml structures to serialize/deserialize lifecycle configs
Alexis211 Aug 29, 2023
abf011c
lifecycle: implement validation into garage's internal data structure
Alexis211 Aug 29, 2023
f7b409f
use a NaiveDate in data model, it serializes to string (iso 8601 format)
Alexis211 Aug 30, 2023
a2e0e34
lifecycle: skeleton for lifecycle worker
Alexis211 Aug 30, 2023
2996dc8
lifecycle worker: implement main functionality
Alexis211 Aug 30, 2023
da8b224
lifecycle worker: skip entire bucket when no lifecycle config is set
Alexis211 Aug 30, 2023
0f1849e
lifecycle worker: launch with the rest of Garage
Alexis211 Aug 30, 2023
7200954
lifecycle worker: add logging
Alexis211 Aug 30, 2023
75ccc5a
lifecycle config: store date as given, try to debug
Alexis211 Aug 30, 2023
d2e94e3
lifecycle config: add missing line in merge() and remove tracing
Alexis211 Aug 30, 2023
a1d5728
bucket_table: bucketparams::new doesn't need to be pub
Alexis211 Aug 30, 2023
5c923d4
reference manual: document support for lifecycle configuration
Alexis211 Aug 30, 2023
d94f1c9
reference manual: remove obsolete caveat about multipart uploads
Alexis211 Aug 30, 2023
f0a395e
s3 bucket apis: remove redundant call
Alexis211 Aug 30, 2023
01c327a
lifecycle worker: avoid building chrono's serde feature
Alexis211 Aug 30, 2023
5fad4c4
update cargo.nix
Alexis211 Aug 30, 2023
b2f6796
lifecycle worker: take into account disabled rules
Alexis211 Aug 30, 2023
be03a46
s3api: remove redundant serde rename attribute
Alexis211 Aug 30, 2023
1cfcc61
lifecycle worker: mitigate potential bugs + refactoring
Alexis211 Aug 30, 2023
adbf592
lifecycle worker: use queue_insert and process objects in batches
Alexis211 Aug 31, 2023
a00a526
lifecycle worker: add log message when starting
Alexis211 Aug 31, 2023
f579d6d
lifecycle worker: fix potential inifinite loop
Alexis211 Aug 31, 2023
1cdc321
lifecycle worker: don't get stuck on non-existent bucket
Alexis211 Aug 31, 2023
8e0c020
lifecycle worker: correct small clippy lints
Alexis211 Aug 31, 2023
3f461d8
Merge pull request 'object lifecycles (fix #309)' (#620) from bucket-…
Alexis211 Sep 4, 2023
4b4f200
lifecycle: fix SkipBucket bug
Alexis211 Sep 6, 2023
71c0188
block manager: skeleton for multi-hdd support
Alexis211 Sep 4, 2023
6c420c0
block manager: multi-directory layout computation
Alexis211 Sep 5, 2023
887b323
block manager: use data paths from layout
Alexis211 Sep 5, 2023
a09f867
block manager: move blocks in write_block if necessary
Alexis211 Sep 5, 2023
3199cab
update cargo.nix
Alexis211 Sep 5, 2023
1b8c265
block manager: get rid of check_block_status
Alexis211 Sep 5, 2023
fd00a47
table queue: increase batch size
Alexis211 Sep 5, 2023
93114a9
block manager: refactoring
Alexis211 Sep 5, 2023
3a74844
block manager: fix dir_not_empty
Alexis211 Sep 5, 2023
a44f486
block manager: refactoring & increase max worker count to 8
Alexis211 Sep 5, 2023
55c5149
block manager: fixes in layout
Alexis211 Sep 5, 2023
e308659
block manager: scrub checkpointing
Alexis211 Sep 5, 2023
f38a31b
block manager: avoid incorrect data_dir configs and avoid losing files
Alexis211 Sep 6, 2023
99ed183
block manager: refactor and fix monitoring/statistics
Alexis211 Sep 7, 2023
bca347a
doc: update page on upgradin clusters
Alexis211 Sep 7, 2023
6595efd
Document multi-hdd support
Alexis211 Sep 7, 2023
6b008b5
block manager: add rebalance operation to rebalance multi-hdd setups
Alexis211 Sep 7, 2023
6a067e3
doc: documentation of rebalance repair
Alexis211 Sep 7, 2023
2f112ac
correct free data space accounting for multiple data dirs on same fs
Alexis211 Sep 7, 2023
eb972a8
doc: update multi-hdd section
Alexis211 Sep 7, 2023
2657b5c
block manager: fix bugs
Alexis211 Sep 7, 2023
be91ef6
block manager: fix bug where rebalance didn't delete old copies
Alexis211 Sep 7, 2023
de5d792
block manager: fix indentation (why not detected by cargo fmt?)
Alexis211 Sep 11, 2023
7f9ba49
block manager: remove data_dir field
Alexis211 Sep 11, 2023
9526328
scrub: clear saved checkpoint when canceling scrub
Alexis211 Sep 11, 2023
ba7ac52
block repair: simpler/more robust iterator progress calculation
Alexis211 Sep 11, 2023
7228fbf
Merge pull request 'multi-hdd support (fix #218)' (#625) from multihd…
Alexis211 Sep 11, 2023
ad6b1cc
Merge branch 'main' into next
Alexis211 Sep 11, 2023
51abbb0
Merge branch 'main' into next
Alexis211 Sep 11, 2023
fd7d8fe
Merge branch 'main' into next
Alexis211 Sep 11, 2023
2e229d4
new layout: improve output display
Alexis211 Sep 12, 2023
015ccb3
new layout: make zone_redundancy optionnal (if not set, is maximum)
Alexis211 Sep 18, 2023
749b486
new layout: improve display and fix comments
Alexis211 Sep 18, 2023
0088599
new layout: fix clippy lints
Alexis211 Sep 18, 2023
013b026
update cargo.nix
Alexis211 Sep 18, 2023
3ecc17f
new layout: use deterministic randomness for reproducible results
Alexis211 Sep 21, 2023
f97168f
garage_db: refactor transactions and add on_commit mechanism
Alexis211 Sep 21, 2023
0635250
garage_table/queue_insert: delay worker notification to after transac…
Alexis211 Sep 21, 2023
1d986bd
Merge pull request 'Refactor db transactions and add on_commit for ta…
Alexis211 Sep 21, 2023
b4a0e63
new layout doc: add examples of unexpected layout, to explain
Alexis211 Sep 22, 2023
405aa42
layout doc: update old text
Alexis211 Sep 22, 2023
8d07888
layout doc: write explanations for bizarre scenarios
Alexis211 Sep 22, 2023
0e5925f
layout doc: reformulate
Alexis211 Sep 22, 2023
aa7eadc
Merge pull request 'New layout: fixes and UX improvements' (#634) fro…
Alexis211 Sep 27, 2023
ad82035
Merge branch 'main' into next
Alexis211 Sep 27, 2023
897cbf2
actually update rmp-serde to 1.1.2 for both garage and netapp depende…
Alexis211 Sep 27, 2023
9ac1d5b
add upgrade test for garage 0.8 -> 0.9
Alexis211 Sep 27, 2023
2e656b5
Merge branch 'main' into next
Alexis211 Oct 3, 2023
920dec3
cli: more precise doc comment
Alexis211 Oct 4, 2023
9ccc1d6
move upgrade test to release build
Alexis211 Oct 5, 2023
6790e24
Add migration to v0.9 guide
Alexis211 Oct 5, 2023
2448eb7
upgrade doc: fixes and precisions
Alexis211 Oct 5, 2023
1c13135
admin api: remove broken GET /v0/key router rule
Alexis211 Oct 5, 2023
0c431b0
admin api: increased compatibility for v0/ endpoints
Alexis211 Oct 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions .drone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ steps:
image: nixpkgs/nix:nixos-22.05
environment:
GARAGE_TEST_INTEGRATION_EXE: result-bin/bin/garage
GARAGE_TEST_INTEGRATION_PATH: tmp-garage-integration
commands:
- nix-build --no-build-output --attr clippy.amd64 --argstr git_version ${DRONE_TAG:-$DRONE_COMMIT}
- nix-build --no-build-output --attr test.amd64
Expand All @@ -31,8 +32,9 @@ steps:
- ./result/bin/garage_util-*
- ./result/bin/garage_web-*
- ./result/bin/garage-*
- ./result/bin/integration-*
- ./result/bin/integration-* || (cat tmp-garage-integration/stderr.log; false)
- rm result
- rm -rv tmp-garage-integration

- name: integration tests
image: nixpkgs/nix:nixos-22.05
Expand Down Expand Up @@ -63,11 +65,16 @@ steps:
- nix-build --no-build-output --attr pkgs.amd64.release --argstr git_version ${DRONE_TAG:-$DRONE_COMMIT}
- nix-shell --attr rust --run "./script/not-dynamic.sh result-bin/bin/garage"

- name: integration
- name: integration tests
image: nixpkgs/nix:nixos-22.05
commands:
- nix-shell --attr integration --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)

- name: upgrade tests
image: nixpkgs/nix:nixos-22.05
commands:
- nix-shell --attr integration --run "./script/test-upgrade.sh v0.8.4 x86_64-unknown-linux-musl" || (cat /tmp/garage.log; false)

- name: push static binary
image: nixpkgs/nix:nixos-22.05
environment:
Expand Down Expand Up @@ -114,11 +121,16 @@ steps:
- nix-build --no-build-output --attr pkgs.i386.release --argstr git_version ${DRONE_TAG:-$DRONE_COMMIT}
- nix-shell --attr rust --run "./script/not-dynamic.sh result-bin/bin/garage"

- name: integration
- name: integration tests
image: nixpkgs/nix:nixos-22.05
commands:
- nix-shell --attr integration --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)

- name: upgrade tests
image: nixpkgs/nix:nixos-22.05
commands:
- nix-shell --attr integration --run "./script/test-upgrade.sh v0.8.4 i686-unknown-linux-musl" || (cat /tmp/garage.log; false)

- name: push static binary
image: nixpkgs/nix:nixos-22.05
environment:
Expand Down Expand Up @@ -283,6 +295,6 @@ trigger:

---
kind: signature
hmac: ac09a5a8c82502f67271f93afa1e1e21ce66383b8e24a6deb26b285cc1c378ba
hmac: 0c4b57eb4b27b7c6a6ff21ab87f0767fe3eb90f5d95d5cbcdccf794e9d2a5d86

...
13 changes: 9 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

193 changes: 99 additions & 94 deletions Cargo.nix

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions doc/book/connect/apps/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Second, we suppose you have created a key and a bucket.
As a reminder, you can create a key for your nextcloud instance as follow:

```bash
garage key new --name nextcloud-key
garage key create nextcloud-key
```

Keep the Key ID and the Secret key in a pad, they will be needed later.
Expand Down Expand Up @@ -139,7 +139,7 @@ a reasonable trade-off for some instances.
Create a key for Peertube:

```bash
garage key new --name peertube-key
garage key create peertube-key
```

Keep the Key ID and the Secret key in a pad, they will be needed later.
Expand Down Expand Up @@ -253,7 +253,7 @@ As such, your Garage cluster should be configured appropriately for good perform
This is the usual Garage setup:

```bash
garage key new --name mastodon-key
garage key create mastodon-key
garage bucket create mastodon-data
garage bucket allow mastodon-data --read --write --key mastodon-key
```
Expand Down Expand Up @@ -379,7 +379,7 @@ Supposing you have a working synapse installation, you can add the module with p
Now create a bucket and a key for your matrix instance (note your Key ID and Secret Key somewhere, they will be needed later):

```bash
garage key new --name matrix-key
garage key create matrix-key
garage bucket create matrix
garage bucket allow matrix --read --write --key matrix-key
```
Expand Down
2 changes: 1 addition & 1 deletion doc/book/connect/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ how to configure this.
Create your key and bucket:

```bash
garage key new my-key
garage key create my-key
garage bucket create backup
garage bucket allow backup --read --write --key my-key
```
Expand Down
4 changes: 2 additions & 2 deletions doc/book/connect/repositories.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ You can configure a different target for each data type (check `[lfs]` and `[att
Let's start by creating a key and a bucket (your key id and secret will be needed later, keep them somewhere):

```bash
garage key new --name gitea-key
garage key create gitea-key
garage bucket create gitea
garage bucket allow gitea --read --write --key gitea-key
```
Expand Down Expand Up @@ -118,7 +118,7 @@ through another support, like a git repository.
As a first step, we will need to create a bucket on Garage and enabling website access on it:

```bash
garage key new --name nix-key
garage key create nix-key
garage bucket create nix.example.com
garage bucket allow nix.example.com --read --write --key nix-key
garage bucket website nix.example.com --allow
Expand Down
11 changes: 3 additions & 8 deletions doc/book/cookbook/real-world.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,11 @@ to store 2 TB of data in total.

- For the metadata storage, Garage does not do checksumming and integrity
verification on its own. If you are afraid of bitrot/data corruption,
put your metadata directory on a BTRFS partition. Otherwise, just use regular
put your metadata directory on a ZFS or BTRFS partition. Otherwise, just use regular
EXT4 or XFS.

- Having a single server with several storage drives is currently not very well
supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)).
For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array.
If you're adventurous, you can try to format each of your disk as
a separate XFS partition, and then run one `garage` daemon per disk drive,
or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge
all your disks in a single union filesystem that spreads load over them.
- Servers with multiple HDDs are supported natively by Garage without resorting
to RAID, see [our dedicated documentation page](@/documentation/operations/multi-hdd.md).

## Get a Docker image

Expand Down
11 changes: 10 additions & 1 deletion doc/book/operations/durability-repairs.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,16 @@ is definitely lost, then there is no other choice than to declare your S3 object
as unrecoverable, and to delete them properly from the data store. This can be done
using the `garage block purge` command.

## Rebalancing data directories

In [multi-HDD setups](@/documentation/operations/multi-hdd.md), to ensure that
data blocks are well balanced between storage locations, you may run a
rebalance operation using `garage repair rebalance`. This is usefull when
adding storage locations or when capacities of the storage locations have been
changed. Once this is finished, Garage will know for each block of a single
possible location where it can be, which can increase access speed. This
operation will also move out all data from locations marked as read-only.


# Metadata operations

Expand All @@ -114,4 +124,3 @@ in your cluster, you can run one of the following repair procedures:

- `garage repair versions`: checks that all versions belong to a non-deleted object, and purges any orphan version
- `garage repair block_refs`: checks that all block references belong to a non-deleted object version, and purges any orphan block reference (this will then allow the blocks to be garbage-collected)

Loading