Conversation
Set default for WipeElementType type_ Co-authored-by: Metin Cakircali <11517091+mcakircali@users.noreply.github.com>
This is of particular relevance when an FDB has been reindexed, as this means there are essentially two catalogues for one store. Extended reindex test to also wipe the FDB.
…few enhancements in wipe tests.
Also run clang format
Tests have had to be further updated to account for the bug reported in FDB-633 being fixed, as this has increased the number of index files written when subtocs are used. format
simondsmart
left a comment
simondsmart
left a comment
There was a problem hiding this comment.
Super happy with this. Only very minor comments, and happy for it to be merged without my re-review after this.
Can you make sure that we include the diagrams that you drew of the overall wipe process/structure, and reference these in an obvious comment at the wipe entry point. There is a lot of implicit overall flow/state knowledge which is needed to work on this code, and we want to give any future developers a heads up...
| [](const auto& pair) { return !pair.second.empty(); }); | ||
|
|
||
| if (doit && unclean && !unsafeWipeAll) { | ||
| eckit::Log::warning() << "Unclean FDB database has the following unknown URIs:" << std::endl; |
There was a problem hiding this comment.
If we are going to error out here, throwing an exception, then this should probably go to Log::error()
|
|
||
| //---------------------------------------------------------------------------------------------------------------------- | ||
|
|
||
| StdDir::StdDir(const eckit::PathName& p) : path_(p), d_(opendir(p.localPath())) { |
There was a problem hiding this comment.
This is copied from elsewhere, no? Is it not possible to put this somewhere sane for common use?
There was a problem hiding this comment.
IIRC FDB's StdDir is confusingly functionally distinct from eckit's StdDir. But I should check.
Adds [CATALOGUE] and [STORE] to the revelevant server logs to allow for better filtering when debugging. Also enable FDB_DEBUG for the server tests, as the serverside logs are otherwise quite sparse and not useful for debugging.
And minor change to remote_api test logging and modify it such that client sleeps while the server flushes the consolidated index.
ChrisspyB
force-pushed
the
wipe-changes
branch
from
665faee to
3560b41
Compare
Also exit wipe early if the catalogue finds nothing to wipe.
danovaro
left a comment
danovaro
left a comment
There was a problem hiding this comment.
I'm very happy with the current state, which is a huge improvement compared to the initial draft. Thanks Chris for your massive effort!
7 participants
Description
Some caveats:
--unsafe-wipe-allwill hit an assert(false) on remote FDB, as I do not want this functionality supported without better testing / some discussion.Contributor Declaration
By opening this pull request, I affirm the following:
🌈🌦️📖🚧 Documentation 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/fdb/pull-requests/PR-184
🌈🌦️📖🚧 Documentation Z3FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/z3fdb/pull-requests/PR-184
🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/fdb/pull-requests/PR-184