Skip to content

Feature/fdb 327 fdb compare #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ozaq merged 1 commit into from
Feb 24, 2026
Merged

Feature/fdb 327 fdb compare #165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions docs/fdb/cli_tools/compare.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
fdb-compare
===========

| Compares data of two different FDBs or different data version within one FDB.
| The comparison is accomplished in multiple steps. First all the index keys (typically MARS identifiers) are compared.
| Second, if all keys match, the messages itself are compared by header and data section.
| The message comparison supports different methods, by default all keys are compared.
| The data section can be compared given a numeric tolerance.

Usage
-----

Compares two entire FDBs for MARS id equality:
::
% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> [options...]


Compare different data in the default (production) FDB:
::
% fdb-compare --request1="..." --request2="..." [options...]


Options
-------

+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--config1=string`` | Path to a FDB config. If not passed, FDB5 specific |
| | environment variables like FDB5_CONFIG_FILE will be evaluated. If ``--config1`` |
| | is not provided, ``--request1`` must be provided.i |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--config2=string`` | Optional: Path to a second different FDB config. Default: Same as ``--config1`` |
| | (if passed), otherwise FDB5 specific environment variables are evaluated. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--request1=string`` | Optional: Specialized mars keys to request data from FDB1 or both FDBs |
| | e.g. ``--request1="class=rd,expver=1234"``. |
| | Allows comparing different MARS subtrees if ``--request2`` is specified with |
| | different arguments. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--request2=string`` | Optional: Specialized mars keys to request different data from the second FDB, |
| | e.g. ``--request2="class=od,expver=abcd".`` |
| | Allows comparing different MARS subtrees. Only valid if ``--request1` has ben specified.|
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--scope=string`` | Optional - Values: ``[mars (default)|header-only|all]`` |
| | The FDBs can be compared in different scopes, |
| | 1) ``[mars]`` Mars Metadata only (default), |
| | 2) ``[header-only]`` includes Mars Key comparison |
| | and the comparison of the data headers (e.g. grib headers). |
| | Note: with ``grib-comparison-type=bit-identical``, the full message is |
| | still compared byte-for-byte; ``header-only`` filtering only applies |
| | to ``grib-keys`` comparison. |
| | 3) ``[all]`` includes Mars key and data header comparison but also |
| | the data sections up to a defined floating point tolerance. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--grib-comparison-type=string`` | Optional - Values: ``[hash-keys|grib-keys(default)|bit-identical]`` |
| | Comparing two Grib messages can be done via either (``grib-comparison-type=hash-keys``) |
| | in a bit-identical way with hash-keys for each sections, |
| | via grib keys in the message (``grib-comparison-type=grib-keys`` (default)) |
| | or by directly comparing bit-identical memory segments |
| | (``grib-comparison-type=bit-identical``). |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--fp-tolerance=real`` | Optional: Floating point tolerance for comparison (default=10.0*EPSILON). |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--mars-keys-ignore=string`` | Optional: Format: ``Key1=Value1,Key2=Value2,...KeyN=ValueN`` |
| | All Messages that contain any of the defined key value pairs will be omitted. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--grib-keys-select=string`` | Optional: Format: ``Key1,Key2,Key3...KeyN`` |
| | Only the specified grib keys will be compared. |
| | Only effective with ``grib-comparison-type=grib-keys``. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--grib-keys-ignore=string`` | Optional: Format: ``Key1,Key2,Key3...KeyN`` |
| | The specified key words will be ignored. |
| | Only effective with ``grib-comparison-type=grib-keys``. |
+----------------------------------------+-----------------------------------------------------------------------------------------+
| ``--verbose`` | Optional: Print additional output, |
| | including a progress bar for grib message comparisons |
+----------------------------------------+-----------------------------------------------------------------------------------------+


Examples
--------

Compares two entire FDBs for equality. Using the `--scope=all` options, not only the headers but also the data is compared:
::
% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> --scope=all

Compares a specific part (class=rd,expver=1234) of two different FDBs for equality:
::
% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> --request1="class=rd,expver=1234"


Compares different data versions within one FDB using ``--request1`` and ``--request2``. In that case the only usable ``--grib-comparison-type`` is `grib-keys` (the default option):
::
% fdb-compare --config1=<path/to/config1.yaml> --request1="class=od,expver=abcd" --request2="class=rd,expver=1234" --grib-comparison-type="grib-keys" | tee out

Compares different data versions within the production FDB if executed on ATOS (the FDB5 config will be inferred from the environment):
::
% fdb-compare --request1="class=od,expver=abcd" --request2="class=rd,expver=1234" --grib-comparison-type="grib-keys" | tee out


Exit Codes
----------

The tool exits with code **0** on success (all compared entries match) and **non-zero** on failure
(any mismatch detected or an error occurred). This makes it suitable for use in scripts and CI pipelines.


Notes
-----

When comparing within a single FDB (using ``--config1`` is enough).
Minimum arguments are either ``--config1`` or ``--request1``. This is explicitly checked,
otherwise the default behaviour would result in a comparison of the default FDB (prodfdb on ATOS) with itself.

1 change: 1 addition & 0 deletions docs/fdb/cli_tools/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ Most common tools are the following:
.. _fdb-write: <cli_tools/write>
.. _fdb-list: list.rst
.. _fdb-read: read.rst
.. _fdb-compare: compare.rst
.. _MARS: mars.rst
26 changes: 26 additions & 0 deletions src/fdb5/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,32 @@ ecbuild_add_executable(
remote/fdb-server.cc
LIBS fdb5 )

if ( HAVE_GRIB )
ecbuild_add_executable( TARGET fdb-compare
CONDITION HAVE_FDB_BUILD_TOOLS
SOURCES tools/compare/fdb-compare.cc
tools/compare/common/Types.cc
tools/compare/common/Types.h
tools/compare/common/Util.cc
tools/compare/common/Util.h
tools/compare/mars/Compare.cc
tools/compare/mars/Compare.h
tools/compare/grib/Compare.cc
tools/compare/grib/Compare.h
tools/compare/grib/CompareKeys.cc
tools/compare/grib/CompareKeys.h
tools/compare/grib/CompareHash.cc
tools/compare/grib/CompareHash.h
tools/compare/grib/CompareDatasection.cc
tools/compare/grib/CompareDatasection.h
tools/compare/grib/CompareBitwise.cc
tools/compare/grib/CompareBitwise.h
tools/compare/grib/Utils.cc
tools/compare/grib/Utils.h
INCLUDES ${ECCODES_INCLUDE_DIRS} # Please don't remove me, I am needed
LIBS fdb5 eckit metkit eccodes)
endif()

if ( HAVE_FDB_BUILD_TOOLS )
target_sources( fdb-lock PRIVATE tools/FDBLock.cc tools/FDBLock.h )
target_sources( fdb-unlock PRIVATE tools/FDBLock.cc tools/FDBLock.h )
Expand Down
Loading