diff --git a/.env.demo.bodc b/.env.demo.bodc new file mode 100644 index 0000000..6c0dcda --- /dev/null +++ b/.env.demo.bodc @@ -0,0 +1,14 @@ +# file checker image +FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app +FILECHECKER_IMAGE_TAG=develop +#availables tags at https://github.com/OneArgo/ArgoFormatChecker/tags + develop + latest + +# External directories to mount to the container +FILECHECKER_SPEC_VOLUME=./file_checker_spec +FILECHECKER_INPUT_VOLUME=./demo/inputs/3901945 +FILECHECKER_OUTPUT_VOLUME=./demo/outputs/3901945 + +# Variable specific to floats to check +DAC_NAME=bodc +FILECHECKER_OPTIONS= +FILES_NAMES= diff --git a/.env.demo.coriolis b/.env.demo.coriolis new file mode 100644 index 0000000..78a0a1f --- /dev/null +++ b/.env.demo.coriolis @@ -0,0 +1,14 @@ +# file checker image +FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app +FILECHECKER_IMAGE_TAG=develop +#availables tags at https://github.com/OneArgo/ArgoFormatChecker/tags + develop + latest + +# External directories to mount to the container +FILECHECKER_SPEC_VOLUME=./file_checker_spec +FILECHECKER_INPUT_VOLUME=./demo/inputs/2903996_coriolis +FILECHECKER_OUTPUT_VOLUME=./demo/outputs/2903996_coriolis + +# Variable specific to floats to check +DAC_NAME=coriolis +FILECHECKER_OPTIONS= +FILES_NAMES= diff --git a/.env.docs b/.env.docs index bcf2339..1c793b6 100644 --- a/.env.docs +++ b/.env.docs @@ -13,3 +13,6 @@ DAC_NAME= FILECHECKER_OPTIONS= FILES_NAMES= +# Variable specific to Docker +DOCKER_UID= +DOCKER_GID= diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..5319791 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +demo/outputs/* +.env + +!.gitkeep \ No newline at end of file diff --git a/README.md b/README.md index eea8623..48de6ce 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,10 @@ # Argo NetCDF file format checker + The Argo NetCDF file format checker performs format and content checks on Argo NetCDF files. -The Argo NetCDF format is described in "Argo user's manual" http://dx.doi.org/10.13155/29825 -More information on https://www.argodatamgt.org/Documentation +The Argo NetCDF format is described in "Argo user's manual" +More information on The format checker has two directories: + - file_checker_exec : the java code - file_checker_spec : the rules applied on the NetCDF file by the java code @@ -10,14 +12,18 @@ The rules files implement the Argo vocabularies managed on [NVS vocabulary serve Vocabularies and format changes are managed on [Argo Vocabs Task Team - AVTT GitHub](https://github.com/orgs/OneArgo/projects/4/views/1). **With each release (from 2.9.2) you will find :** + - **file_checker_exec-[version].jar** which consolidates both application's compiled code and all its dependencies into a single executable file. -- source code +- source code ## Run Argo NetCDF file format checker + ### Using file_checker_exec-{version}.jar : + ```bash java -jar file_checker_exec-{version}.jar $OPTION $DAC_NAME $SPEC $OUTPUT_DIR $INPUT_DIR [$FILES_NAMES] ``` + $FILES_NAMES is a list of file's name from the INPUT_DIR. It is optional : without it, all files from INPUT_DIR will be checked. ### Run the application using Docker @@ -25,36 +31,43 @@ $FILES_NAMES is a list of file's name from the INPUT_DIR. It is optional : witho ```bash docker run --rm -v [ABSOLUTE_PATH_TO_SPEC]:/app/file_checker_spec -v [ABSOLUTE_PATH_TO_DATA_FOLDER]:/app/data -v [ABSOLUTE_PATH_TO_OUTPUT_DIR]:/app/results ghcr.io/oneargo/argoformatchecker/app:{TAG} $DAC_NAME ./file_checker_spec ./results ./data [$FILES_NAMES] ``` + You need to mount external directories to the container : [ABSOLUTE_PATH_TO_SPEC] : the file_checker_spec directory path. [ABSOLUTE_PATH_TO_DATA_FOLDER] : Path to directory containing the argo necdf files to be checked. The fileChecker will not seek files in subfolders -[ABSOLUTE_PATH_TO_OUTPUT_DIR] : the directory where xml results files *.filecheck will be created +[ABSOLUTE_PATH_TO_OUTPUT_DIR] : the directory where xml results files \*.filecheck will be created Example : + ```bash docker run --rm -v D:\test_file_checker\file_checker_spec:/app/file_checker_spec -v D:\test_file_checker\datatest:/app/data -v D:\test_file_checker\results:/app/results ghcr.io/oneargo/argoformatchecker/app:develop -no-name-check coriolis ./file_checker_spec ./results ./data ``` ### Run the application using Docker Compose + To facilitate the use of Argo file checker a compose.yaml and .env files are provided : - Prepare your data. - Copy `.env.docs` as `.env` file, and customize variables to configure the file checker for your environment. - Download compose.yaml - Run te service using Docker Compose: - ```bash + +```bash docker compose -f compose.yaml up ``` + or in background : ```bash docker compose -f compose.yaml up -d ``` -Example of an .env file : -``` + +Example of an .env file : + +```text # file checker image FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app FILECHECKER_IMAGE_TAG=develop @@ -68,9 +81,28 @@ FILECHECKER_OUTPUT_VOLUME='D:\test_compose\results' DAC_NAME=bodc FILECHECKER_OPTIONS= FILES_NAMES= +``` + +### Run the application on demonstration files + +Demonstration data are availables to run the application locally easily. +- Clone the repository : + +```bash +git clone https://github.com/OneArgo/ArgoFormatChecker.git ``` -### Test data : + +- Run the script dedicated + +```bash +./run-file-checker-linux.sh +``` + +output files will be generated in `./demo/outputs`. + +### Test data + To test the Argo File Checker, you will find argo data here : https://www.argodatamgt.org/DataAccess.html The Argo File Checker is not yet designed to checking *prof.nc and *Sprof.nc. It checks only TRAJ, META, TECH and PROFILES files. @@ -93,7 +125,6 @@ git clone https://github.com/OneArgo/ArgoFormatChecker.git - Build the application with maven (will requiert Java jdk installed), in file_checker_exec folder : - ```bash cd file_checker_exec ./mvnw clean install @@ -101,18 +132,16 @@ cd file_checker_exec In target folder you will find both original-file_checker_exec and file_checker_exec-[version]. It is this last one to use. - ### build docker image - - Build the application with Docker : -``` +```bash docker build -t filechecker_2.8.14 . ``` - ### Run integration tests + The source code comes with some netcdf test files. You can run the integration tests with this following command : ```bash diff --git a/compose.yaml b/compose.yaml index 73f5397..4a71e93 100644 --- a/compose.yaml +++ b/compose.yaml @@ -2,8 +2,9 @@ services: argo-netcdf-checker: container_name: ArgoFormatChecker image: ${FILECHECKER_IMAGE}:${FILECHECKER_IMAGE_TAG} - env_file: - - .env + user: "${DOCKER_UID:-0}:${DOCKER_GID:-0}" + group_add: + - 1001 volumes: - ${FILECHECKER_SPEC_VOLUME}:/app/file_checker_spec:ro - ${FILECHECKER_INPUT_VOLUME}:/app/data:ro diff --git a/demo/inputs/2903996_coriolis/2903996_Rtraj.nc b/demo/inputs/2903996_coriolis/2903996_Rtraj.nc new file mode 100644 index 0000000..673446b Binary files /dev/null and b/demo/inputs/2903996_coriolis/2903996_Rtraj.nc differ diff --git a/demo/inputs/2903996_coriolis/2903996_meta.nc b/demo/inputs/2903996_coriolis/2903996_meta.nc new file mode 100644 index 0000000..0fe655a Binary files /dev/null and b/demo/inputs/2903996_coriolis/2903996_meta.nc differ diff --git a/demo/inputs/2903996_coriolis/2903996_tech.nc b/demo/inputs/2903996_coriolis/2903996_tech.nc new file mode 100644 index 0000000..f33a6d9 Binary files /dev/null and b/demo/inputs/2903996_coriolis/2903996_tech.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_001.nc b/demo/inputs/2903996_coriolis/R2903996_001.nc new file mode 100644 index 0000000..cf921fc Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_001.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_001D.nc b/demo/inputs/2903996_coriolis/R2903996_001D.nc new file mode 100644 index 0000000..5257bfb Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_001D.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_002.nc b/demo/inputs/2903996_coriolis/R2903996_002.nc new file mode 100644 index 0000000..5e58b03 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_002.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_003.nc b/demo/inputs/2903996_coriolis/R2903996_003.nc new file mode 100644 index 0000000..d32daf0 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_003.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_009.nc b/demo/inputs/2903996_coriolis/R2903996_009.nc new file mode 100644 index 0000000..15f22d4 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_009.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_010.nc b/demo/inputs/2903996_coriolis/R2903996_010.nc new file mode 100644 index 0000000..7942148 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_010.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_011.nc b/demo/inputs/2903996_coriolis/R2903996_011.nc new file mode 100644 index 0000000..b93c874 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_011.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_012.nc b/demo/inputs/2903996_coriolis/R2903996_012.nc new file mode 100644 index 0000000..3998b8d Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_012.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_013.nc b/demo/inputs/2903996_coriolis/R2903996_013.nc new file mode 100644 index 0000000..cd77af8 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_013.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_014.nc b/demo/inputs/2903996_coriolis/R2903996_014.nc new file mode 100644 index 0000000..0a2c116 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_014.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_015.nc b/demo/inputs/2903996_coriolis/R2903996_015.nc new file mode 100644 index 0000000..fa85f55 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_015.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_016.nc b/demo/inputs/2903996_coriolis/R2903996_016.nc new file mode 100644 index 0000000..791b82e Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_016.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_017.nc b/demo/inputs/2903996_coriolis/R2903996_017.nc new file mode 100644 index 0000000..111c59b Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_017.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_018.nc b/demo/inputs/2903996_coriolis/R2903996_018.nc new file mode 100644 index 0000000..5445bfa Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_018.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_019.nc b/demo/inputs/2903996_coriolis/R2903996_019.nc new file mode 100644 index 0000000..543dd8e Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_019.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_020.nc b/demo/inputs/2903996_coriolis/R2903996_020.nc new file mode 100644 index 0000000..cdc8e68 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_020.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_023.nc b/demo/inputs/2903996_coriolis/R2903996_023.nc new file mode 100644 index 0000000..d4ebeb8 Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_023.nc differ diff --git a/demo/inputs/2903996_coriolis/R2903996_024.nc b/demo/inputs/2903996_coriolis/R2903996_024.nc new file mode 100644 index 0000000..f05471f Binary files /dev/null and b/demo/inputs/2903996_coriolis/R2903996_024.nc differ diff --git a/demo/inputs/3901945/3901945_Rtraj.nc b/demo/inputs/3901945/3901945_Rtraj.nc new file mode 100644 index 0000000..e7bca07 Binary files /dev/null and b/demo/inputs/3901945/3901945_Rtraj.nc differ diff --git a/demo/inputs/3901945/3901945_meta.nc b/demo/inputs/3901945/3901945_meta.nc new file mode 100644 index 0000000..f165578 Binary files /dev/null and b/demo/inputs/3901945/3901945_meta.nc differ diff --git a/demo/inputs/3901945/3901945_tech.nc b/demo/inputs/3901945/3901945_tech.nc new file mode 100644 index 0000000..c9006e7 Binary files /dev/null and b/demo/inputs/3901945/3901945_tech.nc differ diff --git a/demo/inputs/3901945/BD3901945_004.nc b/demo/inputs/3901945/BD3901945_004.nc new file mode 100644 index 0000000..a91f8e2 Binary files /dev/null and b/demo/inputs/3901945/BD3901945_004.nc differ diff --git a/demo/inputs/3901945/D3901945_001D.nc b/demo/inputs/3901945/D3901945_001D.nc new file mode 100644 index 0000000..83a84e7 Binary files /dev/null and b/demo/inputs/3901945/D3901945_001D.nc differ diff --git a/demo/inputs/3901945/D3901945_002.nc b/demo/inputs/3901945/D3901945_002.nc new file mode 100644 index 0000000..92e9bb9 Binary files /dev/null and b/demo/inputs/3901945/D3901945_002.nc differ diff --git a/demo/inputs/3901945/D3901945_003.nc b/demo/inputs/3901945/D3901945_003.nc new file mode 100644 index 0000000..bd3dad2 Binary files /dev/null and b/demo/inputs/3901945/D3901945_003.nc differ diff --git a/demo/inputs/3901945/D3901945_004.nc b/demo/inputs/3901945/D3901945_004.nc new file mode 100644 index 0000000..a806522 Binary files /dev/null and b/demo/inputs/3901945/D3901945_004.nc differ diff --git a/demo/inputs/3901945/D3901945_005.nc b/demo/inputs/3901945/D3901945_005.nc new file mode 100644 index 0000000..2a37b6b Binary files /dev/null and b/demo/inputs/3901945/D3901945_005.nc differ diff --git a/demo/inputs/3901945/D3901945_006.nc b/demo/inputs/3901945/D3901945_006.nc new file mode 100644 index 0000000..b5178d5 Binary files /dev/null and b/demo/inputs/3901945/D3901945_006.nc differ diff --git a/demo/inputs/3901945/D3901945_007.nc b/demo/inputs/3901945/D3901945_007.nc new file mode 100644 index 0000000..ca13c85 Binary files /dev/null and b/demo/inputs/3901945/D3901945_007.nc differ diff --git a/demo/inputs/3901945/D3901945_008.nc b/demo/inputs/3901945/D3901945_008.nc new file mode 100644 index 0000000..3c90927 Binary files /dev/null and b/demo/inputs/3901945/D3901945_008.nc differ diff --git a/demo/inputs/3901945/D3901945_009.nc b/demo/inputs/3901945/D3901945_009.nc new file mode 100644 index 0000000..bb6b819 Binary files /dev/null and b/demo/inputs/3901945/D3901945_009.nc differ diff --git a/demo/inputs/3901945/D3901945_010.nc b/demo/inputs/3901945/D3901945_010.nc new file mode 100644 index 0000000..613795f Binary files /dev/null and b/demo/inputs/3901945/D3901945_010.nc differ diff --git a/demo/inputs/3901945/D3901945_011.nc b/demo/inputs/3901945/D3901945_011.nc new file mode 100644 index 0000000..2120d97 Binary files /dev/null and b/demo/inputs/3901945/D3901945_011.nc differ diff --git a/demo/inputs/3901945/D3901945_012.nc b/demo/inputs/3901945/D3901945_012.nc new file mode 100644 index 0000000..af41520 Binary files /dev/null and b/demo/inputs/3901945/D3901945_012.nc differ diff --git a/demo/inputs/3901945/D3901945_013.nc b/demo/inputs/3901945/D3901945_013.nc new file mode 100644 index 0000000..97e4472 Binary files /dev/null and b/demo/inputs/3901945/D3901945_013.nc differ diff --git a/demo/inputs/3901945/D3901945_014.nc b/demo/inputs/3901945/D3901945_014.nc new file mode 100644 index 0000000..6eddf0a Binary files /dev/null and b/demo/inputs/3901945/D3901945_014.nc differ diff --git a/demo/inputs/3901945/D3901945_015.nc b/demo/inputs/3901945/D3901945_015.nc new file mode 100644 index 0000000..2213cb8 Binary files /dev/null and b/demo/inputs/3901945/D3901945_015.nc differ diff --git a/demo/inputs/3901945/D3901945_016.nc b/demo/inputs/3901945/D3901945_016.nc new file mode 100644 index 0000000..d118579 Binary files /dev/null and b/demo/inputs/3901945/D3901945_016.nc differ diff --git a/demo/inputs/3901945/D3901945_017.nc b/demo/inputs/3901945/D3901945_017.nc new file mode 100644 index 0000000..3216249 Binary files /dev/null and b/demo/inputs/3901945/D3901945_017.nc differ diff --git a/demo/inputs/3901945/D3901945_018.nc b/demo/inputs/3901945/D3901945_018.nc new file mode 100644 index 0000000..8836d13 Binary files /dev/null and b/demo/inputs/3901945/D3901945_018.nc differ diff --git a/demo/inputs/3901945/D3901945_019.nc b/demo/inputs/3901945/D3901945_019.nc new file mode 100644 index 0000000..a4dd875 Binary files /dev/null and b/demo/inputs/3901945/D3901945_019.nc differ diff --git a/demo/inputs/3901945/D3901945_020.nc b/demo/inputs/3901945/D3901945_020.nc new file mode 100644 index 0000000..554c052 Binary files /dev/null and b/demo/inputs/3901945/D3901945_020.nc differ diff --git a/demo/inputs/3901945/D3901945_021.nc b/demo/inputs/3901945/D3901945_021.nc new file mode 100644 index 0000000..bd9813a Binary files /dev/null and b/demo/inputs/3901945/D3901945_021.nc differ diff --git a/demo/inputs/3901945/D3901945_022.nc b/demo/inputs/3901945/D3901945_022.nc new file mode 100644 index 0000000..c731c49 Binary files /dev/null and b/demo/inputs/3901945/D3901945_022.nc differ diff --git a/demo/inputs/3901945/D3901945_023.nc b/demo/inputs/3901945/D3901945_023.nc new file mode 100644 index 0000000..3f174ca Binary files /dev/null and b/demo/inputs/3901945/D3901945_023.nc differ diff --git a/run-file-checker-linux.sh b/run-file-checker-linux.sh new file mode 100755 index 0000000..1d28621 --- /dev/null +++ b/run-file-checker-linux.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +# TODO : add when container can run as non root +# export DOCKER_UID=$UID +# export DOCKER_GID=$(id -g $UID) + +docker compose --env-file .env.demo.bodc down +docker compose --env-file .env.demo.bodc up \ No newline at end of file