Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .env.demo.bodc
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# file checker image
FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app
FILECHECKER_IMAGE_TAG=develop
#availables tags at https://github.com/OneArgo/ArgoFormatChecker/tags + develop + latest

# External directories to mount to the container
FILECHECKER_SPEC_VOLUME=./file_checker_spec
FILECHECKER_INPUT_VOLUME=./demo/inputs/3901945
FILECHECKER_OUTPUT_VOLUME=./demo/outputs/3901945

# Variable specific to floats to check
DAC_NAME=bodc
FILECHECKER_OPTIONS=
FILES_NAMES=
14 changes: 14 additions & 0 deletions .env.demo.coriolis
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# file checker image
FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app
FILECHECKER_IMAGE_TAG=develop
#availables tags at https://github.com/OneArgo/ArgoFormatChecker/tags + develop + latest

# External directories to mount to the container
FILECHECKER_SPEC_VOLUME=./file_checker_spec
FILECHECKER_INPUT_VOLUME=./demo/inputs/2903996_coriolis
FILECHECKER_OUTPUT_VOLUME=./demo/outputs/2903996_coriolis

# Variable specific to floats to check
DAC_NAME=coriolis
FILECHECKER_OPTIONS=
FILES_NAMES=
3 changes: 3 additions & 0 deletions .env.docs
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ DAC_NAME=<The dac name for the file to check>
FILECHECKER_OPTIONS=<optional list of options>
FILES_NAMES=<optional list of file names to process>

# Variable specific to Docker
DOCKER_UID=<user uid for output files>
DOCKER_GID=<user git for output files>
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
demo/outputs/*
.env

!.gitkeep
55 changes: 42 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,60 +1,73 @@
# Argo NetCDF file format checker

The Argo NetCDF file format checker performs format and content checks on Argo NetCDF files.
The Argo NetCDF format is described in "Argo user's manual" http://dx.doi.org/10.13155/29825
More information on https://www.argodatamgt.org/Documentation
The Argo NetCDF format is described in "Argo user's manual" <http://dx.doi.org/10.13155/29825>
More information on <https://www.argodatamgt.org/Documentation>
The format checker has two directories:

- file_checker_exec : the java code
- file_checker_spec : the rules applied on the NetCDF file by the java code

The rules files implement the Argo vocabularies managed on [NVS vocabulary server](https://vocab.nerc.ac.uk/search_nvs/).
Vocabularies and format changes are managed on [Argo Vocabs Task Team - AVTT GitHub](https://github.com/orgs/OneArgo/projects/4/views/1).

**With each release (from 2.9.2) you will find :**

- **file_checker_exec-[version].jar** which consolidates both application's compiled code and all its dependencies into a single executable file.
- source code
- source code

## Run Argo NetCDF file format checker

### Using file_checker_exec-{version}.jar :

```bash
java -jar file_checker_exec-{version}.jar $OPTION $DAC_NAME $SPEC $OUTPUT_DIR $INPUT_DIR [$FILES_NAMES]
```

$FILES_NAMES is a list of file's name from the INPUT_DIR. It is optional : without it, all files from INPUT_DIR will be checked.

### Run the application using Docker

```bash
docker run --rm -v [ABSOLUTE_PATH_TO_SPEC]:/app/file_checker_spec -v [ABSOLUTE_PATH_TO_DATA_FOLDER]:/app/data -v [ABSOLUTE_PATH_TO_OUTPUT_DIR]:/app/results ghcr.io/oneargo/argoformatchecker/app:{TAG} $DAC_NAME ./file_checker_spec ./results ./data [$FILES_NAMES]
```

You need to mount external directories to the container :

[ABSOLUTE_PATH_TO_SPEC] : the file_checker_spec directory path.

[ABSOLUTE_PATH_TO_DATA_FOLDER] : Path to directory containing the argo necdf files to be checked. The fileChecker will not seek files in subfolders

[ABSOLUTE_PATH_TO_OUTPUT_DIR] : the directory where xml results files *.filecheck will be created
[ABSOLUTE_PATH_TO_OUTPUT_DIR] : the directory where xml results files \*.filecheck will be created

Example :

```bash
docker run --rm -v D:\test_file_checker\file_checker_spec:/app/file_checker_spec -v D:\test_file_checker\datatest:/app/data -v D:\test_file_checker\results:/app/results ghcr.io/oneargo/argoformatchecker/app:develop -no-name-check coriolis ./file_checker_spec ./results ./data
```

### Run the application using Docker Compose

To facilitate the use of Argo file checker a compose.yaml and .env files are provided :

- Prepare your data.
- Copy `.env.docs` as `.env` file, and customize variables to configure the file checker for your environment.
- Download compose.yaml
- Run te service using Docker Compose:
```bash

```bash
docker compose -f compose.yaml up
```

or in background :

```bash
docker compose -f compose.yaml up -d
```
Example of an .env file :
```

Example of an .env file :

```text
# file checker image
FILECHECKER_IMAGE=ghcr.io/oneargo/argoformatchecker/app
FILECHECKER_IMAGE_TAG=develop
Expand All @@ -68,9 +81,28 @@ FILECHECKER_OUTPUT_VOLUME='D:\test_compose\results'
DAC_NAME=bodc
FILECHECKER_OPTIONS=
FILES_NAMES=
```

### Run the application on demonstration files

Demonstration data are availables to run the application locally easily.

- Clone the repository :

```bash
git clone https://github.com/OneArgo/ArgoFormatChecker.git
```
### Test data :

- Run the script dedicated

```bash
./run-file-checker-linux.sh
```

output files will be generated in `./demo/outputs`.

### Test data

To test the Argo File Checker, you will find argo data here : https://www.argodatamgt.org/DataAccess.html

The Argo File Checker is not yet designed to checking *prof.nc and *Sprof.nc. It checks only TRAJ, META, TECH and PROFILES files.
Expand All @@ -93,26 +125,23 @@ git clone https://github.com/OneArgo/ArgoFormatChecker.git

- Build the application with maven (will requiert Java jdk installed), in file_checker_exec folder :


```bash
cd file_checker_exec
./mvnw clean install
```

In target folder you will find both original-file_checker_exec and file_checker_exec-[version]. It is this last one to use.


### build docker image


- Build the application with Docker :

```
```bash
docker build -t filechecker_2.8.14 .
```


### Run integration tests

The source code comes with some netcdf test files. You can run the integration tests with this following command :

```bash
Expand Down
5 changes: 3 additions & 2 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@ services:
argo-netcdf-checker:
container_name: ArgoFormatChecker
image: ${FILECHECKER_IMAGE}:${FILECHECKER_IMAGE_TAG}
env_file:
- .env
user: "${DOCKER_UID:-0}:${DOCKER_GID:-0}"
group_add:
- 1001
volumes:
- ${FILECHECKER_SPEC_VOLUME}:/app/file_checker_spec:ro
- ${FILECHECKER_INPUT_VOLUME}:/app/data:ro
Expand Down
Binary file added demo/inputs/2903996_coriolis/2903996_Rtraj.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/2903996_meta.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/2903996_tech.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_001.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_001D.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_002.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_003.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_009.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_010.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_011.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_012.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_013.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_014.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_015.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_016.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_017.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_018.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_019.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_020.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_023.nc
Binary file not shown.
Binary file added demo/inputs/2903996_coriolis/R2903996_024.nc
Binary file not shown.
Binary file added demo/inputs/3901945/3901945_Rtraj.nc
Binary file not shown.
Binary file added demo/inputs/3901945/3901945_meta.nc
Binary file not shown.
Binary file added demo/inputs/3901945/3901945_tech.nc
Binary file not shown.
Binary file added demo/inputs/3901945/BD3901945_004.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_001D.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_002.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_003.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_004.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_005.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_006.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_007.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_008.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_009.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_010.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_011.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_012.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_013.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_014.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_015.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_016.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_017.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_018.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_019.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_020.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_021.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_022.nc
Binary file not shown.
Binary file added demo/inputs/3901945/D3901945_023.nc
Binary file not shown.
8 changes: 8 additions & 0 deletions run-file-checker-linux.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash

# TODO : add when container can run as non root
# export DOCKER_UID=$UID
# export DOCKER_GID=$(id -g $UID)

docker compose --env-file .env.demo.bodc down
docker compose --env-file .env.demo.bodc up
Loading