README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  dev = "png", 
  dev.args = list(type = "cairo"),
  dpi = 96
)
```

# openeddy

<!-- badges: start -->
<!-- badges: end -->

## Overview

The `openeddy` package is a software infrastructure for eddy covariance
data post-processing. It provides tools for handling data with attached
metadata, setting up quality control scheme and summarizing and plotting
the results. It is aligned with
[REddyProc](https://github.com/bgctw/REddyProc) package that provides
methods for uStar-filtering, gap-filling, and flux-partitioning. Thus
the combined use of `openeddy` and `REddyProc` allows to run the whole
eddy covariance post-processing chain.

All `openeddy` functions are extensively documented. The help file of
the given function with the arguments description, details section, and
examples can be viewed using `?function_name`. To see the most common
`openeddy` use-cases, you can explore the commented interactive
[openeddy tutorials](https://github.com/lsigut/openeddy_tutorials). The
applications are also summarized in the form of a poster:
<https://zenodo.org/record/8159040>. For the example of the whole eddy
covariance post-processing chain see
[EC_workflow](https://github.com/lsigut/EC_workflow).

## Installation

1.  Install devtools package if not available yet.

    ``` r
    install.packages("devtools")
    ```

2.  Install openeddy

    ``` r
    devtools::install_github("lsigut/openeddy")
    ```

## Extended Example

An extended example of applied `openeddy` software infrastructure is
available at:

<https://github.com/lsigut/EC_workflow>

## Short Example

Demonstration of selected generally applicable `openeddy` functions.

```{r}
library(openeddy)
library(REddyProc)
library(ggplot2)
```

Example data from `REddyProc` package do not include statistics
extracted from raw data or quality control (QC) flags. Notice that units
are included.

```{r}
data(Example_DETha98, package = "REddyProc")
str(Example_DETha98[1:4])
DETha98 <- fConvertTimeToPosix(Example_DETha98, 'YDH', 
                               Year = 'Year', Day = 'DoY', Hour = 'Hour')
```

Certain variable names are expected by `openeddy`, thus DETha98 variable
names need to be renamed. Net ecosystem exchange (NEE) is already
filtered based on QC information (qc_NEE) that is not included. Though
flag 2 values can be recreated, respective NEE values are missing and
flag 1 cannot be resolved.

```{r}
rename <- names(DETha98) %in% c("DateTime", "Rg")
names(DETha98)[rename] <- c("timestamp", "GR")
DETha98$qc_NEE <- ifelse(is.na(DETha98$NEE), NA, 0)
```

Application of three general filters is presented.

1.  Low covariance between vertical wind component and CO~2~
    concentration (CO~2~) can be caused by frozen ultrasonic
    anemometer or problems with (CO~2~) measurements. Such cases are
    flagged and saved to `qc_NEE_lowcov` filter.
2.  Runs with repeating values are a sign of malfunctioning equipment
    (`qc_NEE_runs`).
3.  Spikes in low frequency data (`qc_NEE_spikesLF`) cause problems during 
    gap-filling and should be excluded. Since DETha98 was already quality 
    checked, the amount of detected spikes is limited. In order to correctly 
    evaluate spikes, preliminary QC (`qc_NEE_prelim`) that combines available QC
    tests or filters should be produced and used in `despikeLF()`. For 
    simplification, daytime is not distinguished from nighttime here 
    (`despikeLF(..., light = NULL)`).

```{r}
DETha98$qc_NEE_lowcov <- 
  apply_thr(DETha98$NEE, c(-0.01, 0.01), "qc_NEE_lowcov", "between")
summary_QC(DETha98, "qc_NEE_lowcov")
DETha98$qc_NEE_runs <- flag_runs(DETha98$NEE, "qc_NEE_runs")
summary_QC(DETha98, "qc_NEE_runs")
DETha98$qc_NEE_prelim <- 
  combn_QC(DETha98, 
           c("qc_NEE", "qc_NEE_lowcov", "qc_NEE_runs"), 
           "qc_NEE_prelim")
DETha98$qc_NEE_spikesLF <- 
  despikeLF(DETha98, "NEE", "qc_NEE_prelim", "qc_NEE_spikesLF", 
            light = NULL)
summary_QC(DETha98, "qc_NEE_spikesLF")
```

The QC results can be summarized in tabular or graphical form using
`summary_QC()`. It is possible to summarize each filter independently or
summarize the cumulative effect of applied filters. Note that the
fraction of flagged records in this example is negligible as DETha98
data set was already quality checked.

```{r}
summary_QC(DETha98, 
           c("qc_NEE", "qc_NEE_lowcov", "qc_NEE_runs", "qc_NEE_spikesLF"))
summary_QC(DETha98, 
           c("qc_NEE", "qc_NEE_lowcov", "qc_NEE_runs", "qc_NEE_spikesLF"),
           cumul = TRUE, plot = TRUE, flux = "NEE")
```

Although individual QC columns should be stored as they are useful to
distinguish the reason why certain records were excluded, only the
combined QC column (`qc_NEE_composite`) is usually used in further data
processing and analysis.

```{r}
DETha98$qc_NEE_composite <- 
  combn_QC(DETha98, 
           c("qc_NEE", "qc_NEE_lowcov", "qc_NEE_runs", "qc_NEE_spikesLF"), 
           "qc_NEE_composite")
```

Function `plot_eddy()` is useful for visualization of the whole dataset
including flux values, its respective QC flags and the most important
meteorological parameters in monthly and weekly time resolution.
Only a two week subset is presented here to limit the extent of output.

```{r}
sub <- DETha98$DoY >= 29 & DETha98$DoY < 43
DETha98_sub <- ex(DETha98, sub)
plot_eddy(DETha98_sub, "NEE", "qc_NEE", "qc_NEE_composite", 
          skip = "monthly", light = "GR")
```

In addition to actual despiking, `despikeLF()` can be used also for
visualization of the internally computed double-differenced time series
in order to inspect selected 13 days blocks. See section Plotting in
`despikeLF()` help file for further description.

```{r}
despikeLF_plots <- 
  despikeLF(DETha98, "NEE", "qc_NEE_prelim", "qc_NEE_spikesLF", 
            light = NULL, plot = TRUE)$plots
despikeLF_plots$`iter 1`$all$`1998-01-27 - 1998-02-08`
```

## Contact

Ladislav Šigut: [sigut.l\@czechglobe.cz](mailto:sigut.l@czechglobe.cz)

Global Change Research Institute CAS, Bělidla 4a, 603 00 Brno, CZ

## References

Publication describing `openeddy` is not yet available. When describing
the proposed quality control scheme, please refer to it as similar to:

McGloin, R., Sigut, L., Havrankova, K., Dusek, J., Pavelka, M., Sedlak,
P., 2018. Energy balance closure at a variety of ecosystems in Central
Europe with contrasting topographies. Agric. For. Meteorol. 248,
418-431. <https://doi.org/10.1016/j.agrformet.2017.10.003>