Fix reproducibility, statistical validity, and code hygiene

.github/workflows/ci.yml

@@ -25,12 +25,31 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
+          pip install -e ".[dev]"
 
       - name: Validate committed data and notebooks
         run: pytest tests/ -q
 
+      - name: Verify group summaries regenerate + reconcile
+        run: python pipeline/build_group_summaries.py
+
+  locked:
+    name: Reproducible install from hashed lock
+    runs-on: ubuntu-latest   # the lock targets x86_64-linux / py3.11
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install from the pinned, hashed lock
+        run: |
+          python -m pip install --upgrade pip
+          pip install --require-hashes -r requirements.lock
+          pip install -e . --no-deps
+      - name: Test against the locked environment
+        run: pytest tests/ -q
+
   security:
     name: Dependency audit (pip-audit)
     runs-on: ubuntu-latest
@@ -43,7 +62,7 @@ jobs:
       - name: Install project + pip-audit
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
+          pip install -e ".[dev]"   # audit the closure the code actually runs in
           pip install pip-audit
       - name: Audit installed dependencies
         run: pip-audit --desc
@@ -68,8 +87,9 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
+          pip install -e ".[dev]"
 
       - name: Execute notebooks
-        run: pytest --nbmake --nbmake-timeout=900 case-study group individual
+        # --nbmake-kernel=python3 overrides any embedded kernel name so a clean
+        # checkout runs regardless of the author's local kernel.
+        run: pytest --nbmake --nbmake-kernel=python3 --nbmake-timeout=900 case-study group individual

.gitignore

@@ -29,3 +29,9 @@ env/
 htmlcov/
 # Original HEIC files (converted to JPG)
 *.HEIC
+
+.claude/
+
+# Generated by pipeline/build_group_summaries.py
+data/group_results/reconstructed/
+mms.egg-info/

.python-version

@@ -1 +1 @@
-3.12
+3.11

CITATION.cff

@@ -15,4 +15,4 @@ keywords:
   - psychometric-testing
   - multimodal-analysis
   - computational-neuropsychology
-license: MIT
+license: CC-BY-4.0

CONTRIBUTING.md

@@ -0,0 +1,104 @@
+# Contributing to Multimodal-Multisensor
+
+Thanks for your interest. This is a small, reproducible research repo: notebooks
+and a shared `mms` analysis package built on top of committed sensor summaries
+from a 10-participant longitudinal study. Contributions that improve clarity,
+reproducibility, or the statistical rigour of the analysis are especially
+welcome. Because the data are special-category health data, please also read
+[DATA_ETHICS.md](DATA_ETHICS.md) before working with anything under `data/`.
+
+## Development setup
+
+The repo targets Python >= 3.11 (CI runs 3.11 and 3.12). Create a virtualenv and
+install the project editable, which also brings in the `mms` package and the
+test toolchain:
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"          # deps + mms + pytest / nbmake / ipykernel
+```
+
+For a byte-for-byte reproducible environment, install from the pinned, hashed
+lockfile instead (generated with `uv pip compile`):
+
+```bash
+pip install -r requirements.lock
+pip install -e . --no-deps       # add the mms package on top, without re-resolving
+```
+
+## Running the tests
+
+The suite validates the committed data and checks every notebook is valid and
+was saved without error outputs. No notebook execution is required:
+
+```bash
+pytest tests/ -q
+```
+
+Optional top-to-bottom notebook execution (slow, and what the manual
+`execute-notebooks` CI job runs) is opt-in:
+
+```bash
+pytest --nbmake --nbmake-kernel=python3 --nbmake-timeout=900 case-study group individual
+```
+
+If you change the analysis code or committed data, also confirm the group
+summaries still regenerate and reconcile:
+
+```bash
+python pipeline/build_group_summaries.py
+```
+
+## Linting
+
+There is no separate linter or formatter configured — keep style consistent with
+the surrounding code. CI does run a `pip-audit` dependency scan; it is advisory
+today, so a flagged CVE won't block a PR, but do check the output.
+
+## Code style
+
+- Clean code, few comments: let names and structure carry the meaning.
+- Put shared loaders and metrics in the `mms` package (`mms.io`, `mms.hrv`,
+  `mms.stats`, `mms.fixation`) and import them from notebooks — don't
+  re-implement analysis inline. New shared logic needs a test in `tests/`.
+- Re-run a notebook top to bottom before committing it, so it lands with clean
+  outputs and no saved errors (the smoke test enforces this).
+- Never commit raw or re-identifiable data. Keep to the de-identified summaries
+  under `data/`; see [DATA_PROVENANCE.md](DATA_PROVENANCE.md) for scope.
+
+## Commit messages
+
+Use [Conventional Commits](https://www.conventionalcommits.org/) with a terse
+description:
+
+```
+feat:     a new analysis, metric, or notebook
+fix:      a correctness fix in mms/, the pipeline, or a notebook
+docs:     README, provenance, or other docs
+test:     tests only
+refactor: restructuring with no behaviour change
+build:    dependencies, packaging, lockfile
+chore:    tooling and housekeeping
+```
+
+For example: `fix: correct ICC(1) confidence interval` or
+`docs: clarify data licence`. Keep one logical change per commit.
+
+## Filing an issue
+
+Open an issue describing what you observed and how to reproduce it. For a numeric
+discrepancy, include the exact command and the values you got versus expected —
+the reliability numbers are reproducible from `mms.stats.icc1`, so a mismatch is
+worth a report.
+
+## Opening a pull request
+
+1. Branch off `main`.
+2. Make your change and keep commits focused.
+3. Run `pytest tests/ -q` locally (and `python pipeline/build_group_summaries.py`
+   if you touched the analysis or data).
+4. Open a pull request against `main`. CI runs the data and notebook validation
+   on Python 3.11 and 3.12 and must be green before merge.
+
+Code (notebooks, `mms/`, scripts) is released under the [MIT license](LICENSE);
+data and figures under the terms in [DATA_LICENSE.md](DATA_LICENSE.md).

DATA_ETHICS.md

@@ -30,10 +30,12 @@ penalty.
   details, dates of birth, or device identifiers).
 - Participants are referred to only by non-reversible pseudonymous codes
   (e.g. `01`, `02`).
-- Recording timestamps are retained for time-series analysis. They carry no
-  location or identity information and, combined only with pseudonymous codes,
-  present low re-identification risk; they can be shifted to relative session
-  time on request.
+- Recording timestamps are retained for time-series analysis. To minimise the
+  residual re-identification risk of absolute appointment dates/times, run
+  [`scripts/deidentify_timestamps.py`](scripts/deidentify_timestamps.py)
+  (`--apply`) to shift each session to a relative origin (`2000-01-01`),
+  preserving within-session alignment while removing the wall-clock date/time.
+  The tool is dry-run by default.
 - If you believe any released field could re-identify a participant, please open
   an issue and it will be removed.
 

DATA_LICENSE.md

@@ -0,0 +1,28 @@
+# Data license
+
+The repository's [`LICENSE`](LICENSE) (MIT) governs the **code** (notebooks, the
+`mms/` package, scripts). MIT is a software license — it speaks of "the
+Software" and grants rights to sell/sublicense — so it does not cleanly govern a
+dataset. The **data** under [`data/`](data/) and the derived figures are
+therefore released separately under:
+
+## Creative Commons Attribution 4.0 International (CC-BY-4.0)
+
+SPDX: `CC-BY-4.0` · Full text: https://creativecommons.org/licenses/by/4.0/legalcode
+
+You are free to share and adapt the data for any purpose, including
+commercially, provided you give appropriate credit (cite via
+[`CITATION.cff`](CITATION.cff)).
+
+### Additional condition — no re-identification
+
+Because this is pseudonymised **special-category health data** from human
+participants (see [`DATA_ETHICS.md`](DATA_ETHICS.md)), one binding condition is
+added on top of CC-BY-4.0:
+
+> You must **not** attempt to re-identify any participant, nor use the data to
+> make decisions about any individual. Report suspected re-identification risks
+> by opening an issue.
+
+This condition reflects the participants' consent terms and does not otherwise
+restrict the CC-BY-4.0 grant.

DATA_PROVENANCE.md

@@ -0,0 +1,91 @@
+# Data Provenance & Reproducibility
+
+This document states, honestly, **what in this repository can be regenerated
+from committed data and what cannot** — the question a reviewer or reuser asks
+first. It also documents the raw→processed→summary pipeline and known data
+issues. It complements [DATA_ETHICS.md](DATA_ETHICS.md) (consent, GDPR,
+de-identification).
+
+## TL;DR reproducibility status
+
+| Layer | Regenerable from this repo? | How |
+|-------|-----------------------------|-----|
+| Headline reliability numbers (ICC) | ✅ Yes | `mms.stats.icc1` on `data/group_results/*.csv` — reproduces the README's 0.22 / 0.45 / 0.61 **and** adds the 95% CIs |
+| Case-study participant's summary row (= **P02**) | ✅ Yes — HRV & pupil exactly, duration to ~2e-5 | `python pipeline/build_group_summaries.py` |
+| Group summary rows for the other 9 participants | ❌ No | Their raw streams were not released (privacy) — see below |
+| Case-study per-session figures (HR, HRV, fixation, clustering) | ✅ Yes | Run the `case-study/` notebooks (data present) |
+
+## The layers
+
+```
+data/case-study/raw/*.txt          # sensor exports (semicolon-delimited), one participant
+        │  (parsing, column rename, derived fixation/duration columns)
+        ▼
+data/case-study/processed/*.csv    # tidy per-session streams: hr_0N, ibi_0N, sed_fix_0N
+        │  (per-session summary statistics — see mms/)
+        ▼
+data/group_results/*.csv           # one row per participant, one column per session
+```
+
+- **raw → processed**: the raw `.txt` files are semicolon-delimited and use
+  dotted names (`gazeDir.x`); the processed `.csv` files are comma-delimited. The
+  plain `sed_*.csv` streams keep raw's `gazeDir.{x,y,z}` names, while
+  `sed_fix_*.csv` renames them to `gaze_{x,y,z}` and adds derived `fixation`,
+  `fixation_id`, `duration`, and `gaze_diff` columns. Only the case-study
+  participant's raw files are present, so only that participant's processed layer
+  is regenerable end-to-end.
+- **processed → summary**: `pipeline/build_group_summaries.py` computes SDNN,
+  pupil-diameter STD and response-duration STD per session. It reproduces the
+  committed **P02** HRV and pupil rows exactly (< 1e-6) and the response-duration
+  row to ~2e-5 — see the reconciliation in
+  `data/group_results/reconstructed/MANIFEST.json` (generated by the pipeline and
+  gitignored, so absent from a fresh clone until you run it).
+
+## Why only one participant is regenerable
+
+The case-study streams correspond to participant **P02** in the group tables.
+This was verified, not assumed: the case-study IBI, pupil and duration data
+reproduce the committed P02 row — HRV and pupil **exactly** (< 1e-6), duration to
+~2e-5. The raw
+streams for P01 and P03–P10 were **not released** — a deliberate,
+privacy-legitimate minimisation choice (only pseudonymised summaries are shared;
+see DATA_ETHICS.md). Consequently their summary rows are provided *as released
+values* and cannot be recomputed from this repository. `build_group_summaries.py`
+does **not** invent them.
+
+## Two recipes: original vs improved
+
+`build_group_summaries.py` reports both, so the choice is explicit:
+
+- **Original recipe** (reproduces the committed values): `ibi.std()` and
+  `pupil.std()` with **no artifact filtering**. This is faithful to how the
+  committed summaries were made, but it lets dropped beats and blink artifacts
+  inflate variance — e.g. P02's Session-2 pupil STD of **1.12** is an artifact
+  spike, ~2.3× the other sessions (more than double).
+- **Improved recipe** (recommended going forward): `mms.hrv.sdnn` applies a
+  300–2000 ms normal-to-normal filter; `mms.fixation.pupil_std` gates on the
+  `pupilQ` quality flag. These are more defensible and remove the S2 spike.
+
+## Known data-integrity issues
+
+- **P01 HRV SDNN, Session 02 == Session 03 (both 65.39).** Byte-identical, a
+  suspected copy-paste artifact; the archived technical report lists a different
+  P01 Session-3 value. Because P01's raw data is not in the repo, **the correct
+  value cannot be recovered here**, so it is *flagged and left unmodified* rather
+  than guessed.
+- **P07 and P08 response-duration STD, Session 01 == 4.409281 (both).** Two
+  different participants share a byte-identical value to 15 decimals — a
+  cross-participant copy-paste artifact. Neither participant's raw data is in the
+  repo, so the true values cannot be recovered; both are flagged and left
+  unmodified. The data owner should restore them from the original source.
+
+`tests/test_data_integrity.py` fails loudly if any **new** duplicate — within a
+participant or across participants — appears; the two known cases above are
+allow-listed so CI stays green while regressions are caught.
+
+## `_modified` psychometric files
+
+`data/case-study/psychometric/*_modified.csv` drop the `Type` column, add a
+parsed `datetime` column, and reformat timestamps relative to the originals.
+They are analysis conveniences derived from the canonical
+`Psychometric_Test_Results_0N.csv` files; the originals are authoritative.

README.md

@@ -49,7 +49,15 @@ Longitudinal study where 10 adults completed standardized psychology tests acros
 
 ## Key findings
 
-How stable are these patterns within a person across sessions? A test–retest reliability check (ICC(1), n = 10, 3 sessions) on the committed summaries gives ICC = 0.22 for HRV SDNN, 0.45 for pupil-dilation variability, and 0.61 for response-duration variability — poor for the physiological measures, moderate at best. So the data do **not** support a strong "stable individual traits" reading: with only 10 participants, these measures look closer to session-to-session fluctuation than to reliable traits. Any stability claim should be read as tentative and underpowered.
+How stable are these patterns within a person across sessions? A test–retest reliability check (ICC(1), n = 10, 3 sessions) on the committed summaries gives:
+
+| Measure | ICC(1) | 95% CI |
+|---------|:------:|:------:|
+| HRV SDNN | 0.22 | **[−0.13, 0.66]** |
+| Pupil-dilation variability | 0.45 | [0.07, 0.79] |
+| Response-duration variability | 0.61 | [0.25, 0.87] |
+
+The confidence intervals are the real story: for HRV the interval **crosses zero**, meaning the data are equally consistent with negative and moderate reliability — i.e. essentially uninformative at n = 10. So the data do **not** support a strong "stable individual traits" reading; these measures look closer to session-to-session fluctuation than to reliable traits. Any stability claim should be read as tentative and underpowered. These numbers are fully reproducible — `mms.stats.icc1` recomputes them (point estimate and CI) from `data/group_results/`.
 
 ![Correlation matrix of HRV SDNN and Pupil Dilation STD across sessions](images/correlation_heatmap_with_values_final.png)
 
@@ -63,6 +71,44 @@ How stable are these patterns within a person across sessions? A test–retest r
 |:---:|:---:|
 | ![PCA K-Means clusters](images/pca_kmeans_clusters.png) | ![Silhouette score vs number of clusters](images/silhouette_score.png) |
 
+## Reproduce this analysis
+
+```bash
+git clone https://github.com/urme-b/Multimodal-Multisensor
+cd Multimodal-Multisensor
+
+python -m venv .venv && source .venv/bin/activate   # Python >= 3.11
+pip install -e ".[dev,notebooks]"                   # deps + mms + tests + jupyterlab
+# Byte-for-byte reproducible install instead (hashed lock):
+#   pip install -r requirements.lock && pip install -e . --no-deps
+
+# Regenerate the group-level summaries from committed data + reconciliation report
+python pipeline/build_group_summaries.py
+
+# Reproduce the headline reliability numbers (ICC + 95% CIs)
+python -c "import mms, pandas as pd; \
+m=mms.io.load_group_summary('HRV_SDNN')[['Session 01','Session 02','Session 03']].to_numpy(float); \
+print(mms.stats.icc1(m))"
+
+# Explore the notebooks
+jupyter lab
+```
+
+**Suggested notebook order** (each folder is independent):
+`case-study/preprocess_raw_to_csv → build_hrv → 1_hr / 1_hrv / 2_fixation → 3_clustering`,
+then `group/group_analysis → group_correlation_matrix → group_correlation_heatmap`.
+
+Shared loaders and metrics live in the [`mms/`](mms/) package (`mms.io`,
+`mms.hrv`, `mms.stats`, `mms.fixation`) so notebooks import tested code instead
+of re-implementing it. Reproducibility scope and known data issues are documented
+in **[DATA_PROVENANCE.md](DATA_PROVENANCE.md)**.
+
+## Reports & publications
+
+- [Multimodal Multisensor Technical Report.pdf](Multimodal%20Multisensor%20Technical%20Report.pdf) — methods and results write-up
+- [Thesis Report.pdf](Thesis%20Report.pdf) — full thesis
+- [CYPSY_Poster.pdf](CYPSY_Poster.pdf) — CyberPsychology conference poster
+
 ## Tech Stack
 
 Python · Jupyter · pandas · NumPy · SciPy · Matplotlib · Seaborn · scikit-learn
@@ -83,4 +129,5 @@ This repository contains human-subjects data (psychometric responses and physiol
 
 ## License
 
-[MIT](LICENSE)
+- **Code** (notebooks, `mms/`, scripts) — [MIT](LICENSE)
+- **Data** (`data/`) and figures — [CC-BY-4.0 with a no-re-identification term](DATA_LICENSE.md) (special-category health data)