From dd195d77caf5789cc4528a63f6b3ddd0f31e0edf Mon Sep 17 00:00:00 2001 From: Jean-Pierre Fouche Date: Thu, 12 Feb 2026 10:08:58 +0000 Subject: [PATCH 1/3] docs/development_standards.md --> 'integration' vs 'unit' Development standards markdown states 'unit' but it should be 'integration' --- docs/development_standards.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development_standards.md b/docs/development_standards.md index 80d48a6ea..e10615911 100644 --- a/docs/development_standards.md +++ b/docs/development_standards.md @@ -207,7 +207,7 @@ As such it is important to ensure the body of the test is signposted with enough In this example, we follow the same structure as if we were writing a unit test but we add more comments as necessary. -> Note that we consider something to be a unit test if there is some I/O bound activity. +> Note that we consider something to be an integration test if there is some I/O bound activity. > Think request over network or database query. Generally, test time can be a good indication From 7aaba853d27fff81666d223529f59b4429999a82 Mon Sep 17 00:00:00 2001 From: Jean-Pierre Fouche Date: Thu, 12 Feb 2026 12:30:10 +0000 Subject: [PATCH 2/3] Cursory Documentation Review Minor typos and technical fixes to docs New GPT-generated arch diagram: this was created by reviewing the importlinter configuration file --- docs/architecture.md | 194 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 000000000..6a8be49bd --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,194 @@ +# System Architecture Overview + +This document provides a high-level overview of the architectural +boundaries and dependency rules enforced across the codebase. + +------------------------------------------------------------------------ + +# 1. High-Level Package Architecture + + ┌──────────────┐ + │ public_api │ + └───────┬──────┘ + │ + │ (via designated interface only) + ▼ + ┌────────────┐ + │ metrics │ + └────────────┘ + + ┌────────────┐ + │ cms │ + └──────┬─────┘ + │ + │ (independent except via interface) + ▼ + metrics + + + ┌────────────┐ + │ ingestion │ + └──────┬─────┘ + │ (via interface only) + ▼ + metrics + + + ┌────────────┐ + │ validation │ + └──────┬─────┘ + │ (via interface only) + ▼ + metrics + + + ┌────────────┐ + │ feedback │ + └────────────┘ + (isolated) + +Key Principles: + +- `metrics` is the core module. +- External modules depend on `metrics` only through designated + interfaces. +- CMS and Metrics are independent except for controlled integration + points. +- Feedback is isolated and behaves like a plugin module. + +------------------------------------------------------------------------ + +# 2. Metrics Internal Layering + +Enforced layered architecture: + + metrics.api + │ + ▼ + metrics.domain + │ + ▼ + metrics.data + +Allowed direction: **top → down**\ +Forbidden direction: **bottom → up** + +Examples: + +- ✔ `api → domain` +- ✔ `domain → data` +- ✖ `data → domain` +- ✖ `domain → api` + +This ensures clean separation of concerns. + +------------------------------------------------------------------------ + +# 3. Metrics Domain Sibling Isolation + +Domain submodules are isolated from one another. + + metrics.domain + │ + ├── bulk_downloads + ├── charts + ├── headlines + ├── tables + ├── trends + └── weather_health_alerts + +No cross-imports between sibling domains: + +- `charts` ✖ `headlines` +- `tables` ✖ `trends` +- `trends` ✖ `weather_health_alerts` + +This prevents hidden coupling and keeps business logic modular. + +------------------------------------------------------------------------ + +# 4. Public API Isolation + +The Public API module has strict boundaries. + + public_api + │ + │ (interface only) + ▼ + metrics.interface + +Forbidden direct dependencies: + +- metrics.api +- metrics.domain +- metrics.data +- cms +- ingestion +- feedback +- validation + +Only designated interfaces are allowed. + +------------------------------------------------------------------------ + +# 5. Ingestion Isolation + + ingestion + │ + │ (interface only) + ▼ + metrics.core_models + +Ingestion cannot depend directly on: + +- metrics.api +- cms +- public_api +- feedback +- validation + +This maintains a clean ingestion boundary. + +------------------------------------------------------------------------ + +# 6. Feedback Module Isolation + +Feedback is intentionally isolated: + + feedback + + (no outward dependencies) + +The rest of the system: + +- Cannot import `feedback` +- Except for explicit URL wiring allowances + +This allows feedback to behave like a replaceable subsystem. + +------------------------------------------------------------------------ + +# 7. Source vs Test Boundary + +Production code must not depend on test code. + + Source Code ✖ tests + +Prevents accidental coupling to test utilities. + +------------------------------------------------------------------------ + +# Architectural Summary + +This architecture represents: + +- A modular monolith +- Strict layered architecture within the core +- Interface-based integration +- Strong dependency direction enforcement +- Domain isolation +- Plugin-style subsystems +- Architectural contracts enforced via Import Linter + +The result is a disciplined, maintainable, and evolution-friendly +system. From a97b9e973f82ec1c180da413c33c53e1ab987ff1 Mon Sep 17 00:00:00 2001 From: Jean-Pierre Fouche Date: Thu, 12 Feb 2026 12:33:47 +0000 Subject: [PATCH 3/3] Cursory Documentation review * minor typos * new architecture.md - using importlinter config in pyproject.toml - generated using ChatGPT --- docs/development_standards.md | 2 +- docs/ingestion_design_guide.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development_standards.md b/docs/development_standards.md index e10615911..3dc7c2014 100644 --- a/docs/development_standards.md +++ b/docs/development_standards.md @@ -320,7 +320,7 @@ to use key word arguments instead of positional arguments. We shall avoid the use of `assert` statements in source code. -For Python processes, the application of th `assert` keyword can be easily switched off in production +For Python processes, the application of the `assert` keyword can be easily switched off in production with the [PYTHONOPTIMIZE](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONOPTIMIZE) flag. This is usually done for compilation optimization purposes. diff --git a/docs/ingestion_design_guide.md b/docs/ingestion_design_guide.md index f00ef6760..774c55e38 100644 --- a/docs/ingestion_design_guide.md +++ b/docs/ingestion_design_guide.md @@ -61,7 +61,7 @@ to the `processed/` or the `failed/` folder within the s3 bucket, depending on t It should be noted that the `source_data/` directory at the root level of the project contains a number of source data files which are used to populate databases in development environments. -These files are ran through and handled by the same `Consumer` class +These files are run through and handled by the same `Consumer` class as the ingestion lambda function in deployed environments. The only exception being, we consume the test dataset with multiprocessing.