Medical-Informatics-Platform · KFilippopolitis · Mar 16, 2026 · Mar 6, 2026 · Mar 6, 2026 · Mar 6, 2026
diff --git a/.env.example b/.env.example
@@ -0,0 +1,24 @@
+FLASK_ENV=development
+FLASK_DEBUG=1
+MAX_UPLOAD_SIZE_MB=20
+
+POSTGRES_PORT=5432
+POSTGRES_DB=postgres
+POSTGRES_PASSWORD=test
+
+BACKEND_PORT=8090
+DB_URL=jdbc:postgresql://datacatalogdb:5432/postgres
+DB_USER=postgres
+DB_PASSWORD=test
+
+FRONTEND_PORT=80
+PUBLIC_HOST=http://localhost
+DQT_PORT=8000
+DQT_URL=http://data_quality_tool:8000
+
+AUTHENTICATION=0
+KEYCLOAK_AUTH_URL=https://iam.ebrains.eu/auth/
+KEYCLOAK_REALM=MIP
+KEYCLOAK_CLIENT_ID=datacatalogue
+KEYCLOAK_CLIENT_SECRET=change-me
+KEYCLOAK_SSL_REQUIRED=none
diff --git a/.gitignore b/.gitignore
@@ -6,3 +6,4 @@
 /frontend/.idea/
 /frontend/node_modules/
 /.stored_data/
+/.env
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,19 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+This repository is split by service. `frontend/` contains the Angular 21 UI; application code lives in `frontend/src/app`, assets in `frontend/src/assets`, and specs next to components as `*.spec.ts`. `backend/` is the Spring Boot service with sources in `backend/src/main/java/ebrainsv2/mip/datacatalog`, config in `backend/src/main/resources`, and tests in `backend/src/test/java`. `data_quality_tool/` is a Flask service for Excel and JSON conversion and validation. Root-level `compose.yaml` starts the local stack, and `kubernetes/` contains the Helm chart.
+
+## Build, Test, and Development Commands
+Run the full stack from the repository root with `docker compose up --build`. For frontend-only work: `cd frontend && npm ci && npm start`; use `npm run build` for a production bundle and `npm test -- --watch=false --browsers=ChromeHeadless` for CI-style tests. For backend work: `cd backend && mvn test` or `mvn spring-boot:run`; when running outside Docker, override `DB_URL`, `PUBLIC_HOST`, and `DQT_URL` for host-local services. For the data quality tool: `cd data_quality_tool && poetry install --with dev`, then `poetry run python controller.py`, `poetry run pytest`, and `poetry run black .`.
+
+## Coding Style & Naming Conventions
+Follow `frontend/.editorconfig`: UTF-8, spaces, 2-space indentation, and single quotes in TypeScript. Keep Angular naming consistent with the existing tree, for example `header.component.ts`, `auth.service.ts`, and `user.interface.ts`. In Java, keep package names lowercase, classes in PascalCase, and methods and fields in camelCase. Place backend code in the matching domain package such as `datamodel`, `federation`, or `user`. Python code should stay `black`-formatted and use snake_case for modules, functions, and tests.
+
+## Testing Guidelines
+Add tests with every behavior change. Frontend tests belong beside the component or service they cover and should follow the `*.spec.ts` pattern. Backend tests use JUnit under `backend/src/test/java`; prefer focused coverage for converters, validators, and service logic. Python tests live in `data_quality_tool/tests` and should cover both happy paths and malformed Excel or JSON inputs. No coverage gate is enforced, so reviewers will expect targeted regression tests in changed areas.
+
+## Commit & Pull Request Guidelines
+Recent history favors short, prefixed subjects such as `feat: ...`, `Fix: ...`, and `chore(scope): ...`; keep commits imperative and scoped to one change. Pull requests should summarize the affected module, list the commands you ran, and link the issue or ticket when available. Include screenshots for frontend UI changes and call out any `.env`, Keycloak, Docker, or Helm impact.
+
+## Configuration Tips
+Keep local secrets in `.env` and Kubernetes secrets, never in tracked files. Do not commit generated frontend output from `frontend/dist/` or local Postgres data under `.stored_data/`. When changing database image versions or mount paths, update `compose.yaml` and the Helm files together so local and cluster deployments stay aligned.
diff --git a/README.md b/README.md
@@ -1,134 +1,107 @@
 # DataCatalog
 
-## Overview
+DataCatalog is part of the [Medical Informatics Platform](https://mip.ebrains.eu/) under [EBRAINS](https://www.ebrains.eu/). It manages federations and data models, visualizes common data elements, and supports Excel and JSON import, export, and validation workflows.
 
-DataCatalog (DC) is a component of the [Medical Informatics Platform](https://mip.ebrains.eu/) (MIP) under the [EBRAINS](https://www.ebrains.eu/) initiative. It enables seamless management, visualization, and access to data models and medical conditions.
+## Repository Layout
 
-### Key Features
+- `frontend/`: Angular 21 application served locally on `http://localhost:4200` or through Nginx in Docker
+- `backend/`: Spring Boot API exposed on `http://localhost:8090/services`
+- `data_quality_tool/`: Flask service for Excel and JSON conversion and validation on `http://localhost:8000`
+- `compose.yaml`: local multi-service setup with PostgreSQL `18.3`
+- `kubernetes/`: Helm chart and templates for cluster deployment
 
-- **Data Presentation:** Displays medical conditions linked to datasets within the MIP.
-- **Visualization:** Interactive tools for exploring Common Data Elements (CDEs) and metadata hierarchies.
-- **Data Management:** Create, edit, delete, and version control data models using XLSX and JSON file formats.
-- **Role-Based Access:** Secure access and operation control via defined user roles and Keycloak integration.
+## Quick Start With Docker Compose
 
-## User Roles and Permissions
-
-### Defined Roles
-
-- **DOMAIN\_EXPERT**:
-  - Responsible for creating, editing, deleting, and releasing unreleased data models.
-- **DC\_ADMIN**:
-  - Manages federations and all associated functionalities.
-
-### Permissions Summary
-
-| Feature                | DOMAIN\_EXPERT | DC\_ADMIN |
-|------------------------|----------------|------------|
-| Create/Edit Data Model | Yes            | No         |
-| Delete Data Model      | Yes            | No         |
-| Manage Federations     | No             | Yes        |
-| Release Data Models    | Yes            | No         |
-
-**Note:** Once a data model is released, it becomes immutable and cannot be modified or deleted.
-
-## Features
-
-### Informative Features (No Login Required)
-
-Users can access the following features without logging in:
+### Prerequisites
 
-#### Federations
+- Docker with the Compose plugin
 
-- View all available federations.
-- Access detailed federation information.
-- Navigate to the public URL of each federation.
+### Local Environment
 
-#### Data Models
+The compose setup now has safe local defaults. A ready-to-use dummy file is included at `.env`, and you can copy values from `.env.example` if you need to rebuild it.
 
-- Browse all released data models within federations.
-- View detailed metadata.
-- Visualize metadata hierarchies using a zoomable tidy tree.
-- Access CDE-specific information.
-- Download data models in JSON or XLSX format.
+```env
+FLASK_ENV=development
+FLASK_DEBUG=1
+MAX_UPLOAD_SIZE_MB=20
 
-### Management Features (Login Required)
+POSTGRES_PORT=5432
+POSTGRES_DB=postgres
+POSTGRES_PASSWORD=test
 
-Authenticated users with proper roles gain access to management features, including:
+BACKEND_PORT=8090
+DB_URL=jdbc:postgresql://datacatalogdb:5432/postgres
+DB_USER=postgres
+DB_PASSWORD=test
+FRONTEND_PORT=80
+PUBLIC_HOST=http://localhost
+DQT_PORT=8000
+DQT_URL=http://data_quality_tool:8000
 
-- **Manage Federations:** Full control over federations (DC\_ADMIN).
-- **Manage Data Models:** Create, edit, delete, and release unreleased data models (DOMAIN\_EXPERT).
+AUTHENTICATION=0
+KEYCLOAK_AUTH_URL=https://iam.ebrains.eu/auth/
+KEYCLOAK_REALM=MIP
+KEYCLOAK_CLIENT_ID=datacatalog
+KEYCLOAK_CLIENT_SECRET=change-me
+KEYCLOAK_SSL_REQUIRED=none
+```
 
-## Installation and Setup
+Set `PUBLIC_HOST=http://localhost:4200` if you run the Angular frontend outside Docker. Set `AUTHENTICATION=1` only when valid Keycloak settings are available.
 
-### Prerequisites
+If port `80` is already in use on your machine, set `FRONTEND_PORT=8080` and update `PUBLIC_HOST=http://localhost:8080`.
 
-- Docker
-- Docker Compose
+### Start The Stack
 
-### Installation Steps
+```bash
+docker compose up --build
+```
 
-1. Clone the repository:
-   ```bash
-   git clone https://github.com/Medical-Informatics-Platform/datacatalog.git
-   cd datacatalog
-   ```
+The frontend is available at `http://localhost` (or your configured `FRONTEND_PORT`), the backend at `http://localhost:8090/services`, and the data quality tool at `http://localhost:8000`. PostgreSQL data is persisted under `.stored_data/datacatalogdb`.
 
-2. Create a `.env` file in the project root with the following configurations:
+## Local Service Development
 
-   ```env
-   # Flask Environment
-   FLASK_ENV=development
-   FLASK_DEBUG=1
+### Frontend
 
-   # Database
-   POSTGRES_PASSWORD=test
-
-   # Backend Configuration
-   DB_URL=jdbc:postgresql://datacatalogdb:5432/postgres
-   DB_USER=postgres
-   DB_PASSWORD=test
-   PUBLIC_HOST=http://localhost ; In case of having frontend on development set this to 'http://localhost:4200'.
-   DQT_URL=http://data_quality_tool:8000
+```bash
+cd frontend
+npm ci
+npm start
+```
 
-   # Keycloak Authentication
-   AUTHENTICATION=1
-   KEYCLOAK_AUTH_URL=https://iam.ebrains.eu/auth/
-   KEYCLOAK_REALM=MIP
-   KEYCLOAK_CLIENT_ID=datacatalogue
-   KEYCLOAK_CLIENT_SECRET=your_secret_here
-   KEYCLOAK_SSL_REQUIRED=none
-   ```
+Useful commands:
 
-3. Deploy using Docker Compose:
+- `npm run build`
+- `npm test -- --watch=false --browsers=ChromeHeadless`
 
-   ```bash
-   docker-compose up --build
-   ```
+### Backend
 
-4. Access the application in your browser at [http://localhost](http://localhost).
+```bash
+cd backend
+DB_URL=jdbc:postgresql://localhost:5432/postgres \
+PUBLIC_HOST=http://localhost:4200 \
+DQT_URL=http://localhost:8000 \
+mvn spring-boot:run
+```
 
-## Development Setup
+Use `mvn test` to run backend tests.
 
-For development purposes, you can run the frontend and backend separately:
+### Data Quality Tool
 
-### Frontend
+```bash
+cd data_quality_tool
+poetry install --with dev
+poetry run python controller.py
+```
 
-1. Navigate to the `frontend` folder:
-   ```bash
-   cd frontend
-   ```
+Use `poetry run pytest` for tests and `poetry run black .` for formatting.
 
-2. Install dependencies using npm:
-   ```bash
-   npm install
-   ```
+## Roles
 
-3. Start the development server:
-   ```bash
-   ng serve
-   ```
+- `DC_DOMAIN_EXPERT`: create, update, delete, and release unreleased data models
+- `DC_ADMIN`: manage federations
 
----
+Released data models are immutable.
 
-**DataCatalog** is maintained by the Medical Informatics Platform team under EBRAINS. For more information, visit [EBRAINS](https://www.ebrains.eu/).
+## Kubernetes
 
+Cluster deployment instructions live in [kubernetes/README.md](kubernetes/README.md).
diff --git a/backend/.dockerignore b/backend/.dockerignore
@@ -0,0 +1,3 @@
+.git
+.idea/
+target/
diff --git a/backend/Dockerfile b/backend/Dockerfile
@@ -46,7 +46,7 @@ RUN apk add --no-cache curl
 #######################################################
 # Install dockerize (for config templating)
 #######################################################
-ENV DOCKERIZE_VERSION=v0.6.1
+ENV DOCKERIZE_VERSION=v0.10.1
 RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
     && tar -C /usr/local/bin -xzvf dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
     && rm dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz
@@ -67,4 +67,4 @@ EXPOSE 8090
 #######################################################
 # Healthcheck
 #######################################################
-HEALTHCHECK --start-period=60s CMD curl -f http://localhost:8090/actuator/health || exit 1
+HEALTHCHECK --start-period=60s CMD curl -f http://localhost:8090/services/actuator/health || exit 1
diff --git a/backend/pom.xml b/backend/pom.xml
@@ -6,7 +6,7 @@
 	<parent>
 		<groupId>org.springframework.boot</groupId>
 		<artifactId>spring-boot-starter-parent</artifactId>
-		<version>3.3.2</version>
+		<version>4.0.3</version>
 		<relativePath/> <!-- lookup parent from repository -->
 	</parent>
 
@@ -18,11 +18,11 @@
 
 	<properties>
 		<java.version>21</java.version>
-		<springdoc.version>2.6.0</springdoc.version>
-		<jsonpath.version>2.9.0</jsonpath.version>
-		<gson.version>2.10.1</gson.version>
-		<poi.version>5.2.3</poi.version>
-		<junit.version>5.7.0</junit.version>
+		<springdoc.version>3.0.2</springdoc.version>
+		<jsonpath.version>3.0.0</jsonpath.version>
+		<gson.version>2.13.2</gson.version>
+		<poi.version>5.5.1</poi.version>
+		<junit.version>6.0.3</junit.version>
 	</properties>
 
 	<dependencies>
@@ -36,7 +36,7 @@
 		<dependency>
 			<groupId>jakarta.persistence</groupId>
 			<artifactId>jakarta.persistence-api</artifactId>
-			<version>3.1.0</version> <!-- Ensure a compatible version -->
+			<version>3.2.0</version>
 		</dependency>
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
@@ -93,7 +93,12 @@
 		<dependency>
 			<groupId>org.flywaydb</groupId>
 			<artifactId>flyway-core</artifactId>
-			<version>9.22.3</version>
+			<version>12.0.3</version>
+		</dependency>
+		<dependency>
+			<groupId>org.flywaydb</groupId>
+			<artifactId>flyway-database-postgresql</artifactId>
+			<version>12.0.3</version>
 		</dependency>
 		<dependency>
 			<groupId>org.postgresql</groupId>
@@ -176,7 +181,7 @@
 			<plugin>
 				<groupId>org.asciidoctor</groupId>
 				<artifactId>asciidoctor-maven-plugin</artifactId>
-				<version>2.2.1</version>
+				<version>3.2.0</version>
 				<executions>
 					<execution>
 						<id>generate-docs</id>
-Original file line number
+Diff line change
@@ -0,0 +1,3 @@
+    .git
+    .idea/
+    target/