Meeting Reports 2024

[mehdibn]

We share in this discussion our meeting reports with the community.

[mehdibn]

Date : 29/11/23 à 10h

Participants:

• Olivier M.(DGFIP)
• Moncef L.(DGFIP)
• Mehdi B.(DGFIP)
• Idir I. (DGFIP)
• Guillaume M. (DGFIP)

Content :

• Kickoff : begin the design and implementation
• first technology to integrate : jupyterhub
• Participation to the KubeCon Paris 2024
• Update of the github repository with new logo: https://github.com/OKDP
• website domain reserved : okdp.io

[mehdibn]

Date: 06/12/2023

Participants:

• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• begin the study of the first technology to integrate : jupyterhub
• separation between OKDP and its requirements in an extra project like kube, minio, longhorn, etc.
• clone used projects into a separate repository in ordre to keep their history
• start with a github jupyterhub-draft project in the okdp repo then design an okdp project to centralize all integrated technologies
• maintaining Issues/PRs + 2 approves like TDP only for official projects and not draft
• creating discussions for decision-making on current topics
• next step: preparation of local environments via kind, vagrant, etc.

[mehdibn]

Date : 17/01/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Nicolas V. (DGFIP)
• Rémi PERRIER (ENEDIS)

Topics :

• new partner joined okdp meetings : Rémi PERRIER - ENEDIS
• prepare okdp presentation for the TOSIT day (February 9th at BPCE)
• work in progress :
  1. OKDP WebSite update with more details about okdp and its roadmap
  2. build of jupyterhub/jupyterlab images
  3. choice between ArgoCD and FluxCD for CD implementation

[mehdibn]

Date : 28/02/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• Back to TOSIT Data Day Event
• Front OKDP : Angular or React can be used for the front dev and SpringBoot for the back
• Superset: integrate its integration into the roadmap for dataViz
• Go-lang : can be used for transverse automation layers for OKDP
• OKDP Web site up to date with last version by Moncef L.
• Spark: advanced Metastore, Spark-ui and spark-hs integration by Idir I.
• Spark: FluxCD deployment integration by Serge A.
• FluxCD : OKDP integration done by Serge A. with some features in progress : security, multi-tenancy, etc.
• ArgoCD can be used for applicative deployments
• Trino : first integration in progress by Moncef L.

[mehdibn]

Date : 06/03/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Edouard ROUSSEAUX (EDF)

Topics :

• RoadMap (to be published via the web site) updated with integration of superset :

  1. Spark integration
  2. Metastore integration (hive metastore in progress)
  3. Trino integration (design and requirements)
  4. SuperSets integration (to-do)
  5. Sanbox implementation (to-do)

• Implementation of an authentication module for spark history server (demo to be planned for tosit members)

• implementation of github discussions with decisions and technical details

[mehdibn]

Date : 13/03/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Edouard R. (EDF)
• Lionel H. (Orange)
• Pierre S. (Adaltas)

Topics :

• Olivier M.: okdp roadmap to be updated
• Mehdi B.: new project to manage authentication in Spark History and its release management
• Pierre S.: topics to share for new contributions
• Pierre S.: Proposal for other DEV environments with other kubernetes distributions
• Lionel H.: Orange contributions:
  • Spark HS to be tested in the Orange context
  • dev of an UI to list spark apps via APIs kube + rex to be planned
  • share Spark images with the community
• presentation to be planned :
  • March 27th: SparkHistory and okdp projects
  • April 3: build of spark images and Spark History
  • April 4: okdp demo for TOSIT members

[mehdibn]

Date : 20/03/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Nicolas V. (DGFIP)
• Pierre S. (Adaltas)

Topics :

• Olivier M.: confirmation of the following meetings: Thursday April 11 for the demo from 9 a.m. to 10 a.m.
• Olivier M.: DGFIP is looking for alternatives to array (potentially supersets)
• Moncef L.: Update roadmap on the web-site and remove of react
• Moncef L.: implementation of the Helm chart for Spark History Server
• Serge A.: implementation of the Helm chart for Hive Metastore and postgres
• Idir I.: build of spark images
• Mehdi & Chung: finalizing the setup of the test environment with Kind
• Moncef: creation of an OKDP channel on the TOSIT mattermost to discuss current technical subjects

[mehdibn]

Date : 27/03/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Nicolas V. (DGFIP)
• Pierre S. (Adaltas)
• Lionel H. (Orange)

Topics :

• Lionel H.: working on a build container for constant dev env.
• Lionel H.: proposal for a PR on the Spark History authentication project to make the identifier configurable for the JWT token (between sub and email)
• Olivier H.: Participation in the Big Data & AI event with a stand in November 2024.
• Pierre S.: working on multi-tenancy feature in SuperSet
• Serge A.: hive metastore integration and release management
• Serge A.: using FluxCD to deploy requirements (cert manager, metalLb, etc.) in Gitops mode
• Idir I. : build of Spark images
• Moncef L.: build factory
• Mehdi B.: preparation of OKDP slides
• Lionel H.: contribution in the form of PR on the Spark build with: dependency checking + Log4j patch + jre deployment for jdk 11 and 17 + kerberos deployment + tests + observability configuration
• Moncef L.: creation of an OKDP sub-channel for specific discussions

[mehdibn]

Date : 16/05/2024

Participants :

• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Lionel H. (Orange)

Topics :

• Moncef L.: integration of Trino in progress via a Helm chart
• Idir I.: integration of Superset with a focus on security
• Serge A.: implementation of the OKDP sandbox via FluxCD
• Moncef L.: preparation of the June demo in progress
• Moncef L.: integration of the Spark History authentication module at the Helm chart level
• Lionel H.: successful testing and integration of the Spark auth module in the Orange context

[mehdibn]

Date : 22/05/2024

Participants :

• Olivier M. (DGFIP)
• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Stephane B. (Orange)
• Lionel H. (Orange)

Topics :

• Olivier M.: preparation of the presentation for the Data & AI Leaders
• Confirmation of participation in Tosit Day on Thursday 13/06 for an OKDP presentation (Olivier M., Idir I. and Mehdi B. on site and Moncef L. remotely)
• Olivier M.: contact BPCE and Arkea on a potential collaboration on Trino
• Olivier M.: OKDP internal presentation on 07/06 and slides to be reviewed by Mehdi B. and Moncef L.
• Moncef L.: preparation of the demo and progress on current topics: Trino (Moncef L.), Sandbox via FluxCD (Serge A.) and Superset (Idir I.)
• Stephane B. and Lionel H.: integration of the spark HS authentication module and review of the integration with OIDC in order to avoid declaring a wlidcard for SparkUI redirects
• Lionel H.: internal validation for Orange's participation in the OKDP presentation

[mehdibn]

Date : 29/05/2024

Participants :

• Olivier M. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Lionel H. (Orange)
• Edouard R. (EDF)
• Nicolas V. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• Olivier M.: preparation of the presentation for the Data & AI Leaders
• Olivier M.: OKDP internal presentation on 07/06 and slides to be reviewed by Mehdi B. and Moncef L.
• Olivier M.: collaboration with an inter-ministerial group around superset
• Edouard R.: proposal for a demo of EDF's containerized Data platform on 06/19/24
• Olivier M.: ongoing update of the website by Mehdi B. with new events (in progress)
• Lionel H.: creation of a new github issue ( followed by Idir I. ) around Spark History Server and its authentication module in order to make it more generic

[mehdibn]

Date : 05/06/2024

Participants :

• Olivier M. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Lionel H. (Orange)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• Update of the OKDP web site with various events (PR ongoing)
• Presentation of OKDP to Arkéa on Thursday, 30/05 (done)
• Presentation of EDF's proprietary solution on Spark, Wednesday, 19/06/2024.
• Testing of the spark authentication module by new users
• Github issue created by Orange on the Authentication/Ldap (IDP) part on GitHub (ongoing)
• DINUM (BlueHats workshop), Friday, June 7, 2024, from 11:00 to 12:30 (Jupyter, Spark + Spark History authentication module, SandBox, SuperSet, end-to-end build and deployment, etc.)
• TOSiT-Day on the afternoon of 13/06, fill in the framadate (done)
• Roundtable at the Data & AI Leaders on 27 and 28/11 (in progress)
• Packaging SuperSet by Idir, publication and documentation to follow
• Build OKDP: Orange will approach DIODE (DGFiP) to work and provide feedback on the Build part
• Participation of partners (BPCE and Orange) in the Hoche Data & AI, Data Mesh, Data Cloud Platform, Data Fabric & Architectures Meetups

[mehdibn]

Date : 12/06/2024

Participants :

• Olivier M. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• Update the OKDP website with our various events (done)
• DINUM (BlueHats workshop): Friday, June 7, 2024, from 11:00 AM to 12:30 PM (Jupyter, Spark + Spark History authentication module, SandBox, SuperSet, end-to-end build and deployment, etc.). There were 40 participants (done)
• TOSiT-Day on 06/13 in the afternoon.
• SuperSet: Meeting on Tuesday, 06/18, for a new collaboration on the support and REX part
• Presentation of EDF’s proprietary solution on Spark, Wednesday, 06/19/2024.
• TODO : MLOPS: integration of kubeflow, MLFlow, Onyxia, other UIs, etc.

[mehdibn]

Date : 03/07/2024

Participants :

• Serge A. (DGFIP)
• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Lionel H. (Orange)

Topics :

• Serge A.: integration of topolvm with vgs management via helm chart for the sandbox
• Idir I.: create github repo for hive/superset images and their sources
• Lionel H.: propose a solution to secure Docker images (currently being tested on Orange side)

[mehdibn]

Date : 17/07/2024

Participants :

• Nicolas V. (DGFIP)
• Idir I. (DGFIP)
• Serge A. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Lionel H. (Orange)

Topics :

• Idir: split helm charters in github (done)
• Idir: start the architecture choices on the front part with a preference for go-lang for its watcher support
• Serge A.: integration of topolvm for minio/sandbox (done)
• Serge A.: automation of minio resources creation in declarative mode (in progress)

[mehdibn]

Date :24/07/2024

Participants :

• Nicolas V. (DGFIP)
• Idir I. (DGFIP)
• Serge A. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Olivier M. (DGFIP)
• Guillaume M. (DGFIP)

Topics :

• Updating slides with new UI features
• preparation of the internal presentation with the new roadmap
• Discussion with other UX design teams
• Minio integration and automatic resource management in progress
• split Helm charts on github finalized

[mehdibn]

Date :24/07/2024

Participants :

• Olivier M. (DGFIP)
• Guillaume M. (DGFIP)
• Nicolas V. (DGFIP)
• Idir I. (DGFIP)
• Moncef L. (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Lionel H. (Orange)

Topics :

• Idir I.: Split Helm charts (done)
• Idir I.: REST API in Golang + Angular for the UI (in progress)
• Serge A.: MinIO and declarative resource management + RBAC (bucket + policies) (in progress)
• Moncef L.: Take over the Kubernetes clusters topic on private cloud
• Lionel H.: New ticket on the Spark authentication module

[mehdibn]

Date : 08/24 -> 11/09/24

Topics :

• Idir I.: Develop some Angular models for the OKDP UI (in progress)
• Serge A.: MinIO and declarative resource management + RBAC (bucket + policies) (in progress)
• Lionel H.: New ticket on the Spark authentication module
• Idir I. : Choosing the backend for the UI, potentially in GoLang.

[mehdibn]

Date : 18/09/2024

Participants :

• Olivier M. (DGFIP)
• Idir I. (DGFIP)
• Nicolas V (DGFIP)
• Mehdi B. (DGFIP)
• Chung N. (DGFIP)
• Guillaume M. (DGFIP)
• Moncef L. (DGFIP)
• Serge A. (DGFIP)
• Lionel H. (Orange)

Topics :

• Backend choice for the UI:

1. GlassKube: https://glasskube.dev/
2. Timoni: https://timoni.sh/
3. Open Component Model OCM: https://ocm.software/
4. Open Application Model (Microsoft, Alibaba): https://oam.dev/ -> kubevela: https://kubevela.io/
5. kusionstack: https://www.kusionstack.io/

• Criteria:

1. Integration with fluxcd as the first selection criterion
2. A user-friendly frontend
3. Multi-cluster support
4. CNCF incubation

• Serge A.: Minio integration in progress for resource management + LDAP integration
• Lionel H.: Spark issue in progress

[mehdibn]

Date : 25/09/2024

Participants :

• Idir I. (DGFIP)
• Nicolas V (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Moncef L. (DGFIP)
• Serge A. (DGFIP)

Topics :

• Sandbox: Integration of KIND for platforms (not macOS: Linux, Windows) [Kube on containers]
• Sandbox: Integration of Vagrant for internal use [Kube on VMs]
• Backend OKDP UI: Choice in progress and discussions on KubeVela and Crossplane (Open Application Model)
• Backend OKDP UI: Crossplane (Open Application Model)
• MinIO: Resource management (in progress)

[mehdibn]

Date : 02/10/2024

Participants :

• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Moncef L. (DGFIP)
• Serge A. (DGFIP)
• Moncef L. (DGFIP)
• Lionel H. (Orange)

Topics :

• Sandbox: Integration of Minikube for better cross-OS compatibility: Windows, Linux, etc.
• UI: Exclude the KubeVella option for the following reasons:

1. No versioning
2. Packaging does not match
3. No GitOps
4. Decreasing trend on GitHub

• Instead, proceed with the following architecture :
  

1. A REST API development using Go-lang will be necessary between the frontend and the Git layer.
2. The choice of Go-lang is based on the following arguments :

• - Continuity with Kubernetes and existing tools like kubectl
• - Simpler exploitation
• - Modularity
• Unit and integration tests in progress on the Spark authentication module by Orange

[mehdibn]

Date : 09/10/2024

Participants :

• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Chung N. (DGFIP)

Topics :

• SANDBOX: minikube tested and validated in containers and VMs mode
• SANDBOX: integration pending the latest updates in KAD
• TODO: documentation of the different GitHub repositories in progress
• KAD: evolution and refactoring in progress
• UI: adding new features: multiple workspaces by namespace and the concept of deployment template
• UI: preparing a Docker image with all dependencies

[mehdibn]

Date : 16/10/2024

Participants :

• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Chung N. (DGFIP)
• Olivier M. (DGFIP)
• Nicolas V. (DGFIP)
• Moncef L. (DGFIP)

Topics :

• Sandbox: Integration of Minikube into the sandbox validated and integration of the new version of KAD in progress
• OKDP UI: Rest API to be done
• OKDP UI: First version of KAD and the front-end (based on Angular) validated by the team
• Chung N.: Request for a presentation of the work that will take place on Friday, October 18th at 2:00 PM

[mehdibn]

Date : 06/11/2024

Participants :

• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Chung N. (DGFIP)
• Olivier M. (DGFIP)

Topics :

• Olivier M.: update of dgfip OKDP support
• Olivier M.: presentation 01/21/25 BPCE TOSIT Day (preparation of the demo)
• Idir I.: dev of the rest api for the OKDP UI backend
• Serge A.: Refactoring of KAD in progress
• Mehdi B.: integration in progress of the latest version of KAD for the sandbox
• Mehdi B.: preparation of the demo for the Data & AI Leaders Event
• Olivier M.: work internally DGFIP to prepare a dataset for the demos

[mehdibn]

Date : 04/12/2024

Participants :

• Moncef L. (DGFIP)
• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Lionel H.(Orange)
• Olivier M. (DGFIP)

Topics :

Olivier M.

• Presentation at the tech show and feedback on the highly anticipated V1 of OKDP.
• TOSIT Day: presentation and demo of OKDP.

Lionel H.

• Presentation of an AI module (KeyOps) at the TOSIT Day: Khiops under BSD license.
• Spark History PR approved on GCP
• compatibility with Java 8/11 in progress (replacement of Caffeine with Guava).

Moncef L.

• Add the front end to OKDP.
• Prepare the management of Spark jobs.

Mehdi B.

• Upload the slides to the DGFIP share.
• Refactor the sandbox.
• Documentation for publication on the website.

[mehdibn]

Date : 11/12/2024

Participants :

• Moncef L. (DGFIP)
• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Lionel H.(Orange)
• Olivier M. (DGFIP)
• Chung N. (DGFIP)
• Nicolas V. (DGFIP)

Topics :

• Python Libraries: Request to integrate a Python library for internal teams: Streamlit.

• Slides for TOSIT Data Day:

  • Preparation of materials to present strategic progress.
  • Preparation of a demonstration using real data, integrating the Spark operator.
  • Orange's feedback on the Spark History authentication module.

• Website: Update and definition of the roadmap to enhance visibility.

• Iceberg and Polaris: Analysis of opportunities for integrating the two technologies.

• Apache Incubation: Progress in the process of integrating OKDP into the Apache incubation program.

• Sandbox Refactoring: Integration of the latest features related to KAD and configuration refactoring.

• Superset: Emphasis on the importance of a gouv chart for its integration into OKDP.

[mehdibn]

Date : 18/12/2024

Participants :

• Moncef L. (DGFIP)
• Idir I. (DGFIP)
• Mehdi B. (DGFIP)
• Guillaume M. (DGFIP)
• Serge A. (DGFIP)
• Olivier M. (DGFIP)

Topics :

• Streamlit: Validation of the addition of the Streamlit library to Python tools.
• BPCE Day: Preparation underway for the event scheduled on 01/21.
• Roadmap 2025: in progress
• Sandbox: in progress + Decision to resolve DNS via /etc/hosts.
• Superset: Integration of the governance chart into the roadmap to be planned.
• OKDP Server: Implementation in progress.