From 107904349d018c6b917b0e20d3728507a7904abc Mon Sep 17 00:00:00 2001 From: Christopher Bull <5499680+chrisb13@users.noreply.github.com> Date: Mon, 4 May 2026 14:09:29 +1000 Subject: [PATCH 1/5] still trying to fix conflicts --- documentation/docs/pages/season2.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/documentation/docs/pages/season2.md b/documentation/docs/pages/season2.md index 99ea208..88371a6 100644 --- a/documentation/docs/pages/season2.md +++ b/documentation/docs/pages/season2.md @@ -1,4 +1,13 @@ # Season 2 + +## Navigating MOM6 as a non-oceanographer: a MOM6 mini-app for GPU work. + +Date: 7/05/2026. + +Presenter: Jorge Gálvez Vallejo (@JorgeG94 ). + +Link: [hackathon_mom6_miniapp](https://github.com/JorgeG94/hackathon_mom6_miniapp) + ## Navier Stokes -> stacked shallow water (adiabatic) ## Generalised vertical coordinates ## Vertical Lagrangian remapping From f61f24b6ca6210ae75e1caca1033fc538557020f Mon Sep 17 00:00:00 2001 From: Jorge Luis Galvez Vallejo Date: Wed, 6 May 2026 08:35:47 +1000 Subject: [PATCH 2/5] attempt 1, flow of consciousness approach, core knowledge to dump --- documentation/docs/pages/season2.md | 66 ++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/documentation/docs/pages/season2.md b/documentation/docs/pages/season2.md index 88371a6..24d6f22 100644 --- a/documentation/docs/pages/season2.md +++ b/documentation/docs/pages/season2.md @@ -6,7 +6,71 @@ Date: 7/05/2026. Presenter: Jorge Gálvez Vallejo (@JorgeG94 ). -Link: [hackathon_mom6_miniapp](https://github.com/JorgeG94/hackathon_mom6_miniapp) +Link to repo: [hackathon_mom6_miniapp](https://github.com/JorgeG94/hackathon_mom6_miniapp) + +### Brief Intro + +Joining a new project in a field you did not study can be extremely daunting! But fear not, for +fortune favors the brave. I'd say there are three options here: + +- You are an oceanographer that has little knowledge of high performance computing +- You are a high performance computing perso that has no knowledge of oceanography +- You are a new student and don't know either + +The third one naturally has the most to overcome, but they also have the most time. The first one +is simply "we have to learn how to code", the second one is "I have to learn oceanography". However, +the second one might not need to learn a lot of oceanography since their contributions don't depend +on them understanding the physics of the ocean dynamics. + +I am a stubborn person that doesn't like to write code I don't understand, so I basically like to +create more work for myself. + +### Detailed notes (flow of conciousness) + +As an HPC person whose goal is to either optimize code or port it to different architectures the key +aspect to understand is *what are the bottlenecks?* in the current code. In ocean simulations, to my +annoyance, there is no _one_ routine that dominates 80% of the time. This makes my work harder, usually +one can quickly say "ah yes, the calculation of this physical thing takes most of the time" so one spends +the most time there. Then naturally you go ask the people that work on the codebase about the bottlenecks and +because everyone is using it for a different thing you will hear different things, this is where the oceanography +knowledge becomes useful! + +So the first step as a non-oceanographer was to ask what are the common problems/bottlenecks in the current workflows. Quickly +you start hearing "ah the barotropic solver", "the continuity bit is a bit slow", "the remapping of vertical coords, definetly". +And, of course, you have no idea what this means. + +What is the best way to understand a gigantic codebase with thousands of lines of code? Simplify it! I.e. the creation of a mini-app. +This is also the best thing to do for a hackathon, because you can capture algorithmic complexity, main pain points, and you +end up with a smaller application that is easy to build, easy to verify, and low stakes if you completely break it! + +How does one create a mini-app? Usually you pick the most expensive part of the codebase and strip it for its parts. For MOM6, +as I pointed out earlier, there are no clear bottlenecks it is a bunch of routines/loops that are executed in different bits +of the code that take the most time. So if you want to get a clear picture of the _overall_ code you need to grab the +multiple bits. In the case for this mini-app we chose the vertical and horizontal viscosities, the continuity equation, the +barotropic solver, and the coriolis force. + +How do you strip something to its parts? Well, this is when understanding a bit of the maths helps! At the very bottom, each +piece is following an algorithm to solve whatever equation it is solving. For example, the vertical viscosity is a tridiagonal +solver. You can search and learn _what_ a tridiagonal solver is and how to implement one (say in Python), then based on that +newly acquired knowledge you look at the `MOM_vert_visc.F90` and see that around all the `if` statements, and `subroutine` calls, +there is just a plain-old, vanilla tridiagonal solver at the bottom. Then you just yank it out, and work _backwards_ to obtain +the input data. + +Looking at code where you understand a bit of the maths (especially in Fortran) helps narrow down your knowledge gap, since you +start noticing the `if` statements that add certain parameters, add drags, what do you do at the bottom?, etc. + +The rest is just plain old elbow grease of spending time correlating code to physics. Thankfully there are lots of lectures on +the internet about physical oceanography, many from ARC centres of excellence. I like to listen to them as if they were +podcasts, on my way to work or while I workout (yeah, I'm that kind of guy, helps keep pace if there's no beat). The more times +you hear "geostrophy", "non-hydrostatic Boussinesq equations", "piecewise parabolic method", "Thomas solver", "continuity", +"vorticity", etc. coupled with your curiosity and lack of notetaking, leading to constant googling of "what is geostrophy" leads +to solidifying your knowledge base. + +Why should you invest into learning both? Well, speedups in code can come from new algorithms, and new algorthms can only be +developed if you understand the physics! If you understand the approximations the equations are working under, you can determine +if a certan trick can be applied for a 10-20% speedup gain in the code. + + ## Navier Stokes -> stacked shallow water (adiabatic) ## Generalised vertical coordinates From e4fd57020d585fc93299d4ecdc95e928588b80a6 Mon Sep 17 00:00:00 2001 From: Jorge Luis Galvez Vallejo Date: Wed, 6 May 2026 11:10:04 +1000 Subject: [PATCH 3/5] a more "practical" approach --- documentation/docs/pages/season2.md | 115 ++++++++++++++++++++-------- 1 file changed, 85 insertions(+), 30 deletions(-) diff --git a/documentation/docs/pages/season2.md b/documentation/docs/pages/season2.md index 24d6f22..ec28d91 100644 --- a/documentation/docs/pages/season2.md +++ b/documentation/docs/pages/season2.md @@ -35,40 +35,95 @@ the most time there. Then naturally you go ask the people that work on the codeb because everyone is using it for a different thing you will hear different things, this is where the oceanography knowledge becomes useful! -So the first step as a non-oceanographer was to ask what are the common problems/bottlenecks in the current workflows. Quickly -you start hearing "ah the barotropic solver", "the continuity bit is a bit slow", "the remapping of vertical coords, definetly". -And, of course, you have no idea what this means. - What is the best way to understand a gigantic codebase with thousands of lines of code? Simplify it! I.e. the creation of a mini-app. This is also the best thing to do for a hackathon, because you can capture algorithmic complexity, main pain points, and you end up with a smaller application that is easy to build, easy to verify, and low stakes if you completely break it! -How does one create a mini-app? Usually you pick the most expensive part of the codebase and strip it for its parts. For MOM6, -as I pointed out earlier, there are no clear bottlenecks it is a bunch of routines/loops that are executed in different bits -of the code that take the most time. So if you want to get a clear picture of the _overall_ code you need to grab the -multiple bits. In the case for this mini-app we chose the vertical and horizontal viscosities, the continuity equation, the -barotropic solver, and the coriolis force. - -How do you strip something to its parts? Well, this is when understanding a bit of the maths helps! At the very bottom, each -piece is following an algorithm to solve whatever equation it is solving. For example, the vertical viscosity is a tridiagonal -solver. You can search and learn _what_ a tridiagonal solver is and how to implement one (say in Python), then based on that -newly acquired knowledge you look at the `MOM_vert_visc.F90` and see that around all the `if` statements, and `subroutine` calls, -there is just a plain-old, vanilla tridiagonal solver at the bottom. Then you just yank it out, and work _backwards_ to obtain -the input data. - -Looking at code where you understand a bit of the maths (especially in Fortran) helps narrow down your knowledge gap, since you -start noticing the `if` statements that add certain parameters, add drags, what do you do at the bottom?, etc. - -The rest is just plain old elbow grease of spending time correlating code to physics. Thankfully there are lots of lectures on -the internet about physical oceanography, many from ARC centres of excellence. I like to listen to them as if they were -podcasts, on my way to work or while I workout (yeah, I'm that kind of guy, helps keep pace if there's no beat). The more times -you hear "geostrophy", "non-hydrostatic Boussinesq equations", "piecewise parabolic method", "Thomas solver", "continuity", -"vorticity", etc. coupled with your curiosity and lack of notetaking, leading to constant googling of "what is geostrophy" leads -to solidifying your knowledge base. - -Why should you invest into learning both? Well, speedups in code can come from new algorithms, and new algorthms can only be -developed if you understand the physics! If you understand the approximations the equations are working under, you can determine -if a certan trick can be applied for a 10-20% speedup gain in the code. +How does one create a mini-app? First, ask the oceanographers what are the pain points! They'll quickly tell you about the +"dynamical core", which is where the basic physics are solved, i.e. the fluid mechanics part of the code. This crucial bit +here is to adopt a "how hard can it be?" mentality, be positive. I was quickly pointed to the file `MOM_dyn_split_RK2.F90`, +which contains the mail RK2 timestepping procedure. Fortran is our friend here, there is not a lot of object orientation +shenanigans happening, i.e. what you read is what you get (most of the time). The RK2 step can be summarized as: + +```fortran +!! The split RK2 scheme: +!! 1. Predictor phase: +!! - Compute horizontal viscosity +!! - Compute Coriolis/momentum advection (CAu, CAv) +!! - Apply vertical viscosity +!! - Advance barotropic mode (fast 2D dynamics) +!! - Update layer thicknesses via continuity +!! 2. Corrector phase: +!! - Recompute tendencies with updated state +!! - Apply vertical viscosity +!! - Final barotropic step +!! - Final continuity update +!! +``` + +Don't get too distracted by initialization procedures, those might look ugly but once you pay attention they are simply setting arrays +to something. The above algorithm has done most of the work for us now! We now know what routines we need to look for: + +- Horizontal/Vertical viscosity +- Coriolis +- Barotropic solver +- Continuity + +A key concept here can be extract from MOM, the first M means "modular". Each physics engine should be a module, i.e. it should be able to +be run a standalone. Why? Because if I only want to optimize the Coriolis routines, why should I have to run everything else? So when creating a miniapp +alwyas try to have a separation of concerns. + +We go into the magical place that is `MOM6/src/core` that is where most of the routines we're interested in are. The viscosities are _parametrizations_, i.e. +approximations to extremely annoying physics. If you're feeling adventurous codeup a non-hydrostatic solver that tries to resolve the vertical physics. It is +_ a w f u l_. Here we have: + +``` +MOM_barotropic.F90 +MOM_continuity_PPM.F90 +MOM_CoriolisAdv.F90 +``` + +So then we follow the `MOM_dyn_split_RK2.F90` to find the function names that are the most interesting, for simplicity we will do the `call btstep(..)`. Inside +a MOM6 subroutine, we can think of them like: + +```fortran +subroutine btstep(....) +real :: x +integer :: y +logical :: a + +! indices mapping +is = G%isc ; ie = G%iec ; js = G%jsc ; je = G%jec ; nz = GV%ke + +! bookkeeping (timing, profiling) + +if(OBC) then + +endif + +! halo updates (look for do_group_pass) +call create_group_pass(...) + +! begin real work +... + +end subroutine btstep +``` +This is not the rule everywhere, specially in helper functions and subroutines. But realizing that _most_ do some bookkeeping, accounting at the start, and then +they check for configs, such as `OBCs` etc. will make your life less overwhelming. Once in the magic is just realizing the loops are just loops! + +```fortran + do j=js,je ; do I=is-1,ie + uhbt0(I,j) = uhbt(I,j) - find_uhbt(dt*ubt(I,j), BTCL_u(I,j)) * Idt + enddo ; enddo +``` + +As you go into the routine you'll realize that there are patterns. Certain routine sloop over certain indices, there's `u`s and `v`s and you start quickly +noticing how the algorithms are structured. This is the way, you simply just continue doing this _with a lot of patience_. + +The mini app is a work of hackathon and obsession, trying to compare lots of stuff (serial, openmp, openacc, cuda-fortran) just to get knowledge about the +advantages and disadvantages of each approach. Testing performance without having to worry too much about breaking MOM6. + From 004cc31336e7e7ef4809aefd30f6239e45dfc6f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jorge=20Luis=20G=C3=A1lvez=20Vallejo?= Date: Tue, 12 May 2026 10:23:58 +1000 Subject: [PATCH 4/5] Update season2.md Typos --- documentation/docs/pages/season2.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/documentation/docs/pages/season2.md b/documentation/docs/pages/season2.md index ec28d91..1e86e2e 100644 --- a/documentation/docs/pages/season2.md +++ b/documentation/docs/pages/season2.md @@ -14,7 +14,7 @@ Joining a new project in a field you did not study can be extremely daunting! Bu fortune favors the brave. I'd say there are three options here: - You are an oceanographer that has little knowledge of high performance computing -- You are a high performance computing perso that has no knowledge of oceanography +- You are a high performance computing person that has no knowledge of oceanography - You are a new student and don't know either The third one naturally has the most to overcome, but they also have the most time. The first one @@ -25,7 +25,7 @@ on them understanding the physics of the ocean dynamics. I am a stubborn person that doesn't like to write code I don't understand, so I basically like to create more work for myself. -### Detailed notes (flow of conciousness) +### Detailed notes (flow of consciousness) As an HPC person whose goal is to either optimize code or port it to different architectures the key aspect to understand is *what are the bottlenecks?* in the current code. In ocean simulations, to my @@ -40,9 +40,9 @@ This is also the best thing to do for a hackathon, because you can capture algor end up with a smaller application that is easy to build, easy to verify, and low stakes if you completely break it! How does one create a mini-app? First, ask the oceanographers what are the pain points! They'll quickly tell you about the -"dynamical core", which is where the basic physics are solved, i.e. the fluid mechanics part of the code. This crucial bit +"dynamical core", which is where the basic physics are solved, i.e. the fluid mechanics part of the code. The crucial bit here is to adopt a "how hard can it be?" mentality, be positive. I was quickly pointed to the file `MOM_dyn_split_RK2.F90`, -which contains the mail RK2 timestepping procedure. Fortran is our friend here, there is not a lot of object orientation +which contains the main RK2 timestepping procedure. Fortran is our friend here, there is not a lot of object orientation shenanigans happening, i.e. what you read is what you get (most of the time). The RK2 step can be summarized as: ```fortran @@ -64,17 +64,17 @@ shenanigans happening, i.e. what you read is what you get (most of the time). Th Don't get too distracted by initialization procedures, those might look ugly but once you pay attention they are simply setting arrays to something. The above algorithm has done most of the work for us now! We now know what routines we need to look for: -- Horizontal/Vertical viscosity +- Horizontal/Vertical viscosities - Coriolis - Barotropic solver - Continuity -A key concept here can be extract from MOM, the first M means "modular". Each physics engine should be a module, i.e. it should be able to -be run a standalone. Why? Because if I only want to optimize the Coriolis routines, why should I have to run everything else? So when creating a miniapp -alwyas try to have a separation of concerns. +A key concept here can be extracted from MOM, the first M means "modular". Each physics engine should be a module, i.e. it should be able to +be run as standalone. Why? Because if I only want to optimize the Coriolis routines, why should I have to run everything else? So when creating a mini-app +always try to have a separation of concerns. -We go into the magical place that is `MOM6/src/core` that is where most of the routines we're interested in are. The viscosities are _parametrizations_, i.e. -approximations to extremely annoying physics. If you're feeling adventurous codeup a non-hydrostatic solver that tries to resolve the vertical physics. It is +We go into the magical place that is `MOM6/src/core`, which is where most of the routines we're interested in are. The viscosities are _parametrizations_, i.e. +approximations to extremely annoying physics. If you're feeling adventurous code up a non-hydrostatic solver that tries to resolve the vertical physics. It is _ a w f u l_. Here we have: ``` @@ -109,8 +109,8 @@ call create_group_pass(...) end subroutine btstep ``` -This is not the rule everywhere, specially in helper functions and subroutines. But realizing that _most_ do some bookkeeping, accounting at the start, and then -they check for configs, such as `OBCs` etc. will make your life less overwhelming. Once in the magic is just realizing the loops are just loops! +This is not the rule everywhere, especially in helper functions and subroutines. But realizing that _most_ do some bookkeeping, accounting at the start, and then +they check for configs, such as `OBCs` etc. will make your life less overwhelming. Once in, the magic is just realizing the loops are just loops! ```fortran do j=js,je ; do I=is-1,ie @@ -118,10 +118,10 @@ they check for configs, such as `OBCs` etc. will make your life less overwhelmin enddo ; enddo ``` -As you go into the routine you'll realize that there are patterns. Certain routine sloop over certain indices, there's `u`s and `v`s and you start quickly +As you go into the routine you'll realize that there are patterns. Certain routines loop over certain indices, there's `u`s and `v`s and you start quickly noticing how the algorithms are structured. This is the way, you simply just continue doing this _with a lot of patience_. -The mini app is a work of hackathon and obsession, trying to compare lots of stuff (serial, openmp, openacc, cuda-fortran) just to get knowledge about the +The mini-app is a work of hackathon and obsession, trying to compare lots of stuff (serial, openmp, openacc, cuda-fortran) just to get knowledge about the advantages and disadvantages of each approach. Testing performance without having to worry too much about breaking MOM6. From 577138143d6ac95640395bc437420cd8b8bd0b56 Mon Sep 17 00:00:00 2001 From: Christopher Bull <5499680+chrisb13@users.noreply.github.com> Date: Tue, 12 May 2026 10:29:04 +1000 Subject: [PATCH 5/5] Update documentation/docs/pages/season2.md --- documentation/docs/pages/season2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/docs/pages/season2.md b/documentation/docs/pages/season2.md index 1e86e2e..e89f810 100644 --- a/documentation/docs/pages/season2.md +++ b/documentation/docs/pages/season2.md @@ -4,7 +4,7 @@ Date: 7/05/2026. -Presenter: Jorge Gálvez Vallejo (@JorgeG94 ). +Presenter: Jorge Gálvez Vallejo (@JorgeG94). Link to repo: [hackathon_mom6_miniapp](https://github.com/JorgeG94/hackathon_mom6_miniapp)