10156 fast redeploy maven

.gitignore

@@ -64,3 +64,7 @@ src/main/webapp/resources/images/dataverseproject.png.thumb140
 /conf/keycloak/docker-dev-volumes
 /docker-dev-volumes
 /.vs
+
+# Personal Docker Compose overrides for local development
+# (use with: docker compose -f docker-compose-dev.yml -f docker-compose.override.yml ...)
+/docker-compose.override.yml

doc/release-notes/10156-fast-redeploy.md

@@ -0,0 +1,56 @@
+## Faster Deployments and a Fast Redeploy Workflow for Container-Based Development
+
+### Tables Are Now Only Created When Missing (Faster Deployments for Everyone)
+
+The application no longer runs EclipseLink DDL generation (`eclipselink.ddl-generation=create-tables`) on every
+deployment. Instead, a quick check at startup detects whether any entity tables are missing from the database
+(first boot on an empty database, or newly added entities) and only then creates them - using the same EclipseLink
+schema framework as before, so the semantics are unchanged. On all other (re)deployments, table creation is skipped
+entirely, which noticeably speeds up deployment - in containers, classic installations and development environments
+alike. Incremental schema changes continue to be handled by Flyway migrations on every startup, as before.
+
+### Fast Redeploy for Container-Based Development
+
+Container-based development gains a portable, Maven-based fast-redeploy workflow that works on any
+platform and with any editor or IDE:
+
+```bash
+# Start the dev environment the usual way
+mvn -Pct clean package docker:run
+
+# Make code changes, then hot-redeploy them into the running container in ~10-15 seconds
+mvn -Pfrd package
+
+# Repeat as needed; when finished, stop the environment the usual way
+mvn -Pct docker:stop
+```
+
+`mvn -Pfrd package` incrementally compiles your changes, refreshes the exploded WAR at `target/dataverse` (bind
+mounted into the application container) and makes Payara hot-redeploy it - no container restarts, no image rebuilds.
+Flyway migrations run on every redeploy, and tables for newly added entities are created automatically (see above).
+
+### Metadata Blocks and Solr Schema Updates in the Dev Environment
+
+A new one-shot service `dev_metadata_update` in `docker-compose-dev.yml` keeps a running dev instance in sync with
+the metadata block TSV files in your working tree: on every start of the stack (and on demand via
+`docker start -a dev_metadata_update`) it reloads the standard metadata blocks and
+updates the Solr schema of an already bootstrapped instance. Previously, TSV changes were only picked up when
+bootstrapping a fresh database. The `dev_bootstrap` service now also uses the TSV files from your working tree
+instead of the ones baked into the config baker image. This is backed by a new `update-metadata.sh` script in the
+config baker image.
+
+### Memory Configuration of the Dev Environment
+
+The application container in `docker-compose-dev.yml` now runs with a 6 GiB memory limit (previously 2.5 GiB).
+The old limit could not support the hot-redeploy workflow: each redeploy retains some memory in the running server
+(roughly 150-200 MiB, mostly classloader leftovers), and measurements showed the container being OOM-killed by the
+kernel after only about 3 redeploys at 2.5 GiB. At 6 GiB, well over 20 consecutive redeploys have been verified.
+Since a limit is not a reservation (an idle instance uses about 1.5 GiB), this does not increase the baseline
+footprint. The `docker-compose.override.yml` file name is gitignored and documented as the place for
+personal local overrides.
+
+### Documentation
+
+See the [Fast Redeploy (Command-Line)](https://guides.dataverse.org/en/latest/container/dev-usage.html#dev-fast-redeploy) section in the Container Guide for complete usage instructions and limitations.
+
+See also #10156 and #11961.

doc/sphinx-guides/source/container/configbaker-image.rst

@@ -109,6 +109,10 @@ Scripts
   * - ``solr-driver.sh``
     - Automate updates to a ``schema.xml`` in a Solr Core. Either run in watch mode or as a oneshot script.
       See ``solr-driver.sh -h`` for usage details. Best used as a sidecar or a one-off job.
+  * - ``update-metadata.sh``
+    - Update the standard metadata blocks and the Solr schema of a running, already bootstrapped instance by
+      reloading the metadata block TSV files. Idempotent, best used as a one-off job.
+      See ``update-metadata.sh -h`` for usage details and :ref:`dev-fast-redeploy` for an example use case.
 
 Solr Template
 ^^^^^^^^^^^^^

doc/sphinx-guides/source/container/dev-usage.rst

@@ -121,6 +121,18 @@ this console unopened.
 Note that data is persisted in ``./docker-dev-volumes`` in the root of the Git repo. For a clean start, you should
 remove this directory before running the ``mvn`` commands above.
 
+The application container runs with a memory limit of 6 GiB by default (see ``docker-compose-dev.yml``). This is
+sized for iterative development: every hot redeploy of the application retains some memory in the running server
+(roughly 150-200 MiB each, mostly classloader leftovers - a long-known Jakarta EE issue), and with the previous
+2.5 GiB limit the container got OOM-killed by the kernel after only about 3 redeploys. With 6 GiB, well over 20
+redeploys in a row have been verified. Note that a limit is not a reservation: an idle instance uses about 1.5 GiB,
+and usage only grows toward the limit during long redeploy sessions or heavy load. If you need to tweak this (or
+anything else) for your local setup, put your personal overrides into a (gitignored)
+``docker-compose.override.yml`` file and add it to your Compose commands:
+``docker compose -f docker-compose-dev.yml -f docker-compose.override.yml up``. Note that because we are not using
+the default Compose file name, the override file is *not* picked up automatically - neither by
+``docker compose -f docker-compose-dev.yml ...`` nor by the Maven commands above.
+
 
 .. _dev-logs:
 
@@ -198,10 +210,11 @@ The safest and most reliable way to redeploy code is to stop the running contain
 Safe, but also slowing down the development cycle a lot.
 
 Triggering redeployment of changes using an IDE can greatly improve your feedback loop when changing code.
-You have at least two options:
+You have at least three options:
 
 #. Use builtin features of IDEs or `IDE plugins from Payara <https://docs.payara.fish/community/docs/documentation/ecosystem/ecosystem.html>`_.
 #. Use a paid product like `JRebel <https://www.jrebel.com/>`_.
+#. Use the IDE independent, command-line based :ref:`dev-fast-redeploy` workflow.
 
 The main differences between the first and the second options are support for hot deploys of non-class files and limitations in what the JVM HotswapAgent can do for you.
 Find more details in a `blog article by JRebel <https://www.jrebel.com/blog/java-hotswap-guide>`_.
@@ -408,72 +421,75 @@ The steps below describe options to enable the later in different IDEs.
 Fast Redeploy (Command-Line)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-For developers who prefer command-line workflows over IDE integration, Dataverse provides scripts for fast iterative development without full container rebuilds.
+For developers who prefer command-line workflows over IDE integration, the ``frd`` ("fast redeploy") Maven profile
+enables fast iterative development without full container rebuilds. It is IDE and editor independent and platform
+independent - the only requirements are Maven and the Docker CLI.
 
 **Initial Setup**
 
-Run once per development session:
+Start the dev environment the usual way (see :ref:`dev-run`):
 
 .. code-block:: bash
 
-   ./scripts/dev/dev-start-frd.sh
-
-This command:
+   mvn -Pct clean package docker:run
 
-- Builds the full Dataverse WAR with ``mvn package``
-- Extracts it into ``target/dataverse/`` as an exploded WAR
-- Configures JPA settings for development (``ddl-generation=none``)
-- Starts the dev stack with ``SKIP_DEPLOY=1``
-- Manually deploys the application via ``asadmin``
+The application container deploys the application from the exploded WAR at ``target/dataverse``, which
+``docker-compose-dev.yml`` bind mounts into the container - this is what makes the fast redeploy below possible.
 
 **Iterative Development**
 
 After making code changes, run:
 
 .. code-block:: bash
 
-   ./scripts/dev/dev-frd.sh
+   mvn -Pfrd package
 
-This script:
+This command:
 
-- Compiles Java sources incrementally (``mvn compile``, ~5-10s)
-- Syncs updated classes and webapp resources into the mounted exploded WAR
-- Forces Payara to redeploy the application without restarting containers
-- Key features:
-  - Skips full Maven rebuilds (only compiles changed Java files)
-  - Avoids container restarts (uses hot-redeployment)
-  - Completes in ~12 seconds vs. ~54s for traditional full rebuild workflow (4.5x faster)
-  - Preserves database state between deployments
+- Compiles Java sources incrementally (only changed files)
+- Refreshes the exploded WAR at ``target/dataverse`` with compiled classes and webapp resources (XHTML etc.)
+- Makes Payara hot-redeploy the application inside the running container (via ``docker exec dev_dataverse redeploy.sh``), without restarting any containers
 
-**Typical Workflow**
+A redeploy completes in roughly 10-15 seconds, compared to about a minute for stopping the containers, rebuilding the
+images and starting them again. (Performance varies a lot between machines, treat these numbers as a relative
+comparison only.)
 
-.. code-block:: bash
+Database state is preserved between redeploys. And because a redeployment restarts the application, everything that
+usually happens on application startup happens on every redeploy, too: new Flyway migration scripts under
+``src/main/resources/db/migration`` are applied and tables for newly added JPA entities are created automatically.
 
-   # Start dev environment once
-   ./scripts/dev/dev-start-frd.sh
+**Updating Metadata Blocks**
 
-   # Edit Java or XHTML files...
+Changes to the standard metadata block TSV files under ``scripts/api/data/metadatablocks`` are not part of the
+deployed application. Instead, the one-shot service ``dev_metadata_update`` loads them into the running instance and
+updates the Solr schema accordingly, straight from your working tree. It runs automatically on every start of the
+stack (both the Maven and the Compose variants) and can also be run on demand, without restarting anything:
 
-   # Fast redeploy
-   ./scripts/dev/dev-frd.sh
+.. code-block:: bash
 
-   # Repeat as needed
+   docker start -a dev_metadata_update
 
-   # When finished, stop containers
-   ./scripts/dev/dev-down-frd.sh
+If the changed fields affect data you already created, trigger a reindex with
+``curl http://localhost:8080/api/admin/index`` afterwards.
 
-**Memory Configuration**
+**Stopping**
 
-The fast-redeploy workflow includes ``docker-compose.override.yml`` that increases the memory limit to 8GB 
-(from the default 2GB limit set for GitHub Actions CI) which is insufficient for local Dataverse development. 
-The override file is automatically used by the scripts.
+Stop the environment as usual with ``mvn -Pct docker:stop`` or ``docker compose -f docker-compose-dev.yml down``.
+Your data is kept in ``docker-dev-volumes/`` either way.
 
 **Limitations**
 
-- Does not update dependencies (run full ``mvn package`` + restart if ``pom.xml`` changes)
-- Static resources (CSS, JS) may require browser cache clear
-- For database schema changes, use ``dev-rebuild.sh`` instead
-- Performance timings may vary depending on your hardware configuration
+- Dependency changes in ``pom.xml`` require a full image rebuild and restart: ``mvn -Pct clean package docker:run``.
+- Deleted source and webapp files linger in the exploded WAR until a full rebuild and restart (files are only added
+  and updated, never removed).
+- Hot-redeployment reuses the running JVM, and each redeploy retains some memory (roughly 150-200 MiB, mostly
+  classloader leftovers). With the default 6 GiB memory limit there is room for roughly 25-30 redeploys - if the
+  application becomes slow or unresponsive after a long session, simply restart the stack.
+- The OpenAPI schema is not regenerated on fast redeploys (the one from the last full build is kept).
+
+**Tip**: most of a no-change cycle is Payara's own redeployment (~8-10s); the Maven part is only 1-2 seconds. If you
+want to shave off the JVM startup overhead of Maven itself, the ``frd`` profile works fine with the
+`Maven daemon <https://maven.apache.org/tools/mvnd.html>`_: ``mvnd -Pfrd package``.
 
 **Note**: This workflow complements IDE-based redeployment. Use whichever fits your development style.
 

doc/sphinx-guides/source/developers/sql-upgrade-scripts.rst

@@ -17,7 +17,7 @@ In the past (before adopting Flyway) we used to keep SQL upgrade scripts in ``sc
 How to Determine if You Need to Create a SQL Upgrade Script
 -----------------------------------------------------------
 
-If you are creating a new database table (which maps to an ``@Entity`` in JPA), you do not need to create or update a SQL upgrade script. The reason for this is that we use ``create-tables`` in ``src/main/resources/META-INF/persistence.xml`` so that new tables are automatically created by the app server when you deploy your war file.
+If you are creating a new database table (which maps to an ``@Entity`` in JPA), you do not need to create or update a SQL upgrade script. The reason for this is that missing tables are automatically created at startup when you deploy your war file (see the ``ConditionalSchemaCreator`` session event listener registered in ``src/main/resources/META-INF/persistence.xml``).
 
 If you are doing anything other than creating a new database table such as adding a column to an existing table, you must create or update a SQL upgrade script.
 

docker-compose-dev.yml

@@ -84,8 +84,12 @@ services:
     tmpfs:
       - /dumps:mode=770,size=2052M,uid=1000,gid=1000
       - /tmp:mode=770,size=2052M,uid=1000,gid=1000
-    mem_limit: 2684354560 # 2.5 GiB
-    mem_reservation: 1024m
+    # A reasonable default for local development and CI.
+    # For personal tweaks, use a local (gitignored) docker-compose.override.yml:
+    # https://docs.docker.com/compose/how-tos/multiple-compose-files/merge
+    # Note: keep these as raw byte values - the Docker Maven Plugin cannot parse "6g" etc.
+    mem_limit: 6442450944 # 6 GiB
+    mem_reservation: 2147483648 # 2 GiB
     privileged: false
 
   dev_bootstrap:
@@ -99,6 +103,27 @@ services:
       - dataverse
     volumes:
       - ./docker-dev-volumes/solr/data:/var/solr
+      # Use the metadata block TSV files from your working tree (instead of the ones baked into
+      # the image), so a fresh instance is bootstrapped with your current state.
+      - ./scripts/api/data/metadatablocks:/scripts/bootstrap/base/data/metadatablocks:ro
+
+  # On every "up", this one-shot service refreshes the standard metadata blocks and the Solr
+  # schema of an already bootstrapped instance from the TSV files in your working tree.
+  # (On a fresh instance it is a no-op - dev_bootstrap takes care of the initial load.)
+  # It can also be run on demand against a running stack, without a restart - this works no
+  # matter whether the stack was started via Maven or via Compose:
+  # docker start -a dev_metadata_update
+  dev_metadata_update:
+    container_name: "dev_metadata_update"
+    image: gdcc/configbaker:unstable
+    restart: "no"
+    command:
+      - update-metadata.sh
+    networks:
+      - dataverse
+    volumes:
+      - ./docker-dev-volumes/solr/data:/var/solr
+      - ./scripts/api/data/metadatablocks:/scripts/bootstrap/base/data/metadatablocks:ro
 
   dev_dv_initializer:
     container_name: "dev_dv_initializer"