WPB-22811 migrate conversation codes table to postgres

changelog.d/0-release-notes/WPB-22811

@@ -0,0 +1,5 @@
+Conversation codes can now be migrated to PostgreSQL. For existing installations:
+- Set `postgresMigration.conversationCodes: migration-to-postgresql` in both `galley` and `background-worker`.
+- Run the backfill with `migrateConversationCodes: true`.
+- Wait for `wire_conv_codes_migration_finished` to reach `1.0`.
+- Switch to `postgresMigration.conversationCodes: postgresql` and disable `migrateConversationCodes`.

changelog.d/5-internal/WPB-22811

@@ -1 +1 @@
-Moved CodeStore from galley to subsystems
+Migration of conversation codes from cassandra to postgres (#4959, #4961)

charts/background-worker/templates/configmap.yaml

@@ -90,6 +90,7 @@ data:
     {{- end }}
 
     migrateConversations: {{ .migrateConversations }}
+    migrateConversationCodes: {{ .migrateConversationCodes }}
     migrateConversationsOptions:
 {{toYaml .migrateConversationsOptions | indent 6 }}
 

charts/background-worker/values.yaml

@@ -69,6 +69,10 @@ config:
   migrateConversationsOptions:
     pageSize: 10000
     parallelism: 2
+  # This will start the migration of conversation codes.
+  # It's important to set `settings.postgresMigration.conversationCodes` to `migration-to-postgresql`
+  # before starting the migration.
+  migrateConversationCodes: false
 
   backendNotificationPusher:
     pushBackoffMinWait: 10000 # in microseconds, so 10ms
@@ -87,6 +91,7 @@ config:
   # Controls where conversation data is stored/accessed
   postgresMigration:
     conversation: cassandra
+    conversationCodes: cassandra
 
 secrets:
   {}

charts/galley/values.yaml

@@ -71,6 +71,7 @@ config:
 
   postgresMigration:
     conversation: cassandra
+    conversationCodes: cassandra
   settings:
     httpPoolSize: 128
     maxTeamSize: 10000

docs/src/developer/reference/config-options.md

@@ -1811,79 +1811,94 @@ galley:
   config:
     postgresMigration:
       conversation: postgresql
+      conversationCodes: postgresql
 background-worker:
   config:
     postgresMigration:
       conversation: postgresql
+      conversationCodes: postgresql
     migrateConversations: false
 ```
 
 #### Migration for existing installations
 
-Existing installations should migrate the conversation data to PostgreSQL from
+Existing installations should migrate conversation data to PostgreSQL from
 Cassandra. This is necessary for channel search and management of channels from
 the team-management UI. It is highly recommended to take a backup of the Galley
 Cassandra before triggering the migration.
 
-The migration needs to happen in 3 steps:
+Migrations are independent and can be run separately, in batches, or all at
+once. This is expected, because migrations will be released over time. The
+pattern below applies per store. Use it for `conversation` and
+`conversationCodes` now, and for future stores as they are added.
 
-1. Prepare wire-server for migration.
+**Migration pattern per store(s)**
 
-   This step make sure that wire-server keep working as expected during the
-   migration. To do this deploy wire-server with this config change:
-
-   Configure both `galley` and `background-worker` so that newly created
-   conversations are written to PostgreSQL while existing data still reads from
-   Cassandra:
+1. Prepare the selected store(s) for migration by setting
+   `postgresMigration.<store>` to `migration-to-postgresql`. This enables the
+   migration interpreter for that store, which ensures data is written to
+   PostgreSQL (store-specific details are handled internally).
+   The configuration must be consistent across `galley` and
+   `background-worker`.
 
    ```yaml
    galley:
      config:
        postgresMigration:
          conversation: migration-to-postgresql
+         conversationCodes: migration-to-postgresql
    background-worker:
      config:
        postgresMigration:
          conversation: migration-to-postgresql
+         conversationCodes: migration-to-postgresql
        migrateConversations: false
+       migrateConversationCodes: false
    ```
 
-   This change should restart all the galley pods, any new conversations will
-   now be written to PostgreSQL.
-
-2. Trigger the migration and wait.
+   This change should restart all the galley pods, and new writes will follow
+   the migration interpreter.
 
-   This step will actually carry out the migration. To do this deploy
-   wire-server with this config change:
+2. Run the backfill for the selected store(s) via background-worker.
 
    ```yaml
    background-worker:
      config:
        migrateConversations: true
+       migrateConversationCodes: true
    ```
 
-   This change should restart the background-worker pods. It is recommended to
-   watch the logs and wait for both of these two metrics to report `1.0`:
-   `wire_local_convs_migration_finished` and `wire_user_remote_convs_migration_finished`.
-   This can take a long time depending on number of conversations in the DB.
-
-3. Configure wire-server to only use PostgreSQL for conversations.
+   Wait for the store-specific migration metrics to reach `1.0`. For
+   conversations: `wire_local_convs_migration_finished` and
+   `wire_user_remote_convs_migration_finished`. For conversation codes:
+   `wire_conv_codes_migration_finished`.
 
-   This will be the configuration which must be used from now on for every new
-   release.
+3. Cut over reads and writes to PostgreSQL for the selected store(s). This
+   configuration must be used from now on for every new release.
 
    ```yaml
    galley:
      config:
        postgresMigration:
          conversation: postgresql
+         conversationCodes: postgresql
    background-worker:
      config:
        postgresMigration:
          conversation: postgresql
+         conversationCodes: postgresql
        migrateConversations: false
+       migrateConversationCodes: false
    ```
 
+**How to run migrations independently or in batches**
+
+- To migrate a single store, set only that store’s `postgresMigration.<store>`
+  and `migrate<Store>` flags; leave others unchanged.
+- To migrate a batch, set multiple stores to `migration-to-postgresql` and
+  enable only the matching `migrate<Store>` flags together.
+- To reduce load, run large stores alone and group small stores together.
+
 ## Configure Cells
 
 If Cells integration is enabled, gundeck must be configured with the name of

hack/helm_vars/common.yaml.gotmpl

@@ -13,11 +13,9 @@ dynBackendDomain1: dynamic-backend-1.{{ requiredEnv "NAMESPACE_1" }}.svc.cluster
 dynBackendDomain2: dynamic-backend-2.{{ requiredEnv "NAMESPACE_1" }}.svc.cluster.local
 dynBackendDomain3: dynamic-backend-3.{{ requiredEnv "NAMESPACE_1" }}.svc.cluster.local
 
-{{- if (eq (env "PREFERRED_STORE") "") }}
-conversationStore: cassandra
-{{- else }}
-conversationStore: {{ env "PREFERRED_STORE" }}
-{{- end }}
+{{- $preferredStore := default "cassandra" (env "PREFERRED_STORE") }}
+conversationStore: {{ $preferredStore }}
+conversationCodesStore: {{ $preferredStore }}
 
 {{- if (eq (env "UPLOAD_XML_S3_BASE_URL") "") }}
 uploadXml: {}

hack/helm_vars/wire-server/values.yaml.gotmpl

@@ -306,6 +306,7 @@ galley:
     enableFederation: true # keep in sync with brig.config.enableFederation, cargohold.config.enableFederation and tags.federator!
     postgresMigration:
       conversation: {{ .Values.conversationStore }}
+      conversationCodes: {{ .Values.conversationCodesStore }}
     settings:
       maxConvAndTeamSize: 16
       maxTeamSize: 32
@@ -673,6 +674,7 @@ background-worker:
     federationDomain: integration.example.com
     postgresMigration:
       conversation: {{ .Values.conversationStore }}
+      conversationCodes: {{ .Values.conversationCodesStore }}
     rabbitmq:
       port: 5671
       adminPort: 15671

integration/integration.cabal

@@ -126,7 +126,6 @@ library
     Test.Client
     Test.Connection
     Test.Conversation
-    Test.Conversation.Migration
     Test.Demo
     Test.DNSMock
     Test.DomainVerification
@@ -174,6 +173,9 @@ library
     Test.LegalHold
     Test.Login
     Test.MessageTimer
+    Test.Migration.Conversation
+    Test.Migration.ConversationCodes
+    Test.Migration.Util
     Test.MLS
     Test.MLS.Clients
     Test.MLS.History

integration/test/Test/Conversation/Migration.hs → integration/test/Test/Migration/Conversation.hs

@@ -10,26 +10,22 @@
 -- The tests are from the perspective of mel, a user on the dynamic backend,
 -- called backendM (migrating backend). There are also users called mark and mia
 -- on this backend.
-module Test.Conversation.Migration where
+module Test.Migration.Conversation where
 
 import API.Galley
 import Control.Applicative
-import Control.Concurrent (threadDelay)
 import Control.Monad.Codensity
 import Control.Monad.Reader
 import Data.IntMap (IntMap)
 import qualified Data.IntMap as IntMap
 import qualified Data.IntSet as IntSet
-import Data.Text (Text)
-import qualified Data.Text as Text
-import qualified Data.Text.Encoding as Text
 import GHC.Stack
 import MLS.Util
 import Notifications
 import SetupHelpers hiding (deleteUser)
+import Test.Migration.Util
 import Testlib.Prelude
 import Testlib.ResourcePool
-import Text.Regex.TDFA ((=~))
 import UnliftIO
 
 -- | Our test setup cannot process updates to many MLS convs concurrently, so we
@@ -84,7 +80,9 @@ testMigrationToPostgresMLS = do
 
                 actualConvs `shouldMatchSet` ((convIdToQidObject <$> expectedConvs) <> otherMelConvs)
 
-                when (phase == 3) $ waitForMigration domainM
+                when (phase == 3) $ do
+                  waitForMigration domainM convMigrationFinishedCounterName
+                  waitForMigration domainM userMigrationFinishedCounterName
         runPhase 1
         runPhase 2
         runPhase 3
@@ -191,7 +189,9 @@ testMigrationToPostgresProteus = do
 
                 actualConvs `shouldMatchSet` ((convIdToQidObject <$> expectedConvs) <> otherMelConvs)
 
-                when (phase == 3) $ waitForMigration domainM
+                when (phase == 3) $ do
+                  waitForMigration domainM convMigrationFinishedCounterName
+                  waitForMigration domainM userMigrationFinishedCounterName
         runPhase 1
         runPhase 2
         runPhase 3
@@ -292,17 +292,11 @@ instance Semigroup TestConvList where
         addMelConvs = IntMap.unionWith (<>) l1.addMelConvs l2.addMelConvs
       }
 
-waitForMigration :: (HasCallStack) => String -> App ()
-waitForMigration domainM = do
-  metrics <-
-    getMetrics domainM BackgroundWorker `bindResponse` \resp -> do
-      resp.status `shouldMatchInt` 200
-      pure $ Text.decodeUtf8 resp.body
-  let (_, _, _, convFinishedMatches) :: (Text, Text, Text, [Text]) = (metrics =~ Text.pack "^wire_local_convs_migration_finished\\ ([0-9]+\\.[0-9]+)$")
-  let (_, _, _, userFinishedMatches) :: (Text, Text, Text, [Text]) = (metrics =~ Text.pack "^wire_user_remote_convs_migration_finished\\ ([0-9]+\\.[0-9]+)$")
-  when (convFinishedMatches /= [Text.pack "1.0"] || userFinishedMatches /= [Text.pack "1.0"]) $ do
-    liftIO $ threadDelay 100_000
-    waitForMigration domainM
+convMigrationFinishedCounterName :: String
+convMigrationFinishedCounterName = "^wire_local_convs_migration_finished"
+
+userMigrationFinishedCounterName :: String
+userMigrationFinishedCounterName = "^wire_user_remote_convs_migration_finished"
 
 phase1Overrides, phase2Overrides, phase3Overrides, phase4Overrides, phase5Overrides :: ServiceOverrides
 phase1Overrides =