feat: start mgo with arg to use CA cert by wallyworld · Pull Request #21563 · juju/juju

wallyworld · 2026-01-12T05:42:27Z

Mongo 4.4.30 requires client connections be properly authenticated.
Some enablement work is need to make juju compatibile with this mongo release.
The mongo service needs to be started with a CA cert to use and client connections need to be configured with certificates.

We'll use the controller's CA cert when starting mongo.
For client connections, issue a new certificate off the CA cert. The client cert is valid for 15 minutes. Once the connection is created, it stays open. If a new connection is needed and the cert has expired, a new one is generated.

Also update the mongo args to use the new "tls" terminology, but only where the mongo version is 4.4. Juju 2.8 uses an older deb mongo version and early 2.9 uses 4.0 and the newer "tls" args were introduced in mongodb 4.1. So we need to stick with the old "ssl" args for those mongo versions.

For pre merge tests, the seed-repository target needed to optionally push the old juju-db 4.4 image so the older juju could bootstrap.

Drive by: remove obsolete client pre-merge tests.

QA steps

juju bootstrap lxd test
juju switch controller
juju enable-ha
juju deploy ubuntu

You can exec into the controller and snap info juju-db shows 4.4.30 is installed.

Check upgrades from a juju version using mongodb 4.4.24

juju bootstrap lxd --agent-version 2.9.53
juju switch controller
juju upgrade-controller --build-agent

The juju-db snap config in /var/snap/juju-db/common/juju-=db.config gets regenerated with the new auth related config which is compatible with mongodb 4.4.24

cat juju-db.config 
# WARNING
# autogenerated by juju on 2026-01-15 23:55:41.193281297 +0000 UTC m=+0.906492080
# manual changes to this file are likely to be overwritten
port = 37017
tlsCertificateKeyFile = /var/snap/juju-db/common/server.pem
tlsCertificateKeyFilePassword=ignored = true
slowms = 1000
auth = true
oplogSize = 1024
storageEngine = wiredTiger
quiet = true
keyFile = /var/snap/juju-db/common/shared-secret
journal = true
replSet = juju
dbpath = /var/snap/juju-db/common/db
logpath = /var/snap/juju-db/common/logs/mongodb.log
bind_ip_all = true
tlsMode = requireTLS
ipv6 = true
tlsCAFile = /var/snap/juju-db/common/ca.crt
tlsAllowInvalidHostnames = true

On k8s

make microk8s-operator-update
juju bootstrap microk8s test
juju switch controller
juju deploy snappass-test

On older mongo
Get a juju 2.8 cli client

./juju bootstrap lxd
juju switch controller
juju ssh 0
cat /etc/systemd/system/juju-db.service
...
ExecStart=/usr/bin/mongod --auth --bind_ip_all --dbpath '/var/lib/juju/db' --ipv6 --journal --keyFile '/var/lib/juju/shared-secret' --oplogSize 1024 --port 37017 --quiet --replSet juju --slowms 1000 --sslMode requireSSL --sslPEMKeyFile '/var/lib/juju/server.pem' --sslPEMKeyPassword=ignored --storageEngine wiredTiger --syslog

juju upgrade-controller --build-agent
juju ssh 0
cat /etc/systemd/system/juju-db.service
...
ExecStart=/usr/bin/mongod --auth --bind_ip_all --dbpath '/var/lib/juju/db' --ipv6 --journal --keyFile '/var/lib/juju/shared-secret' --oplogSize 1024 --port 37017 --quiet --replSet juju --slowms 1000 --sslAllowInvalidHostnames --sslCAFile '/var/lib/juju/ca.crt' --sslMode requireSSL --sslPEMKeyFile '/var/lib/juju/server.pem' --sslPEMKeyPassword=ignored --storageEngine wiredTiger --syslog

...

Links

Jira card: JUJU-9072

SimonRichardson

Code looks good, though I can't replicated the PR steps because when I attempt to bootstrap I get:

root@juju-92f03c-0:~# systemctl status snap.juju-db.*
● snap.juju-db.logrotate.timer - Timer logrotate for snap application juju-db.logrotate
     Loaded: loaded (/etc/systemd/system/snap.juju-db.logrotate.timer; disabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2026-01-12 09:17:41 UTC; 3min 58s ago
    Trigger: Tue 2026-01-13 00:00:00 UTC; 14h left
   Triggers: ● snap.juju-db.logrotate.service

Jan 12 09:17:41 juju-92f03c-0 systemd[1]: Started Timer logrotate for snap application juju-db.logrotate.

● snap.juju-db.daemon.service - Service for snap application juju-db.daemon
     Loaded: loaded (/etc/systemd/system/snap.juju-db.daemon.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/snap.juju-db.daemon.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Mon 2026-01-12 09:17:43 UTC; 3min 56s ago
    Process: 5060 ExecStart=/usr/bin/snap run juju-db.daemon (code=exited, status=14)
   Main PID: 5060 (code=exited, status=14)
        CPU: 175ms

Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Scheduled restart job, restart counter is at 4.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Stopped Service for snap application juju-db.daemon.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Start request repeated too quickly.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Failed with result 'exit-code'.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Failed to start Service for snap application juju-db.daemon.

Here is the output from the bootstrap:

Installing Juju machine agent
2026-01-12 09:17:35 INFO juju.cmd supercommand.go:56 running jujud [2.9.54.1 2882471b366bfaabdd1f26fb04c5f2ecf7b58321 gc go1.25.5]
2026-01-12 09:17:35 INFO juju.agent identity.go:22 writing system identity file
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied
2026-01-12 09:17:35 WARNING juju.mongo mongo.go:484 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/juju-db/common
2026-01-12 09:17:35 INFO juju.mongo mongo.go:489 Ensuring mongo server is running; data directory /var/snap/juju-db/common; port 37017
2026-01-12 09:17:35 WARNING juju.mongo service.go:338 configuring mongod  with --noauth flag enabled
2026-01-12 09:17:35 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2026-01-12 09:17:35 INFO juju.packaging.manager run.go:88 Running: snap install  --channel 4.4/candidate juju-db
2026-01-12 09:22:47 INFO juju.worker.peergrouper initiate.go:68 finished InitiateMongoServer
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1

wallyworld · 2026-01-12T23:10:30Z

Code looks good, though I can't replicated the PR steps because when I attempt to bootstrap I get:

root@juju-92f03c-0:~# systemctl status snap.juju-db.*
● snap.juju-db.logrotate.timer - Timer logrotate for snap application juju-db.logrotate
     Loaded: loaded (/etc/systemd/system/snap.juju-db.logrotate.timer; disabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2026-01-12 09:17:41 UTC; 3min 58s ago
    Trigger: Tue 2026-01-13 00:00:00 UTC; 14h left
   Triggers: ● snap.juju-db.logrotate.service

Jan 12 09:17:41 juju-92f03c-0 systemd[1]: Started Timer logrotate for snap application juju-db.logrotate.

● snap.juju-db.daemon.service - Service for snap application juju-db.daemon
     Loaded: loaded (/etc/systemd/system/snap.juju-db.daemon.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/snap.juju-db.daemon.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Mon 2026-01-12 09:17:43 UTC; 3min 56s ago
    Process: 5060 ExecStart=/usr/bin/snap run juju-db.daemon (code=exited, status=14)
   Main PID: 5060 (code=exited, status=14)
        CPU: 175ms

Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Scheduled restart job, restart counter is at 4.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Stopped Service for snap application juju-db.daemon.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Start request repeated too quickly.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Failed with result 'exit-code'.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Failed to start Service for snap application juju-db.daemon.

Here is the output from the bootstrap:

Installing Juju machine agent
2026-01-12 09:17:35 INFO juju.cmd supercommand.go:56 running jujud [2.9.54.1 2882471b366bfaabdd1f26fb04c5f2ecf7b58321 gc go1.25.5]
2026-01-12 09:17:35 INFO juju.agent identity.go:22 writing system identity file
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied
2026-01-12 09:17:35 WARNING juju.mongo mongo.go:484 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/juju-db/common
2026-01-12 09:17:35 INFO juju.mongo mongo.go:489 Ensuring mongo server is running; data directory /var/snap/juju-db/common; port 37017
2026-01-12 09:17:35 WARNING juju.mongo service.go:338 configuring mongod  with --noauth flag enabled
2026-01-12 09:17:35 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2026-01-12 09:17:35 INFO juju.packaging.manager run.go:88 Running: snap install  --channel 4.4/candidate juju-db
2026-01-12 09:22:47 INFO juju.worker.peergrouper initiate.go:68 finished InitiateMongoServer
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1

I just tested again with no issue.
Can you look at the content of /var/snap/juju-db/common/logs/mongodb.log?
Also, the content of /var/snap/juju-db/common/juju-db.config should be

# WARNING
# autogenerated by juju on 2026-01-12 23:06:20.51554226 +0000 UTC m=+1.695387613
# manual changes to this file are likely to be overwritten
bind_ip_all = true
auth = true
tlsAllowInvalidHostnames = true
oplogSize = 1024
tlsMode = requireTLS
tlsCertificateKeyFile = /var/snap/juju-db/common/server.pem
tlsCertificateKeyFilePassword=ignored = true
port = 37017
tlsCAFile = /var/snap/juju-db/common/ca.crt
storageEngine = wiredTiger
replSet = juju
dbpath = /var/snap/juju-db/common/db
logpath = /var/snap/juju-db/common/logs/mongodb.log
keyFile = /var/snap/juju-db/common/shared-secret
journal = true
slowms = 1000
quiet = true
ipv6 = truer

Mongo 4.4.30 requires client connections be properly authenticated. The mongo service needs to be started with a CA cert to use and client connections need to be configured with certificates.

wallyworld · 2026-01-14T10:55:50Z

/build

sinanawad · 2026-01-20T06:13:15Z

@SimonRichardson Please take another look, you and @hpidcock are reviewing this.

SimonRichardson

QA for bootstrap correctly works now. So something had changed between when I originally tested it and now.

Small efficiency fixes also.

…ository

manadart

Looks like we're safe with this, at least as far as 2.9 upgrades go.

We should check 3.6 carefully when it merges forward.

wallyworld · 2026-01-28T00:26:28Z

/merge

#21675 Merge 2.9 Brings in several dependabot updates, plus: #21644 [from wallyworld/fix-ociimage-pagination](facb908) #21563 [from wallyworld/add-mongo-4.4.30-support](fd845a0) Drops 2.9 only fixes: #21570 [from wallyworld/fix-noble-support](f4c0be5) #21494 [from wallyworld/fix-charmci-tests](589b0ac) We also use podman to build and push test images for the k8s upgrade smoke test. We need to be able to force these images to use the docker v2 format. Sadly juju can't handle the oci format and that's what we now get using docker cli. This PR adds support for the oci format so we're ok moving forward. We also drop the use of skopeo to push the test images since podman does the job. Most conflicts due to dropped commits. ``` # Conflicts: # .github/workflows/update-brew-formulae.yml # .github/workflows/upgrade.yml # CONTRIBUTING.md # agent/agent_test.go # agent/agentbootstrap/bootstrap_test.go # apiserver/facades/client/application/application.go # apiserver/facades/client/client/client.go # apiserver/facades/client/client/client_test.go # apiserver/facades/client/modelupgrader/findtools.go # apiserver/facades/client/modelupgrader/upgrader_test.go # cloudconfig/podcfg/image.go # cloudconfig/podcfg/podcfg_test.go # cmd/containeragent/initialize/package_test.go # cmd/juju/application/deploy_test.go # cmd/juju/application/deployer/bundlehandler.go # cmd/juju/application/deployer/charm.go # cmd/juju/application/deployer/charm_test.go # cmd/juju/application/deployer/deployer.go # cmd/juju/application/deployer/deployer_test.go # cmd/juju/application/unexpose_test.go # cmd/juju/commands/bootstrap.go # cmd/juju/commands/bootstrap_test.go # cmd/juju/machine/upgrademachine.go # cmd/jujud/agent/bootstrap_test.go # cmd/jujud/agent/machine.go # controller/config.go # core/arch/arches.go # core/series/supportedseries.go # core/series/supportedseries_test.go # docker/registry/internal/acr_test.go # docker/registry/internal/base_client.go # docker/registry/internal/base_manifests_test.go # docker/registry/internal/ecr_test.go # docker/registry/mocks/registry_mock.go # go.mod # go.sum # internal/provider/kubernetes/bootstrap.go # internal/provider/kubernetes/bootstrap_test.go # mongo/mongo.go # mongo/mongodfinder_test.go # mongo/service.go # mongo/service_test.go # service/snap/snap_test.go # snap/local/wrappers/fetch-oci # state/backups/restore.go # state/backups/restore_test.go # upgrades/upgradevalidation/upgrade_test.go ```

jujubot assigned wallyworld Jan 12, 2026

jujubot added the 2.9 label Jan 12, 2026

wallyworld force-pushed the add-mongo-4.4.30-support branch 2 times, most recently from f47ee3e to aa47584 Compare January 12, 2026 06:42

wallyworld requested a review from hpidcock January 12, 2026 06:48

wallyworld force-pushed the add-mongo-4.4.30-support branch from aa47584 to 2882471 Compare January 12, 2026 06:51

wallyworld requested a review from SimonRichardson January 12, 2026 06:51

SimonRichardson reviewed Jan 12, 2026

View reviewed changes

hpidcock reviewed Jan 13, 2026

View reviewed changes

Comment thread mongo/open.go Outdated

wallyworld force-pushed the add-mongo-4.4.30-support branch 4 times, most recently from a80725a to 394d39f Compare January 14, 2026 01:37

wallyworld added 4 commits January 14, 2026 13:12

feat: start mgo with arg to use CA cert

49ce4d5

Mongo 4.4.30 requires client connections be properly authenticated. The mongo service needs to be started with a CA cert to use and client connections need to be configured with certificates.

chore: remove obsolete client (macos) tests

781c22b

feat: generate the mongo client cert in the machine agent

d2b4a52

feat: mongo client certs are valid for 15 minutes

dff2ab4

wallyworld force-pushed the add-mongo-4.4.30-support branch from 394d39f to dff2ab4 Compare January 14, 2026 06:34

wallyworld requested a review from hpidcock January 14, 2026 06:43

feat: update default juju-db snap cjhannel to 4.4.30

e21db72

wallyworld force-pushed the add-mongo-4.4.30-support branch from 5fa6379 to e21db72 Compare January 14, 2026 07:51

wallyworld added 2 commits January 16, 2026 16:14

feat: seed test repo with legacy juju-db oci image

b191ec4

feat: when upgrading and mongo 4.0 is used, use ssl args not tls

6979e99

wallyworld force-pushed the add-mongo-4.4.30-support branch from c6e681e to 6979e99 Compare January 19, 2026 04:02

SimonRichardson approved these changes Jan 20, 2026

View reviewed changes

wallyworld force-pushed the add-mongo-4.4.30-support branch 2 times, most recently from e724025 to 0163274 Compare January 21, 2026 22:42

chore: add back support for older juju-db tracks

e521883

Small efficiency fixes also.

wallyworld force-pushed the add-mongo-4.4.30-support branch 2 times, most recently from d937168 to 2868511 Compare January 22, 2026 08:35

feat: add support for using skopeo for copying oci images in seed_rep…

7b07e88

…ository

wallyworld force-pushed the add-mongo-4.4.30-support branch from 2868511 to 7b07e88 Compare January 22, 2026 08:41

manadart approved these changes Jan 27, 2026

View reviewed changes

Comment thread mongo/service.go Outdated

Comment thread mongo/service.go Outdated

refactor: improve legact ssl arg selection for mongo

636fc7b

jujubot merged commit fd845a0 into juju:2.9 Jan 28, 2026
16 of 21 checks passed

wallyworld mentioned this pull request Jan 28, 2026

chore: merge 2.9 #21675

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: start mgo with arg to use CA cert#21563

feat: start mgo with arg to use CA cert#21563
jujubot merged 10 commits intojuju:2.9from
wallyworld:add-mongo-4.4.30-support

wallyworld commented Jan 12, 2026 •

edited

Loading

Uh oh!

SimonRichardson left a comment •

edited

Loading

Uh oh!

wallyworld commented Jan 12, 2026

Uh oh!

Uh oh!

wallyworld commented Jan 14, 2026

Uh oh!

sinanawad commented Jan 20, 2026

Uh oh!

SimonRichardson left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manadart left a comment

Uh oh!

Uh oh!

Uh oh!

wallyworld commented Jan 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

wallyworld commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QA steps

Links

Uh oh!

SimonRichardson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wallyworld commented Jan 12, 2026

Uh oh!

Uh oh!

wallyworld commented Jan 14, 2026

Uh oh!

sinanawad commented Jan 20, 2026

Uh oh!

SimonRichardson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manadart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wallyworld commented Jan 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wallyworld commented Jan 12, 2026 •

edited

Loading

SimonRichardson left a comment •

edited

Loading