JAMES-4142 Upgrade to Cassandra 5.0.x

[chibenwa]

• Step 1: validate James compatibility
• Step 2: upgrade perf test environment and bench
• Step 3: document the upgrade
• Step 4: update Ansible playbooks
• Step 5: Rolling adoption across all platforms
  • LNG prod
  • CNB preprod
  • CNB prod
  • MU preprod
  • MU prod
  • TWP platforms (dev, preprod, prod, cozy)

[chibenwa]

Likely a very good topic for this summer too...

[Arsnael]

James upgrade: apache#2780

Note: no driver update, it's the latest one already. Supposed to be compatible without issues.

[Arsnael]

sandbox: running cassandra 4.1.0

Cluster installed manually. I will try to figure out how to roll upgrade

Probably will do:

• run perf tests with latest release on current cassandra
• upgrade cassandra to 4.1.9 and run tests again
• upgrade to 5.0.4 and run tests again

[Arsnael]

Cassandra 4.1.0

JMAP

IMAP

JMAP honestly is more or less the same as last 1.0.8 release run I had, not much to say.

IMAP is a bit slower than 1.0.8 release (while it's exactly the same conditions) but we know it's the deal with ovh sometimes...

I would be more interested to see results after upgrade to 4.1.9 as I think it's more what we have with the most recent cassandra prod environments?

[chibenwa]

Just to confirm

After 5.0 upgrade did you run sstable upgrade ?

[Arsnael]

After 5.0 upgrade did you run sstable upgrade ?

Did not upgrade to 5 yet? Trying to figure out how it's been installed first... Dont wanna kill all the provisionning we have with a stupid mistake... But will keep in mind thanks

[Arsnael]

apache#2780 (comment)

We are stuck for now. Need an update of the java cassandra driver with the related fix for the compression option change chunk_length_kb to chunk_length_in_kb

I'm still doing some cleanup on our sandbox casandra (like opensearch, been a while since it got some good care). It allows me to understand a bit more on how to admin a cassandra cluster as well...

Will upgrade to 4.1.9 after some cleanup but not to v5 yet

[Arsnael]

actually technically we could upgrade to 5 still on sandbox no? The sstable upgrade should update the schema up to latest v5 cassandra and thus the compression option on that cache table? And I dont think we intend to create an other keyspace on it right?

But I might track down as well tables that we can safely remove since the time... disk space usage on cassandra node is around 75 ~ 80%, starts being quite high, it might be an issue potentially for upgrade as those migration operations requires extra space from what I can read, correct @chibenwa ?

[Arsnael]

To be a bit more precise on disk space:

• all nodes have 197GB size disk
• node1: 45GB left (77% usage)
• node2: 39GB left (80% usage)
• node3: 29GB left (85% usage)

Is that an issue for upgrading? Or for performance in general as well?

Meanwhile I:

• ran nodetool repair
• cleanup unused keyspaces
• cleanup unused tables ( did not really reclaim much space here... i guess there wasnt much data on those unusued tables)