Custom Replication Factor — Follow-up on PR #920

[Daniilchik]

Hi all,
I came across PR #920, which introduced support for custom replication factors in Ratis buckets. This is something we’re also interested in — our use case requires the ability to specify arbitrary replication factors (like 4, 6, etc.), beyond the existing ONE and THREE.

I also noticed that @maobaolong @elek @xiaoyuyao were involved in the discussion around this PR, and I wanted to ask:
• Were there any technical or design objections to the approach in #920 that prevented it from being merged?
• If there were no fundamental blockers, we’d like to pick up that work, adapt it to the current code, and move the discussion forward.

Please let us know if there are any outstanding concerns or alternative proposals.
Thanks!

[maobaolong]

@Daniilchik It is 5 year past, so many things changed, so I think you can pick it up and make the replication factor can be customize.

[errose28]

#920 didn't add support for custom replication factors, it only added the ability to specify them in the API. Ozone's underlying replication architecture is still based on Raft/Ratis in the datanodes, which requires an odd number of replicas (and currently only tested with 1 or 3), so supporting even numbers like 4 and 6 would require re-architecting the replicated write path, probably to use chain replication like HDFS.

If you're looking for higher fault tolerance, EC can do that, and the EC write path does not use Ratis:

• Ratis 3 (3 identical copies): handles 2 faults
• EC 3-2: handles 2 faults
• EC 6-3: handles 3 faults (like the 4-way replication you mentioned, but with less space amplification)
• EC 10-4: handles 4 faults

Currently there is no replication option that handles more than 4 faults, but I don't know how much demand there is for handling that many simultaneous issues. In practice most deployments I've seen start looking at cross cluster/datacenter backups to increase redundancy in those cases.