Freon DFSG creates a huge amount of "Container Pre-allocated" space

[silvanias]

Hello Ozone team, I have been trying to stress test my HA cluster a bit, and have been using the freon dfsg tool to do so.
When running very small tests like so:
ozone freon dfsg -n 10 -r "ofs://dev2/benchmark-vol/freon-bucket/ultra-small-run"
All executes correctly. However, this small test generates up to 90Gb of temporary Container Pre-allocated space, according to my recon UI. After the execution completes, the pre-allocated space dissapears and returns to normal.

The consequence of this is that if I run any larger tests (100 files of 100mb e.g.) my cluster fills up immediately and the execution gets stuck on ratis election between 3 datanodes, as they all appear full.

Is this intended for the tool to generate so much Pre-allocated space for containers? If so, is there a way to reduce this space? I am aware that containers are by default 5Gb, so I would not have thought so many containers need to be allocated in advance.

Thank you!

[silvanias]

For the type of errors that occur, Freon outputs like so:

2026-01-28 16:50:42,187 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 10)
2026-01-28 16:50:42,987 [grpc-default-executor-0] ERROR impl.OrderedAsync:  client-08A64C11390C: Failed* RaftClientRequest:client-08A64C11390C->19039502-c106-450e-a699-cfaf9b8781a7@group-F2DFE4BE526C, cid=14, seq=2*, RW, cmdType: WriteChunk
traceID: ""
containerID: 72
datanodeUuid: "19039502-c106-450e-a699-cfaf9b8781a7"
writeChunk {
  blockID {
    containerID: 72
    localID: 115816896921600120
    blockCommitSequenceId: 0
    replicaIndex: 0
  }
  chunkData {
    chunkName: "115816896921600120_chunk_1"
    offset: 0
    len: 4194304
    checksumData {
      type: CRC32
      bytesPerChecksum: 16384
      checksums: "\025O@I"
      ...
      ...
   }
  }
}
version: 3
, data.size=4194304
java.util.concurrent.CompletionException: java.io.IOException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@72ed4963 rejected from java.util.concurrent.ThreadPoolExecutor@50399cbc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]

and from one of the datanodes perspective, the logs show many warnings about failing request votes:

2026-01-28 16:51:54,138 [grpc-default-executor-86] INFO org.apache.ratis.server.RaftServer$Division: 06dd58a2-aadc-45e3-919a-0882c2a07acc@group-88DE07F17A64: receive requestVote(PRE_VOTE, 19039502-c106-450e-a699-cfaf9b8781a7, group-88DE07F17A64, 2, (t:1, i:513))
2026-01-28 16:51:54,138 [grpc-default-executor-86] WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 06dd58a2-aadc-45e3-919a-0882c2a07acc: Failed requestVote 19039502-c106-450e-a699-cfaf9b8781a7->06dd58a2-aadc-45e3-919a-0882c2a07acc#0: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 06dd58a2-aadc-45e3-919a-0882c2a07acc@group-88DE07F17A64 is not in [RUNNING]: current state is CLOSED
2026-01-28 16:51:57,136 [grpc-default-executor-86] INFO org.apache.ratis.server.RaftServer$Division: 06dd58a2-aadc-45e3-919a-0882c2a07acc@group-F2DFE4BE526C: receive requestVote(PRE_VOTE, 6e8f1819-7557-4ec6-9443-ee16c716679f, group-F2DFE4BE526C, 2, (t:2, i:51))
2026-01-28 16:51:57,136 [grpc-default-executor-86] WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 06dd58a2-aadc-45e3-919a-0882c2a07acc: Failed requestVote 6e8f1819-7557-4ec6-9443-ee16c716679f->06dd58a2-aadc-45e3-919a-0882c2a07acc#0
java.lang.IllegalStateException: 06dd58a2-aadc-45e3-919a-0882c2a07acc@group-F2DFE4BE526C-SegmentedRaftLog is expected to be opened but it is CLOSED
        at org.apache.ratis.util.OpenCloseState.assertOpen(OpenCloseState.java:63)
        at org.apache.ratis.server.raftlog.RaftLogBase.checkLogState(RaftLogBase.java:112)

[silvanias]

Further testing with a lower thread amount yield the same errors with the same amount of "container pre-allocated space" (80Gb).

ozone freon dfsg -n 100 -t 2 -r "ofs://dev2/benchmark-vol/freon-bucket/ultra-small-run"

This is on Version 2.0.0 , with default container size.

For just a singular file it successfully executes, a new container is created, and 15gb of Container pre-allocated space is created:

ozone freon dfsg -n 1 -s 1024 -t 1 -r "ofs://dev2/benchmark-vol/freon-bucket/single-file-test"

It appears that containers scale linearly with file count, and close when they only have a few mbs/ 1 or 2 key's in. E.g.

{
  "state" : "CLOSED",
  "stateEnterTime" : "2026-01-28T14:32:55.101Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "replicationType" : "RATIS"
  },
  "usedBytes" : 4194304,
  "numberOfKeys" : 3,
  "lastUsed" : "2026-01-29T14:59:11.424322Z",
  "owner" : "dev2",
  "containerID" : 16,
  "deleteTransactionId" : 37,
  "sequenceId" : 38,
  "open" : false,
  "deleted" : false
}

Only one container stays open at a time. The issue for this is unclear, as it appears all pipelines are fine. Connectivity between datanodes is also fine when curling. The layout of the bucket that freon is running on is FILE_SYSTEM_OPTIMIZED.

[adoroszlai]

Ozone creates multiple containers and pipelines to improve concurrent writes. With only 3 datanodes, single disk, limited space, it's better to reduce some of the settings, like:

• ozone.scm.datanode.pipeline.limit: 1
• ozone.scm.pipeline.owner.container.count: 1
• ozone.scm.container.size: 1GB

[devabhishekpal]

Thanks @adoroszlai for the answer.
Closing this discussion since it has been answered, please feel free to re-open in case of follow up questions