Huge downloading performance gap between s3g and ozone cli

[Lucas12138]

Hi community,
We're usually using s3 APIs to access our ozone clusters (our clusters are using EC) and we recently detected a performance drop after switch pvc providers. So, we did some deep dive and found that the
s3g speed: 16.2MB/s (observed from a manually created pod within the same namespace with s3 cli)
ozone speed: 823MB/s (observed from om node via ozone cli)
There's a huge downloading performance gap if we go through s3g.

What are the common bottlenecks that we should look out for our s3g? We're wondering if we need to fine tune some configs? I checked the cpu/mem/io seems none of them are bottlenecked

Our s3g settings currently are:

{
  "hadoop.hdds.db.rocksdb.WAL_size_limit_MB": "0MB",
  "hadoop.hdds.db.rocksdb.WAL_ttl_seconds": 1200,
  "hadoop.hdds.db.rocksdb.logging.enabled": false,
  "hadoop.hdds.db.rocksdb.logging.level": "INFO",
  "hadoop.hdds.db.rocksdb.writeoption.sync": false,
  "hdds.block.token.enabled": false,
  "hdds.block.token.expiry.time": "1d",
  "hdds.command.status.report.interval": "30s",
  "hdds.container.action.max.limit": 20,
  "hdds.container.checksum.verification.enabled": true,
  "hdds.container.chunk.write.sync": false,
  "hdds.container.close.threshold": 0.9,
  "hdds.container.ipc.port": 9859,
  "hdds.container.ipc.random.port": false,
  "hdds.container.ratis.admin.port": 9857,
  "hdds.container.ratis.datastream.enabled": false,
  "hdds.container.ratis.datastream.port": 9855,
  "hdds.container.ratis.datastream.random.port": false,
  "hdds.container.ratis.enabled": false,
  "hdds.container.ratis.ipc.port": 9858,
  "hdds.container.ratis.ipc.random.port": false,
  "hdds.container.ratis.leader.pending.bytes.limit": "1GB",
  "hdds.container.ratis.log.appender.queue.byte-limit": "32MB",
  "hdds.container.ratis.log.appender.queue.num-elements": 1,
  "hdds.container.ratis.log.purge.gap": 1000000,
  "hdds.container.ratis.log.queue.byte-limit": "4GB",
  "hdds.container.ratis.log.queue.num-elements": 1024,
  "hdds.container.ratis.num.container.op.executors": 10,
  "hdds.container.ratis.num.write.chunk.threads.per.volume": 10,
  "hdds.container.ratis.replication.level": "MAJORITY",
  "hdds.container.ratis.rpc.type": "GRPC",
  "hdds.container.ratis.segment.preallocated.size": "4MB",
  "hdds.container.ratis.segment.size": "64MB",
  "hdds.container.ratis.server.port": 9856,
  "hdds.container.ratis.statemachine.max.pending.apply-transactions": 10000,
  "hdds.container.ratis.statemachinedata.sync.timeout": "10s",
  "hdds.container.replication.compression": "NO_COMPRESSION",
  "hdds.container.report.interval": "60m",
  "hdds.container.token.enabled": false,
  "hdds.crl.status.report.interval": "60000ms",
  "hdds.datanode.client.bind.host": "0.0.0.0",
  "hdds.datanode.client.port": 19864,
  "hdds.datanode.df.refresh.period": "5m",
  "hdds.datanode.dir": "/data/storage",
  "hdds.datanode.du.refresh.period": "1h",
  "hdds.datanode.handler.count": 1,
  "hdds.datanode.http-address": "0.0.0.0:9882",
  "hdds.datanode.http-bind-host": "0.0.0.0",
  "hdds.datanode.http.auth.kerberos.keytab": "***REDACTED***",
  "hdds.datanode.http.auth.kerberos.principal": "***REDACTED***",
  "hdds.datanode.http.auth.type": "kerberos",
  "hdds.datanode.http.enabled": true,
  "hdds.datanode.https-address": "0.0.0.0:9883",
  "hdds.datanode.https-bind-host": "0.0.0.0",
  "hdds.datanode.metadata.rocksdb.cache.size": "64MB",
  "hdds.datanode.ratis.server.request.timeout": "2m",
  "hdds.datanode.storage.utilization.critical.threshold": 0.95,
  "hdds.datanode.storage.utilization.warning.threshold": 0.75,
  "hdds.datanode.volume.min.free.space": "5GB",
  "hdds.db.profile": "DISK",
  "hdds.grpc.tls.enabled": false,
  "hdds.grpc.tls.provider": "OPENSSL",
  "hdds.heartbeat.interval": "30s",
  "hdds.key.dir.name": "keys",
  "hdds.key.len": 2048,
  "hdds.node.report.interval": "60000ms",
  "hdds.pipeline.action.max.limit": 20,
  "hdds.pipeline.report.interval": "60000ms",
  "hdds.priv.key.file.name": "private.pem",
  "hdds.profiler.endpoint.enabled": false,
  "hdds.prometheus.endpoint.enabled": true,
  "hdds.public.key.file.name": "public.pem",
  "hdds.ratis.client.exponential.backoff.base.sleep": "4s",
  "hdds.ratis.client.exponential.backoff.max.sleep": "40s",
  "hdds.ratis.client.multilinear.random.retry.policy": "5s, 5, 10s, 5, 15s, 5, 20s, 5, 25s, 5, 60s, 10",
  "hdds.ratis.client.request.watch.timeout": "3m",
  "hdds.ratis.client.request.write.timeout": "5m",
  "hdds.ratis.client.retry.policy": "***REDACTED***",
  "hdds.ratis.client.retrylimited.max.retries": 180,
  "hdds.ratis.client.retrylimited.retry.interval": "1s",
  "hdds.ratis.leader.election.minimum.timeout.duration": "5s",
  "hdds.ratis.raft.client.async.outstanding-requests.max": 32,
  "hdds.ratis.raft.client.rpc.request.timeout": "60s",
  "hdds.ratis.raft.client.rpc.watch.request.timeout": "180s",
  "hdds.ratis.raft.grpc.flow.control.window": "5MB",
  "hdds.ratis.raft.server.datastream.client.pool.size": 10,
  "hdds.ratis.raft.server.datastream.request.threads": 20,
  "hdds.ratis.raft.server.delete.ratis.log.directory": true,
  "hdds.ratis.raft.server.leaderelection.pre-vote": true,
  "hdds.ratis.raft.server.log.appender.wait-time.min": "0us",
  "hdds.ratis.raft.server.notification.no-leader.timeout": "300s",
  "hdds.ratis.raft.server.rpc.request.timeout": "60s",
  "hdds.ratis.raft.server.rpc.slowness.timeout": "300s",
  "hdds.ratis.raft.server.watch.timeout": "180s",
  "hdds.ratis.raft.server.write.element-limit": 1024,
  "hdds.ratis.server.retry-cache.timeout.duration": "600000ms",
  "hdds.ratis.snapshot.threshold": 10000,
  "hdds.recon.heartbeat.interval": "60s",
  "ozone.s3g.client.buffer.size": "4KB",
  "ozone.s3g.http-address": "0.0.0.0:9878",
  "ozone.s3g.http-bind-host": "0.0.0.0",
  "ozone.s3g.http.auth.kerberos.keytab": "***REDACTED***",
  "ozone.s3g.http.auth.kerberos.principal": "***REDACTED***",
  "ozone.s3g.http.auth.type": "kerberos",
  "ozone.s3g.http.enabled": true,
  "ozone.s3g.kerberos.keytab.file": "***REDACTED***",
  "ozone.s3g.kerberos.principal": "***REDACTED***",
  "ozone.s3g.volume.name": "s3v",
  "ozone.s3g.secret.http.auth.type": "kerberos",
  "ozone.s3g.secret.http.enabled": false,
  "ozone.csi.s3g.address": "http://***REDACTED***",
  "ozone.csi.socket": "/var/lib/csi/csi.sock",
  "ozone.om.address.sorcerer-ozone.om0": "***REDACTED***",
  "ozone.om.address.sorcerer-ozone.om1": "***REDACTED***",
  "ozone.om.address.sorcerer-ozone.om2": "***REDACTED***",
  "ozone.recon.address": "***REDACTED***"
}

[errose28]

Hi @Lucas12138 are you still running the Ozone 1.4.x line? This sounds similar to #8946. Did you try the answer there in this case as well?

[Lucas12138]

Yes, still 1.4.1. This has already used the props suggested from #8946

Do you know what other props may cause this issue?

[yandrey321]

@Lucas12138 can you try this test with 2.0?

[ivandika3]

hdds.ratis.raft.server.watch.timeout=180s

By the way, this value is too large and can be reduced to 30s (see HDDS-10972).

[Lucas12138]

Thanks, we will try reducing this

[Lucas12138]

This prop change didn't help

[jojochuang]

@yandrey321

[ChenSammi]

"ozone.s3g.client.buffer.size": "4KB" // change this to 4MB.

[yandrey321]

yes, the default value is 4 MB, I'd experiment with values between 1MB and 4MB, to see what works best with your IO patterns.

[spacemonkd]

@Lucas12138 has there been any update on this?

Did setting the prop as suggested by @ChenSammi help?