ozone-trino docker-compose connection refused

[tolstykh-da]

hi, I'm trying to use ozone along with trino through this config.

  datanode:
      image: apache/ozone:1.3.0
      networks:
       iceberg_net:
      ports:
         - 9864
      command: ["ozone","datanode"]
      env_file:
         - ./docker-config
  om:
      image: apache/ozone:1.3.0
      networks:
       iceberg_net:
      ports:
         - 9874:9874
      environment:
         ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
         WAITFOR: scm:9876
      env_file:
         - ./docker-config
      command: ["ozone","om"]
  scm:
      image: apache/ozone:1.3.0
      networks:
       iceberg_net:
      ports:
         - 9876:9876
      env_file:
         - ./docker-config
      environment:
         ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
      command: ["ozone","scm"]
  recon:
      image: apache/ozone:1.3.0
      networks:
       iceberg_net:
      ports:
         - 9888:9888
      env_file:
         - ./docker-config
      command: ["ozone","recon"]
  s3g:
      image: apache/ozone:1.3.0
      ports:
         - 9878:9878
      env_file:
         - ./docker-config
      environment:
      - AWS_ACCESS_KEY_ID=minio
      - AWS_SECRET_ACCESS_KEY=minio123
      - AWS_REGION=us-east-1
      networks:
       iceberg_net:
        #aliases:
         # - dwh.s3g
      command: ["ozone","s3g"]

#docker-config
OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.container.size=5GB
OZONE-SITE.XML_ozone.scm.datanode.ratis.volume.free-space.min=10MB
OZONE-SITE.XML_ozone.scm.pipeline.creation.interval=30s
OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1
OZONE-SITE.XML_ozone.scm.names=scm
OZONE-SITE.XML_ozone.scm.datanode.id.dir=/data
OZONE-SITE.XML_ozone.scm.block.client.address=scm
OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_hdds.scmclient.max.retry.timeout=30s
OZONE-SITE.XML_ozone.s3g.volume.name=s3g

HADOOP_OPTS="-Dhadoop.opts=test"
HDFS_STORAGECONTAINERMANAGER_OPTS="-Dhdfs.scm.opts=test"
HDFS_OM_OPTS="-Dhdfs.om.opts=test"
HDDS_DN_OPTS="-Dhdds.dn.opts=test"

no_proxy=om,scm,s3g,recon,kdc,localhost,127.0.0.1

In trino, I turn to this (works with minio)

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://rest:8181
hive.s3.aws-access-key=minio
hive.s3.aws-secret-key=minio123
hive.s3.endpoint=http://localhost:9878
hive.s3.path-style-access=true
hive.s3.ssl.enabled=false
iceberg.file-format=ORC

also for minio i specify

        aliases:
          - dwh.minio

when ozone is running i can create via aws s3api cli bucket=dwh

aws s3api --endpoint http://localhost:9878 create-bucket --bucket=dwh
{
    "Location": "http://localhost:9878/dwh"
}
#I can enter the bucket through cyberduck 

when I try to create table from trino, I get
Unable to execute HTTP request: Connect to localhost:9878 [localhost/127.0.0.1] failed: Connection refused

tell me how to properly configure ozone and how to access it through s3?
may need http://localhost:9878/s3g? or http://s3g.localhost:9878?

[tolstykh-da]

if you refer to http://s3g:9878 then I get an error Error committing write to Hive
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null)

[adoroszlai]

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null)

Please check s3g log for INTERNAL_ERROR (or any exception).

[errose28]

500 could also mean there was a problem in the Ozone Manager that did not translate to a known S3 error code, so probably want to check the OM log as well.

[tolstykh-da]

we understand what is needed endpoint=virtual-host-style URL
added parameter ozone.s3g.domain.name=s3g.internal
got the look = http://bucket.endpoin:9878
but error= the specified bucket does not exist from cyberduck
and spark= WARN Tasks: Retrying task after failure: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 627938ca-f214-40d5-b091-53c34da272f1, Extended Request ID: lUOmWDhq8Lbu)
what could be the problem?

[adoroszlai]

@tolstykh-da I've created an example docker-compose-based setup for integrating Ozone with Iceberg and Trino. Based on this experience, I recommend configuring Iceberg and Trino to access Ozone S3 Gateway using path-style URLs. Here are the relevant configs:

ozone/hadoop-ozone/dist/src/main/compose/ozone/iceberg.yaml

Lines 46 to 47 in 41be4fb

	- CATALOG_S3_ENDPOINT=http://s3g:9878/
	- CATALOG_S3_PATH__STYLE__ACCESS=true

ozone/hadoop-ozone/dist/src/main/compose/ozone/spark.conf/spark-defaults.conf

Lines 24 to 25 in 41be4fb

	spark.sql.catalog.demo.s3.endpoint http://s3g:9878/
	spark.sql.catalog.demo.s3.path-style-access true

ozone/hadoop-ozone/dist/src/main/compose/ozone/trino.catalog/iceberg.properties

Lines 23 to 24 in 41be4fb

	hive.s3.endpoint=http://s3g:9878/
	hive.s3.path-style-access=true

See #5016 for the complete example.

[hchautrung]

Hi @adoroszlai
Could you please to provide the docker-compose.yml file? I'm quite interrested in Open Lakehouse using Apache Ozone, Iceberg and Spark and want to try.

Thank you

[adoroszlai]

Sorry @hchautrung, I just saw your comment.

https://github.com/adoroszlai/ozone/tree/HDDS-8971/hadoop-ozone/dist/src/main/compose/ozone has all necessary files. Docker Compose definition is split into 3 files:

• docker-compose.yaml
• iceberg.yaml
• trino.yaml

See test-iceberg.sh for usage.