Ozone ofs in presto #7259
Replies: 12 comments 49 replies
-
|
Yes, ofs is a Hadoop compatible protocol and should be able to be used by Presto. Our Ozone cluster is currently being used by Presto to query Hive tables on Ozone (instead of HDFS). |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ivandika3 Thanks for the quick response. I tried the OFS protocol by adding the jar
|
Beta Was this translation helpful? Give feedback.
-
|
@lukmanulhakkeem Does the client runs on the docker environment as well? Maybe you can take a look at @adoroszlai patch #5016 for docker-compose setup and script for Trino, Iceberg, and Ozone. I'm personally not familiar with the details for Trino/Presto on Ozone, just saying that it's possible to run Trino/Presto on top of Ozone. Edit: Seems #5016 runs it on top of S3G, instead of ofs. |
Beta Was this translation helpful? Give feedback.
-
|
No, the client is running in my local. As you mentioned the above PR is using S3G. Is there any idea how this internal IP is getting to the client. I am not aware of the internal communication inside the ozone. I guess it will work If I managed to supply the public IP instead of internal IP to the client. But not sure how it can be done. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, maybe because the datanode IPs was not exposed to localhost. Can try upload the key using ozone cli using I'll try to come up with PoC when I have time. |
Beta Was this translation helpful? Give feedback.
-
|
Sure @ivandika3 i will try it out. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, usually I will follow https://cwiki.apache.org/confluence/display/OZONE/Run+Ozone+cluster+from+IDE to setup local Ozone cluster from Intellij. Can try this order:
Afterwards, you can do some sanity checking first by uploading 1 or 2 objects. |
Beta Was this translation helpful? Give feedback.
-
|
This is not only the issue from presto. I followed the below documentation to try locally using hadoop I can create volume and schema using |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 @adoroszlai - So presto should be there in the same machine where ozone om is installed right? I am assuming this would be the same regardless of my setup like docker or in a physical cluster. Please correct me If I am wrong |
Beta Was this translation helpful? Give feedback.
-
|
@lukmanulhakkeem Here's the client requirements:
So not necessarily the same machine, you can whether you can connect by using ping or telnet on OM and DN. |
Beta Was this translation helpful? Give feedback.
-
No, docker will create an internal network that cannot be accessed from outside unless you use port forwarding. Even if you port forward the datanodes, the datanode network address returned by SCM and OM to client will be the docker internal network address, which cannot be used by client to connect from outside docker network. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I understand that. But will that be the same if I am having ozone in a physical machine, for eg say ozone in Cloudera. |
Beta Was this translation helpful? Give feedback.
-
|
Not familiar with Ozone on Cloudera, but should be similar client setup. |
Beta Was this translation helpful? Give feedback.
-
|
Does the client directly interact with data nodes both during read and write? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, client will directly connect to datanodes for read and write ops. |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 - I tried presto and ozone in the same machine. This is working with hive catalog in presto. But I tried the remote connectivity to an ozone instance in cloudera. It throws the same error I posted in the previous thread. The client is trying to access the internal IP of the datanode which is in the cloudera VM. So this is not only with the docker setup. Is there a way I can specify datanode host at the client side using ozone-site.xml or hdfs-site.xml? |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 I made the changes to use hostname always for the connection [ But Select is not working. I am getting below error when I am doing I checked |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 @adoroszlai any idea on this? |
Beta Was this translation helpful? Give feedback.
-
@lukmanulhakkeem Glad to know it works.
Since Ozone uses gRPC to read (with Netty transport), perhaps you can change the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks very much @ivandika3 . I will try this |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 Now select is working..Thanks |
Beta Was this translation helpful? Give feedback.
-
|
@lukmanulhakkeem try using Docker's host network driver. This eliminates container-specific IP addresses and NAT. On the flip side, you'll need to configure different set of ports for each service with multiple instances (e.g. each datanode) to avoid port conflict. |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 @adoroszlai - I cloned the repo and checkout to 
But I am getting the below error in presto when used the above jar 
But it was working fine with the below Jar from the maven repository. https://mvnrepository.com/artifact/org.apache.ozone/ozone-filesystem-hadoop3/1.4.0 Is there any change in the protobuf in ozone? |
Beta Was this translation helpful? Give feedback.
-
|
Where did you put the |
Beta Was this translation helpful? Give feedback.
-
|
@adoroszlai Maybe related to #6359? |
Beta Was this translation helpful? Give feedback.
-
|
I manually installed the jar to local maven repository and added that dependency in the |
Beta Was this translation helpful? Give feedback.
-
|
@lukmanulhakkeem Ozone comes with two different FS jars for Hadoop 3, where the only difference is that:
Protobuf itself is not present in either jar, it needs to be provided, just like Hadoop's other dependencies. Since it was working fine with non-shaded protobuf, I guess your environment comes with non-shaded protobuf. Please use |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @adoroszlai . Then I will check from the presto end. |
Beta Was this translation helpful? Give feedback.
-
|
@ivandika3 @adoroszlai Thanks for all the support. I am just briefing what I have done here. I wanted to run queries in Presto using Apache Ozone as the storage backend with the OFS protocol. To achieve this, I added the Ozone JAR to the required modules in Presto. The connection was successful when Ozone and Presto were running on the same machine. However, it did not work when Ozone and Presto were on different machines (e.g., two separate cloud VMs). Upon investigation, I noticed that the Ozone Manager (OM) always returns the internal IP addresses of the DataNodes during read and write operations. Based on @ivandika3's guidance, I added a configuration in Ozone to use the hostnames of the DataNodes instead of their internal IPs for read and write operations. I tested the above changes on an Ozone instance in Cloudera, and the setup worked fine with Presto using this configuration. However, the issue persists when Ozone is running inside a Docker container, as the hostname of the Docker container differs from what is expected. I attempted to change the hostname of the Docker container, but it still did not resolve the problem. As a result, we are unable to remotely access Ozone storage using the OFS protocol. This issue is not limited to Presto; it can also be reproduced with the hdfs dfs command. Is it possible for the community to provide a fix for this issue in the future? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @lukmanulhakkeem for the summary. I don't think this is something that can be fixed internally in Ozone. I hope someone can help with Docker networking. |
Beta Was this translation helpful? Give feedback.
-
|
@adoroszlai, does anyone test ozone's remote access using the ofs client from the community? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry @lukmanulhakkeem, I don't know. Asking on the dev mailing list may reach different parts of the community (there may be list subscribers not checking GitHub discussions), and thus may help getting help with this problem. |
Beta Was this translation helpful? Give feedback.
-
|
Could you add me there? |
Beta Was this translation helpful? Give feedback.
-
|
Please see https://ozone.apache.org/community/ for info on how to subscribe. |
Beta Was this translation helpful? Give feedback.
-
|
@adoroszlai @ivandika3 Thanks for your help. Probably I can close this discussion if you feel this is expected behavior / do you think there is a workaround Finally, i am facing an issue. Just checking if you have faced this I am using the ozone jar |
Beta Was this translation helpful? Give feedback.
-
|
I got it. Inside the |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the only difference between |
Beta Was this translation helpful? Give feedback.
-
|
If you can, please consider contributing a user doc. Given the amount of interest we have around Trino/Presto I think this is going to help the community a lot. I am trying to add more coverage for user howto / applications but I am shorthanded. https://ozone.apache.org/docs/edge/integration.html |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Sure I will try |
Beta Was this translation helpful? Give feedback.
-
|
I faced with same issue. We are trying to deploy Ozone into our k8s clusster. For security reason our development process enforces us to use port-forward for interaction with dbs. When I tried to pf Ozone and connect to it with pyspark I recived always IP address (which I can't redirect to 127.0.0.1). I saw property hdds.datanode.use.datanode.hostname and tried to set it with:
gateway = spark.sparkContext._gateway None of that works :( Also, even it would work, createChannel uses hardcoded IP instead of hostname. It's frustrating because in hdfs this problem is solved with one simple setup @adoroszlai @ivandika3 I'm ready to contribute this thing, but I 100% will need help to figure out why configuration property doesn't work here |
Beta Was this translation helpful? Give feedback.
-
|
Sadly, but SELECT operations still doesn't work, even with this parameter, due to hardcoded https://github.com/apache/ozone/blob/ozone-1.4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java#L189-L190 |
Beta Was this translation helpful? Give feedback.
-
|
Can make smth like that Which will probably fix it |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @KochankovID for sharing the details. Would you like to submit a PR? nit: store IP or hostname in variable, use it in Also, please replace |
Beta Was this translation helpful? Give feedback.
-
|
PR: #8518 🙏 Please take a look |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @KochankovID for the PR |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi , I am able to connect to ozone using the iceberg catalog from presto using the s3 gateway. But is it possible to connect to ozone from presto using the ofs protocol? Did anyone try this?
Beta Was this translation helpful? Give feedback.
All reactions