HIVE-29579: K8s operator

[ayushtkn]

What changes were proposed in this pull request?

Add Hive Operators to deploy Hive

Why are the changes needed?

To deploy Hive on K8s

Does this PR introduce any user-facing change?

No

How was this patch tested?

Deployed on Docker Desktop -> Follow Readme -> Build Hive master (should have Tez AM PR) -> Build hive image -> Then Scenario 3 in README

After building the apache/hive-4.3.0-SNPASHOT docker image

mvn clean package -pl packaging/src/kubernetes -Pkubernetes -DskipTests
cd packaging/src/kubernetes
kubectl apply -f src/gen/hiveclusters.hive.apache.org-v1.yml

kubectl create namespace hive-operator

kubectl apply -f config/rbac/service-account.yaml
kubectl apply -f config/rbac/cluster-role.yaml
kubectl apply -f config/rbac/cluster-role-binding.yaml

export HIVE_VERSION=4.3.0-SNAPSHOT
envsubst < config/operator/deployment.yaml | kubectl apply -f -

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zookeeper
        image: zookeeper:3.9
        ports:
        - containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  selector:
    app: zookeeper
  ports:
  - port: 2181
EOF

helm repo add ozone https://apache.github.io/ozone-helm-charts/
helm install ozone ozone/ozone --version 0.2.0 --wait

kubectl exec statefulset/ozone-om -- ozone sh volume create /s3v
kubectl exec statefulset/ozone-om -- ozone sh bucket create /s3v/hive

kubectl create secret generic hive-db-secret \
  --from-literal=password=hive123

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        env:
        - name: POSTGRES_DB
          value: metastore
        - name: POSTGRES_USER
          value: hive
        - name: POSTGRES_PASSWORD
          value: hive123
        ports:
        - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector:
    app: postgres
  ports:
  - port: 5432
EOF

envsubst < config/samples/hivecluster-full-ha.yaml | kubectl apply -f -

Running Pods:

Beeline

kubectl exec -it deployment/my-hive-hiveserver2 -- beeline -u "jdbc:hive2://my-hive-hiveserver2:10000/"

HiveServer2 UI:

ayushsaxena@Q3NW54Y0C5 kubernetes % kubectl port-forward svc/my-hive-hiveserver2 10002:10002
Forwarding from 127.0.0.1:10002 -> 10002
Forwarding from [::1]:10002 -> 10002

[zhangbutao]

I can't wait to try out this feature!!!
BTW, besides the LLAP mode, can we also execute normal Tez tasks?

[ayushtkn]

BTW, besides the LLAP mode, can we also execute normal Tez tasks?

Thanks @zhangbutao for taking a look! Currently, the operator only supports LLAP mode (along with a Tez Local mode strictly for development and testing). Standard Tez mode isn't on my immediate radar right now; I can look into Tez in a follow-up maybe after them, though LLAP generally makes more sense in cloud environments

[zhangbutao]

Standard Tez

@ayushtkn That sounds great. I think LLAP on cloud/k8s is definitely a milestone for Hive — a really nice feature.

However, as far as I know, in the real world, there are still a large number of Hive on standard Tez mode jobs running on YARN. Many users want to migrate these Tez jobs to native Kubernetes(without relying on YARN's scheduling), just like Spark/Flink on k8s. I’m really looking forward to further updates on Tez on k8s as well.

Thanks!

[ayushtkn]

Thanks @zhangbutao for sharing the use case. I’ll definitely look into it. Most likely, we’ll have support for both LLAP mode and Tez mode before the release — or at least I’ll try to cover as much as possible. I will create a ticket to track that.

I took a quick look at what might be needed, and it doesn’t seem like it would require too many changes after this. I could be wrong, but I’ll definitely give it a shot.

In the end, it’s up to the users which mode they want to use. Whichever mode they choose, they’re still using Hive, which is a win-win for us :-)

[sonarqubecloud]

Quality Gate passed

Issues
56 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.2% Duplication on New Code

See analysis details on SonarQube Cloud