Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion charts/clickhouse/templates/chi.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
{{- $service_name := tpl (include "clickhouse.serviceTemplateName" . ) . -}}
{{- $extraPortNames := list }}
{{- range .Values.clickhouse.extraPorts }}
{{- $extraPortNames = append $extraPortNames .name }}
{{- end }}
{{- $serviceHttpsPort := (((.Values.clickhouse).settings).https_port) | default "" -}}
{{- $serviceSecureTcpPort := (((.Values.clickhouse).settings).tcp_port_secure) | default "" -}}
---
apiVersion: "clickhouse.altinity.com/v1"
kind: ClickHouseInstallation
Expand Down Expand Up @@ -62,6 +68,16 @@ spec:
targetPort: {{ .containerPort }}
{{- end }}
{{- end }}
{{- if and $serviceHttpsPort (not (has "https" $extraPortNames)) }}
- name: https
port: {{ $serviceHttpsPort }}
targetPort: {{ $serviceHttpsPort }}
{{- end }}
{{- if and $serviceSecureTcpPort (not (has "tcp-secure" $extraPortNames)) }}
- name: tcp-secure
port: {{ $serviceSecureTcpPort }}
targetPort: {{ $serviceSecureTcpPort }}
{{- end }}
selector:
{{- include "clickhouse.selectorLabels" . | nindent 12 }}
{{- if .Values.clickhouse.lbService.enabled }}
Expand Down Expand Up @@ -96,6 +112,16 @@ spec:
targetPort: {{ .containerPort }}
{{- end }}
{{- end }}
{{- if and $serviceHttpsPort (not (has "https" $extraPortNames)) }}
- name: https
port: {{ $serviceHttpsPort }}
targetPort: {{ $serviceHttpsPort }}
{{- end }}
{{- if and $serviceSecureTcpPort (not (has "tcp-secure" $extraPortNames)) }}
- name: tcp-secure
port: {{ $serviceSecureTcpPort }}
targetPort: {{ $serviceSecureTcpPort }}
{{- end }}
selector:
{{- include "clickhouse.selectorLabels" . | nindent 12 }}
{{- end }}
Expand Down Expand Up @@ -190,7 +216,8 @@ spec:
{{- if .Values.clickhouse.settings }}
settings:
{{- range $key, $value := .Values.clickhouse.settings }}
{{ $key }}: "{{ $value }}"
{{- $valueIsNumeric := regexMatch "^[0-9]+$" ( $value | toString ) }}
{{ $key }}: {{ ternary $value ( $value | quote ) $valueIsNumeric }}
Comment on lines -193 to +220
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tangential optimization to not wrap every clickhouse.settings.* value in quotes. Instead, if the value is fully numeric, it won't quote the value, allowing the type to remain as a number.

This allows the generated spec to look similar to the documented example:

spec:
  configuration:
    clusters:
    # ...
    settings:
      tcp_port: 9000
      tcp_port_secure: 9440
      https_port: 8443

{{- end }}
{{- end }}
clusters:
Expand Down
39 changes: 34 additions & 5 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ The test suite provides comprehensive coverage across multiple dimensions:
#### **2. ClickHouse Functionality**
- ✅ Version verification
- ✅ Connection testing
- ✅ Server-side TLS/SSL and HTTPS
- ✅ Query execution
- ✅ Cluster topology (system.clusters)
- ✅ Replication health (system.replicas)
Expand Down Expand Up @@ -246,7 +247,6 @@ Areas that **need additional testing or are not fully covered**:
- ❌ **Backup and restore** - No automated backup/restore testing
- ❌ **Disaster recovery** - No full cluster failure scenarios
- ❌ **Network policies** - Limited testing of K8s network restrictions
- ❌ **TLS/SSL** - No certificate or encryption testing
- ❌ **Monitoring integration** - Prometheus scraping tested only via annotations
- ❌ **Logging integration** - No FluentD/ElasticSearch integration tests
- ❌ **Multi-cluster** - No federation or distributed query tests
Expand All @@ -273,11 +273,12 @@ Areas that **need additional testing or are not fully covered**:
## 🌍 Supported Environment

- **Operating System**: [Ubuntu](https://ubuntu.com/) 22.04 / 24.04
- **Python**: >= 3.10.12
- **Python**: >= 3.10.12, <= 3.12 (3.13+ has `lzma` package that is incompatible with the test framework)
- **Kubernetes**: >= 1.24
- **Helm**: >= 3.8.0
- **Minikube**: >= 1.28.0 (for local testing)
- **Docker**: Required as Minikube driver
- (alternatively) **OrbStack**: >= 2.0
- **kubectl**: Latest stable version

---
Expand All @@ -286,7 +287,9 @@ Areas that **need additional testing or are not fully covered**:

### Kubernetes Cluster

You need access to a Kubernetes cluster. For **local testing**, use Minikube:
You need access to a Kubernetes cluster. For **local testing**, two providers are supported:

#### Option 1: Minikube (default)

```bash
# Install Minikube
Expand All @@ -297,6 +300,27 @@ sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube version
```

#### Option 2: OrbStack (account required, may need license)

1. Install OrbStack by following the [_Quick start_ guide \(docs.orbstack.dev\)](
https://docs.orbstack.dev/quick-start#installation).
2. Enable Kubernetes in OrbStack:
1. Open the OrbStack app
2. Go to Settings... (`Cmd ⌘ + ,`) → Kubernetes
3. Toggle the `Enable Kubernetes cluster` option
4. Click the `Apply and Restart` button
3. Verify that OrbStack is running.
```sh
$ orb status
# Running
```
4. Verify the Kubernetes context.
```sh
$ kubectl config get-contexts orbstack
# CURRENT NAME CLUSTER AUTHINFO NAMESPACE
# * orbstack orbstack orbstack
```

### Helm

Install Helm 3:
Expand All @@ -323,6 +347,7 @@ pip3 install -r tests/requirements.txt
- `testflows.texts==2.0.211217.1011222` - Text utilities
- `PyYAML==6.0.1` - YAML parsing
- `requests==2.32.3` - HTTP requests
- `cryptography==46.0.5` - TLS validation

---

Expand All @@ -334,14 +359,18 @@ To run the complete test suite (all active fixtures + upgrades):

```bash
# From the repository root
# With Minikube (default)
python3 ./tests/run/smoke.py

# With OrbStack
LOCAL_K8S_PROVIDER=orbstack python3 ./tests/run/smoke.py
```

This will:
1. Start/restart Minikube with 4 CPUs and 6GB memory
1. \[Minikube only\] Start/restart Minikube with 4 CPUs and 6GB memory
2. Run all enabled fixture deployments
3. Run upgrade scenarios
4. Clean up and delete Minikube
4. \[Minikube only\] Clean up and delete Minikube

**Expected Duration**: 10 minutes

Expand Down
98 changes: 98 additions & 0 deletions tests/fixtures/10-tls.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
# Single-node deployment with TLS and load-balancer service
# Tests: Basic deployment with TLS using inline cert + secret refs, no keeper, minimal config
# Expected pods: 1 ClickHouse
clickhouse:
replicasCount: 1
shardsCount: 1

defaultUser:
password: "TestTLSPassword123"
allowExternalAccess: true

persistence:
enabled: true
size: 2Gi
accessMode: ReadWriteOnce

service:
type: ClusterIP

lbService:
enabled: true

settings:
https_port: 8444

configurationFiles:
# To regenerate the public certificate:
# cd tests/fixtures/tls/ # this directory
# openssl req -x509 -key test-server.key -days 10950 \
# -out server.crt -config public-cert.cnf -extensions v3_req
#
# Update inlineFileContent with the new certificate content for
# the public certificate to be used in the TLS test fixture.
#
# To verify the public certificate:
# openssl x509 -in server.crt -text -noout
#
# To verify that the private key & public certificate moduli match:
# openssl x509 -noout -modulus -in server.crt | openssl md5
# openssl rsa -noout -modulus -in test-server.key | openssl md5
config.d/foo.crt: |
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIUXWU1ixzpK9OTqFfA+ZhK/EOtLIYwDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTKi5zdmMuY2x1c3Rlci5sb2NhbDAgFw0yNjAyMTQwMTIz
NTNaGA8yMDU2MDIwNzAxMjM1M1owHjEcMBoGA1UEAwwTKi5zdmMuY2x1c3Rlci5s
b2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAIvGupZ9FR3DLC1R
/+7hbUySKYSIYOnfZQBnfkMVyLBDdp1WX9aspJLEaazO0rU4l0YjqnLsnckuBxmr
OOzzeNA+8ExBkPEANR/mROMIcwXhrdFO3sWH2amVncHFUxspwgDhbZJ0zfVNtQo0
Q/JthTWGqYW+4HbDnzOWWkUo23oZcClyELTbhQitxgrsOUyDIcR2ZNae3yueVAoK
F12fH4Sms75FLvwwlUuWU3F1lJKr/U7nPxBdl6CY/sPXITov2LcmwlQLebCchjVB
3kEvPKJRBPmW0Dyrr9IRyyExU3qfYdzsJdZOHY/qOB0Dw42qXolkf2L5m6GtW3EF
hnqM5SsCAwEAAaN2MHQwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMB
MDEGA1UdEQQqMCiCFSouKi5zdmMuY2x1c3Rlci5sb2NhbIIJbG9jYWxob3N0hwR/
AAABMB0GA1UdDgQWBBQzPiDb07CH5dKWb5xYa1PoGtz45jANBgkqhkiG9w0BAQsF
AAOCAQEActxS3ySYtlVmykOlaqTaTj8wyZjQ+gFBQ9APEuKwT4+F2HcuIRWWxJIT
Pt+CigSMG0XGFQi/+ZRfksXSmfcmErMgDUEr7jsxwvqn6esbAvHX02Vk82oAGkKu
t3JVxRAz9Dsl1Hm0W+IAGO336QFwd7iTo367TKSz+jcyHRfnftEZnxtbIvqAkR7V
8z1NRYhRR5ZeLpwjiqwdAU4AgXKQifQWeWjPeJJr+4pEvy2ivpzhhalX7/tB1TuP
4hpQdRi/c+0h5pDtrBpW61gUb0xVWySIkMifLLXfLNjynLKBJmWG91M34fTK1O8+
LYd+XhJEhXtFt8+3kktghDcehcuK7g==
-----END CERTIFICATE-----

bar.key:
valueFrom:
secretKeyRef:
name: clickhouse-certs
key: server.key

dhparam.pem:
valueFrom:
secretKeyRef:
name: clickhouse-certs
key: dhparam.pem

config.d/openssl.xml: |
<clickhouse>
<openSSL>
<server>
<certificateFile>/etc/clickhouse-server/config.d/foo.crt</certificateFile>
<privateKeyFile>/etc/clickhouse-server/secrets.d/bar.key/clickhouse-certs/server.key</privateKeyFile>
<dhParamsFile>/etc/clickhouse-server/secrets.d/dhparam.pem/clickhouse-certs/dhparam.pem</dhParamsFile>
<verificationMode>relaxed</verificationMode>
<loadDefaultCAFile>true</loadDefaultCAFile>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
</server>
</openSSL>
</clickhouse>

keeper:
enabled: false

operator:
enabled: true


13 changes: 13 additions & 0 deletions tests/fixtures/tls/dhparam.pem
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-----BEGIN DH PARAMETERS-----
MIICDAKCAgEA7QR588LscfM+0JHrq22Xc13COYt5l2p+SqWKAijcdN6n7TWpaSVQ
3N1Lj+EFt2sXZ3NpNtfv8YAMZRdbAd1oe+kquGuugtxmUUUGUXiXqeIS6Gp7bGAj
5WWobY7WtkD/HsGrvI6kxhTk3nXIEYolUHMJGb0yVLcvYR73j5F6K3ONfd107T49
/8PoVOr3ZcoBymfQK/a/mNVADKPPQ/ALAHjpIZEkQlCEj9Jw4Osaro0BEySoKJhK
5lIybQ5TJO023r9rpbKNxILaRy5esq4Vir3tlPb9eKumte6X4HFvTU36aTp5ZX/m
Ef25jhGRxnkH/N/WDEHXZPOToqyNJzdlmhvZjLj+Ru2SknS+pAZ9ZbbovzG1qPhW
BxFwotZLmTaD1+Xhm374HEY8PGeMytnrRq5W5oMpzY9PbDL+MhwxwChvWMpfPmbE
YN+InQjWNrw+C/VGLwyiOsQRhKnsCJSNckDv4cDOKuaIajhInnjGrQn7c51X2qT/
8ScJ18FLrokEw/n+61xo4TFq7L9RSddiWbaTLvXrX6ZJvE/G0APA7eDeSN/p83TV
/pYgtiHOsgaSQ8qMFAIa03hdzfw/XqA8DTu5gf6JbV9BcPJ/381Kv3oC53I7XjQP
ZfpINvFBEs1Ss4fULqnQ3V65DktpS2HhC2gDVlw+084dSy6cnXX7pv8CAQICAgFF
-----END DH PARAMETERS-----
24 changes: 24 additions & 0 deletions tests/fixtures/tls/public-cert.cnf
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = *.svc.cluster.local

[v3_req]
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
# Wildcard pattern covers all Kubernetes service DNS names:
# - t10-tls-service.t10-tls.svc.cluster.local
# - t01-tls-clickhouse.t01-tls.svc.cluster.local
# - chi-<release>-<cluster>-<shard>-<replica>.<namespace>.svc.cluster.local
# This makes the certificate resilient to test fixture renames.
DNS.1 = *.*.svc.cluster.local

# Localhost for local testing
DNS.2 = localhost
IP.1 = 127.0.0.1
20 changes: 20 additions & 0 deletions tests/fixtures/tls/server.crt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIUXWU1ixzpK9OTqFfA+ZhK/EOtLIYwDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTKi5zdmMuY2x1c3Rlci5sb2NhbDAgFw0yNjAyMTQwMTIz
NTNaGA8yMDU2MDIwNzAxMjM1M1owHjEcMBoGA1UEAwwTKi5zdmMuY2x1c3Rlci5s
b2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAIvGupZ9FR3DLC1R
/+7hbUySKYSIYOnfZQBnfkMVyLBDdp1WX9aspJLEaazO0rU4l0YjqnLsnckuBxmr
OOzzeNA+8ExBkPEANR/mROMIcwXhrdFO3sWH2amVncHFUxspwgDhbZJ0zfVNtQo0
Q/JthTWGqYW+4HbDnzOWWkUo23oZcClyELTbhQitxgrsOUyDIcR2ZNae3yueVAoK
F12fH4Sms75FLvwwlUuWU3F1lJKr/U7nPxBdl6CY/sPXITov2LcmwlQLebCchjVB
3kEvPKJRBPmW0Dyrr9IRyyExU3qfYdzsJdZOHY/qOB0Dw42qXolkf2L5m6GtW3EF
hnqM5SsCAwEAAaN2MHQwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMB
MDEGA1UdEQQqMCiCFSouKi5zdmMuY2x1c3Rlci5sb2NhbIIJbG9jYWxob3N0hwR/
AAABMB0GA1UdDgQWBBQzPiDb07CH5dKWb5xYa1PoGtz45jANBgkqhkiG9w0BAQsF
AAOCAQEActxS3ySYtlVmykOlaqTaTj8wyZjQ+gFBQ9APEuKwT4+F2HcuIRWWxJIT
Pt+CigSMG0XGFQi/+ZRfksXSmfcmErMgDUEr7jsxwvqn6esbAvHX02Vk82oAGkKu
t3JVxRAz9Dsl1Hm0W+IAGO336QFwd7iTo367TKSz+jcyHRfnftEZnxtbIvqAkR7V
8z1NRYhRR5ZeLpwjiqwdAU4AgXKQifQWeWjPeJJr+4pEvy2ivpzhhalX7/tB1TuP
4hpQdRi/c+0h5pDtrBpW61gUb0xVWySIkMifLLXfLNjynLKBJmWG91M34fTK1O8+
LYd+XhJEhXtFt8+3kktghDcehcuK7g==
-----END CERTIFICATE-----
28 changes: 28 additions & 0 deletions tests/fixtures/tls/test-server.key
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCLxrqWfRUdwywt
Uf/u4W1MkimEiGDp32UAZ35DFciwQ3adVl/WrKSSxGmsztK1OJdGI6py7J3JLgcZ
qzjs83jQPvBMQZDxADUf5kTjCHMF4a3RTt7Fh9mplZ3BxVMbKcIA4W2SdM31TbUK
NEPybYU1hqmFvuB2w58zllpFKNt6GXApchC024UIrcYK7DlMgyHEdmTWnt8rnlQK
Chddnx+EprO+RS78MJVLllNxdZSSq/1O5z8QXZegmP7D1yE6L9i3JsJUC3mwnIY1
Qd5BLzyiUQT5ltA8q6/SEcshMVN6n2Hc7CXWTh2P6jgdA8ONql6JZH9i+ZuhrVtx
BYZ6jOUrAgMBAAECggEAA/WLassnjPdD9L4CMixVIeUbTfNlo1o+NyZh+i56HRwG
wRV66M7CaUcs+HBx9st1Os8J0JregUikz4JSaMvLW+1cdccpqYSS/KX+GjGEEzeS
6n8sajXwjBApLsg9vrg5FDijwjvoFY8E62wS50wDiOzujP5HcgsVIwCa6qM/TD8K
KVFBwLe9mL9yrGlSPCoj0vvt1R5VbEJh7PeYGfjUl30r0lWmP8ZlN6u2Q+aafEdy
p3gIHPDRRv6vPSiR24tVQgv0FaTnXRAjYtwhNqlXPMhtHnBEtNA1kuGE9IZimPw/
6P3L3SkbGKL5TRZL6wRfrYada6TeFX9ueE45ES2UuQKBgQDBK/2nCvQr6wlF45kb
cQ7iunAcu5/DAlqQEXsOoef+WNNzIPxHCTH7YW4GhAUUSYyorLWi20tknrKJrITN
IotSVgItzpfia7HB3EIuSBysMOS81W+R3HYRoigQigT8YwxiRSACf6x1O5J0J/jC
tq2Galhwz8ILWaJ5SdACv+sBnQKBgQC5POFBz/GaDFooMFXU4kBw5e1YjkNkEQUh
bFQL9wZBiP2Xv1QiRV+kbrHwlW1p9gUjOogWz05L2z7qPbtdaIV6MPwEgr2xBCt5
mkuSWhUDtzQ2pVFsSY75SOg8CE0lHnHSyUEaPhac06qn3KShUcakpm5A684aUjjy
1IH0huuLZwKBgAdBvc+uq6mStNB5UmEjiCmgU2Hg8omC5yAOaA8OqgZ2E8t5a8DH
aadF67o273HpqW0Uv+YUUuq+w3pEjuCd8ZnwPTi3UCFjZlQgECRo9RrK42zsn7pd
C9pxuwuUA8fveKGgcylk3new+zl93uyBrFcmW5gxVdrTTTU9PqE70HpJAoGAAaDH
Wgy50uDI6hGCr5xNdLCQpXaaoQaFRQXutyw0od7SW8MSujph3NAcQEEP9R50bRrW
l1y7E2+Z3fUs8GU6xxgnHuMHR8cBmtAAWgjwple13cUWMh1zZD1/zQdFpk3eMjwS
lmh1SmuR1GfcCo7tcAUGcwufhBu05G15tux4pYECgYEAkzun1vyNNoC2CrGR3wIO
SMwDy5JxZLIDAeAkidHsDrGzYzOekHLrHR2r47F4mJPUyiGVNCIz69fiVkcngshr
sEeYFP5cN6Ip7chEp683xVBbKM+jsQtMmhadaL5tsulqgScPquCap+dPvaq/ogJ1
2jbKCT447cR8wtviH/OnOgY=
-----END PRIVATE KEY-----
3 changes: 2 additions & 1 deletion tests/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
requests==2.32.3
testflows==2.4.13
testflows.texts==2.0.211217.1011222
PyYAML==6.0.1
PyYAML==6.0.1
cryptography==46.0.5
Loading