From 5ce5a056ebb91efed86ec55bf41920328255ee6b Mon Sep 17 00:00:00 2001 From: Brendan Galloway Date: Tue, 10 Jan 2023 11:01:50 +0200 Subject: [PATCH 1/5] docs: move developer setup to contributing.md file --- CONTRIBUTING.md | 43 +++++++++++++++++++++++++++++++++++++++++++ README.md | 42 ------------------------------------------ 2 files changed, 43 insertions(+), 42 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e043fa85a..f7bea34c1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,46 @@ +## Setup + +### Setup local db link as environment variable. + +```bash +export DB_URL=postgres://@localhost:5432/config +``` + +### Create `config` database. + +```sql +create database config +``` + +### Scape config and serve + +Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified). + +```bash +make build + +./.bin/config-db serve +``` + +To explicitly run scraping with a particular configuration: + +```bash +./.bin/config-db run -vvv +config-db serve +``` + +See `fixtures/` for example scraping configurations. + +### Migrations + +Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations. + +Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory. + +```bash +GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down +``` + ## Adding a new scraper 1. Create a new file in `scrapers/` which implements the `api/v1/Scraper` interface diff --git a/README.md b/README.md index d53f8435c..09a1faf74 100644 --- a/README.md +++ b/README.md @@ -2,48 +2,6 @@ **config-db** is developer first, JSON based configuration management database (CMDB). -## Setup - -### Setup local db link as environment variable. - -```bash -export DB_URL=postgres://@localhost:5432/config -``` - -### Create `config` database. - -```sql -create database config -``` - -### Scape config and serve - -Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified). - -```bash -make build - -./.bin/config-db serve -``` - -To explicitly run scraping with a particular configuration: - -```bash -./.bin/config-db run -vvv -config-db serve -``` - -See `fixtures/` for example scraping configurations. - -### Migrations - -Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations. - -Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory. - -```bash -GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down -``` ## Principles * **JSON Based** - Configuration is stored in JSON, with changes recorded as JSON patches that enables highly structured search. From 6333fcf0171396544c5d9cb901362ffeb9c5cb32 Mon Sep 17 00:00:00 2001 From: Brendan Galloway Date: Tue, 10 Jan 2023 14:05:49 +0200 Subject: [PATCH 2/5] docs: add install guide and align values file keys --- README.md | 90 +++++++++++++++++++++++++++++++++++ chart/templates/postgres.yaml | 7 +-- chart/values.yaml | 14 +++--- 3 files changed, 99 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 09a1faf74..d60afdd46 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,96 @@ * View and search change history in any dimension (node, zone, environment, application, technology) * Compare and diff configuration across environments. +## Quick Start + +Before installing the Canary Checker, please ensure you have the [prerequisites installed](docs/prereqs.md) on your Kubernetes cluster. + +The recommded method for installing Canary Checker is using [helm](https://helm.sh/) + +### Install Helm + +The following steps will install the latest version of helm + +```bash +curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 +chmod 700 get_helm.sh +./get_helm.sh +``` + +### Add the Flanksource helm repository + +```bash +helm repo add flanksource https://flanksource.github.io/charts +helm repo update +``` + +### Configurable fields + +See the [values file](chart/values.yaml) for the full list of configurable fields. Mandatory configuration values are for the configuration of the database, and it is recommended to also configure the UI ingress. + +#### DB + +ConfigDB requires a Postgres server to function. A basic postgres server can be installed by the helm chart. + +#### Chart-managed Server + +| db.external.create | `true` | +| db.external.storageClass | Set to name of a storageclass available in the cluster | +| db.external.storage | Set to volume of storage to request | + +The helm chart will create a postgres server statefulset, with a random password and default port, along with a canarychecker database hosted on the server. + +To specify a username and password for the chart-managed Postgres server, create a secret in the namespace that the chart will install to, named `postgres-connection`, which contains `POSTGRES_USER` and `POSTGRES_PASSWORD` keys. If no pre-existing secret is created, a user call'ed 'postgres' will be given a random password. + +#### Prexisting Server + +In order to connect to an existing Postgres server, a database must be created on the server, along with a user that has admin permissions + +| db.external.create | `false` | +| db.external.secretKeyRef.name | Set to name of name of secret that contains a key containging the postgres connection URI | +| db.external.secretKeyRef.key | Set to the name of the key in the secret that contains the postgres connection URI | + +The connection URI must be specified in the format `postgresql://"$user":"$password"@"$host"/"$database"` + +#### Ingress + +In order to view the ConfigDB UI, it must be exposed using an ingress: + +| | | +|---------------------|-------------------| +| ingress.host | URL at which the UI will be accessed | +| ingress.annotations | Map of annotations required by the ingress controller or certificate issuer | +| ingress.tls | Map of configuration options for TLS | + +More details regarding ingress configuration can be found in the [kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) + +### Deploy using Helm + +To install into a new `config-db` namespace, run + +```bash +helm install config-db-demo --wait -n config-db --create-namespace flanksource/config-db -f values.yaml +``` + +where `values.yaml` contains the configuration options detailed above. eg + +```yaml +db: + create: true + storageClass: default + storage: 30Gi +ingress: + host: config-db.flanksource.com + annotations: + kubernetes.io/ingress.class: nginx + kubernetes.io/tls-acme: "true" + tls: + - secretName: config-db-tls + hosts: + - config-db.flanksource.com +``` + + ## Configuration Sources * AWS diff --git a/chart/templates/postgres.yaml b/chart/templates/postgres.yaml index 2567cd5aa..ad403ff70 100644 --- a/chart/templates/postgres.yaml +++ b/chart/templates/postgres.yaml @@ -1,4 +1,4 @@ -{{- if eq .Values.db.enabled true }} +{{- if eq .Values.db.create true }} --- # PostgreSQL StatefulSet @@ -52,7 +52,6 @@ spec: --- -{{- if .Values.db.secretKeyRef.create }} apiVersion: v1 kind: Secret metadata: @@ -61,7 +60,7 @@ metadata: "helm.sh/resource-policy": "keep" type: Opaque stringData: - {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" ) }} + {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" }} {{- $secretData := ( get $secretObj "data" ) }} {{- $user := (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }} {{- $password := (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }} @@ -74,8 +73,6 @@ stringData: POSTGRES_HOST: {{ $host | quote }} {{ .Values.db.secretKeyRef.key }}: {{ $configDbUrl | quote }} -{{- end }} - --- {{- end }} diff --git a/chart/values.yaml b/chart/values.yaml index 6470a058b..7bc04b95c 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -20,14 +20,14 @@ image: scrapeRuleConfigMaps: - config-db-rules db: - # Setting this to true will create a postgres stateful set for config-db to connect to. - enabled: true + # Setting create to true will create a postgres stateful set for config-db to connect to. + # If create=true, the secretKeyRef will be created by helm with the specified name and key + # Optionally a secret named 'postgres-connection' with POSTGRES_USER and POSTGRES_PASSWORD to set the created username and password, otherwise a random password will be created for a 'postgres' user + # If create=false, a prexisting secret containing the URI to an existing postgres database must be provided + # The URI must be in the format 'postgresql://"$user":"$password"@"$host"/"$database"' + create: true secretKeyRef: - create: true - # Setting the name of the secret will disable secret creation in this chart and look for an existing secret (whose name is specified in this field) to mount. - # When setting this up in a fresh environment as a standalone app, it's best to leave the value empty. - name: - # This is the key that either the secret will create(if secretRefKey is empty) or this is the key it'll look for in the secret(if secretRefKey is mentioned). + name: config-db-postgres # The name of the key is mandatory to set. key: DB_URL storageClass: From 3356372c1d2674dac1a69402e814f65d18054917 Mon Sep 17 00:00:00 2001 From: Brendan Galloway Date: Tue, 24 Jan 2023 11:34:24 +0200 Subject: [PATCH 3/5] docs: fix review comments --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d60afdd46..a5561c6b8 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,9 @@ ## Quick Start -Before installing the Canary Checker, please ensure you have the [prerequisites installed](docs/prereqs.md) on your Kubernetes cluster. +Before installing Config-DB , please ensure you have the [prerequisites installed](docs/prereqs.md) on your Kubernetes cluster. -The recommded method for installing Canary Checker is using [helm](https://helm.sh/) +The recommended method for installing Config-DB is using [helm](https://helm.sh/) ### Install Helm @@ -47,6 +47,8 @@ ConfigDB requires a Postgres server to function. A basic postgres server can be #### Chart-managed Server +| | | +|---------------------|--------| | db.external.create | `true` | | db.external.storageClass | Set to name of a storageclass available in the cluster | | db.external.storage | Set to volume of storage to request | @@ -59,6 +61,8 @@ To specify a username and password for the chart-managed Postgres server, create In order to connect to an existing Postgres server, a database must be created on the server, along with a user that has admin permissions +| | | +|---------------------|---------| | db.external.create | `false` | | db.external.secretKeyRef.name | Set to name of name of secret that contains a key containging the postgres connection URI | | db.external.secretKeyRef.key | Set to the name of the key in the secret that contains the postgres connection URI | From 314061d5887865753aa9b7a51e444ccd0b28dbe5 Mon Sep 17 00:00:00 2001 From: Brendan Galloway Date: Tue, 24 Jan 2023 11:40:41 +0200 Subject: [PATCH 4/5] fix: postgres secret creation logic --- chart/templates/postgres.yaml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/chart/templates/postgres.yaml b/chart/templates/postgres.yaml index ad403ff70..55fb91794 100644 --- a/chart/templates/postgres.yaml +++ b/chart/templates/postgres.yaml @@ -60,10 +60,14 @@ metadata: "helm.sh/resource-policy": "keep" type: Opaque stringData: - {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" }} + # Potential existing secret with user requested login details to inject into chart + {{- $secretInj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" }} + # Potential previously created version of this secret. Read in to prevent helm creating a new password string every deploy + {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace .Values.db.secretKeyRef.name }} + {{- $injData := ( get $secretInj "data" ) }} {{- $secretData := ( get $secretObj "data" ) }} - {{- $user := (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }} - {{- $password := (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }} + {{- $user := (( get $injData "POSTGRES_USER" ) | b64dec ) | default (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }} + {{- $password := (( get $injData "POSTGRES_PASSWORD" ) | b64dec ) | default (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }} {{- $host := print "postgres." .Release.Namespace ".svc.cluster.local:5432" }} {{- $url := print "postgresql://" $user ":" $password "@" $host }} {{- $configDbUrl := ( get $secretData .Values.db.secretKeyRef.key ) | default ( print $url "/config-db" ) }} From f6cb32faf815bc22e799b77890fd4f6a60baefe9 Mon Sep 17 00:00:00 2001 From: Brendan Galloway Date: Fri, 27 Jan 2023 10:08:24 +0200 Subject: [PATCH 5/5] docs: fix typos and formatting --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a5561c6b8..b08ca6543 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ See the [values file](chart/values.yaml) for the full list of configurable field ConfigDB requires a Postgres server to function. A basic postgres server can be installed by the helm chart. -#### Chart-managed Server +##### Chart-managed Server | | | |---------------------|--------| @@ -53,11 +53,11 @@ ConfigDB requires a Postgres server to function. A basic postgres server can be | db.external.storageClass | Set to name of a storageclass available in the cluster | | db.external.storage | Set to volume of storage to request | -The helm chart will create a postgres server statefulset, with a random password and default port, along with a canarychecker database hosted on the server. +The helm chart will create a postgres server statefulset, with a random password and default port, along with a configdb database hosted on the server. -To specify a username and password for the chart-managed Postgres server, create a secret in the namespace that the chart will install to, named `postgres-connection`, which contains `POSTGRES_USER` and `POSTGRES_PASSWORD` keys. If no pre-existing secret is created, a user call'ed 'postgres' will be given a random password. +To specify a username and password for the chart-managed Postgres server, create a secret in the namespace that the chart will install to, named `postgres-connection`, which contains `POSTGRES_USER` and `POSTGRES_PASSWORD` keys. If no pre-existing secret is created, a user called 'postgres' will be given a random password. -#### Prexisting Server +##### Prexisting Server In order to connect to an existing Postgres server, a database must be created on the server, along with a user that has admin permissions