diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e043fa85a..f7bea34c1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,46 @@ +## Setup + +### Setup local db link as environment variable. + +```bash +export DB_URL=postgres://@localhost:5432/config +``` + +### Create `config` database. + +```sql +create database config +``` + +### Scape config and serve + +Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified). + +```bash +make build + +./.bin/config-db serve +``` + +To explicitly run scraping with a particular configuration: + +```bash +./.bin/config-db run -vvv +config-db serve +``` + +See `fixtures/` for example scraping configurations. + +### Migrations + +Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations. + +Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory. + +```bash +GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down +``` + ## Adding a new scraper 1. Create a new file in `scrapers/` which implements the `api/v1/Scraper` interface diff --git a/README.md b/README.md index d53f8435c..b08ca6543 100644 --- a/README.md +++ b/README.md @@ -2,59 +2,111 @@ **config-db** is developer first, JSON based configuration management database (CMDB). -## Setup +## Principles -### Setup local db link as environment variable. +* **JSON Based** - Configuration is stored in JSON, with changes recorded as JSON patches that enables highly structured search. +* **SPAM Free** - Not all configuration data is useful, and overly verbose change histories are difficult to navigate. +* **GitOps Ready** - Configuration should be stored in Git, config-db enables the extraction of configuration out of Git repositories with branch/environment awareness. +* **Topology Aware** - Configuration can often have an inheritance or override hierarchy. -```bash -export DB_URL=postgres://@localhost:5432/config -``` +## Capabilities -### Create `config` database. +* View and search change history in any dimension (node, zone, environment, application, technology) +* Compare and diff configuration across environments. -```sql -create database config -``` +## Quick Start -### Scape config and serve +Before installing Config-DB , please ensure you have the [prerequisites installed](docs/prereqs.md) on your Kubernetes cluster. -Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified). +The recommended method for installing Config-DB is using [helm](https://helm.sh/) -```bash -make build +### Install Helm + +The following steps will install the latest version of helm -./.bin/config-db serve +```bash +curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 +chmod 700 get_helm.sh +./get_helm.sh ``` -To explicitly run scraping with a particular configuration: +### Add the Flanksource helm repository ```bash -./.bin/config-db run -vvv -config-db serve +helm repo add flanksource https://flanksource.github.io/charts +helm repo update ``` -See `fixtures/` for example scraping configurations. +### Configurable fields + +See the [values file](chart/values.yaml) for the full list of configurable fields. Mandatory configuration values are for the configuration of the database, and it is recommended to also configure the UI ingress. + +#### DB + +ConfigDB requires a Postgres server to function. A basic postgres server can be installed by the helm chart. + +##### Chart-managed Server + +| | | +|---------------------|--------| +| db.external.create | `true` | +| db.external.storageClass | Set to name of a storageclass available in the cluster | +| db.external.storage | Set to volume of storage to request | -### Migrations +The helm chart will create a postgres server statefulset, with a random password and default port, along with a configdb database hosted on the server. -Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations. +To specify a username and password for the chart-managed Postgres server, create a secret in the namespace that the chart will install to, named `postgres-connection`, which contains `POSTGRES_USER` and `POSTGRES_PASSWORD` keys. If no pre-existing secret is created, a user called 'postgres' will be given a random password. -Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory. +##### Prexisting Server + +In order to connect to an existing Postgres server, a database must be created on the server, along with a user that has admin permissions + +| | | +|---------------------|---------| +| db.external.create | `false` | +| db.external.secretKeyRef.name | Set to name of name of secret that contains a key containging the postgres connection URI | +| db.external.secretKeyRef.key | Set to the name of the key in the secret that contains the postgres connection URI | + +The connection URI must be specified in the format `postgresql://"$user":"$password"@"$host"/"$database"` + +#### Ingress + +In order to view the ConfigDB UI, it must be exposed using an ingress: + +| | | +|---------------------|-------------------| +| ingress.host | URL at which the UI will be accessed | +| ingress.annotations | Map of annotations required by the ingress controller or certificate issuer | +| ingress.tls | Map of configuration options for TLS | + +More details regarding ingress configuration can be found in the [kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) + +### Deploy using Helm + +To install into a new `config-db` namespace, run ```bash -GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down +helm install config-db-demo --wait -n config-db --create-namespace flanksource/config-db -f values.yaml ``` -## Principles -* **JSON Based** - Configuration is stored in JSON, with changes recorded as JSON patches that enables highly structured search. -* **SPAM Free** - Not all configuration data is useful, and overly verbose change histories are difficult to navigate. -* **GitOps Ready** - Configuration should be stored in Git, config-db enables the extraction of configuration out of Git repositories with branch/environment awareness. -* **Topology Aware** - Configuration can often have an inheritance or override hierarchy. - -## Capabilities +where `values.yaml` contains the configuration options detailed above. eg + +```yaml +db: + create: true + storageClass: default + storage: 30Gi +ingress: + host: config-db.flanksource.com + annotations: + kubernetes.io/ingress.class: nginx + kubernetes.io/tls-acme: "true" + tls: + - secretName: config-db-tls + hosts: + - config-db.flanksource.com +``` -* View and search change history in any dimension (node, zone, environment, application, technology) -* Compare and diff configuration across environments. ## Configuration Sources diff --git a/chart/templates/postgres.yaml b/chart/templates/postgres.yaml index 2567cd5aa..55fb91794 100644 --- a/chart/templates/postgres.yaml +++ b/chart/templates/postgres.yaml @@ -1,4 +1,4 @@ -{{- if eq .Values.db.enabled true }} +{{- if eq .Values.db.create true }} --- # PostgreSQL StatefulSet @@ -52,7 +52,6 @@ spec: --- -{{- if .Values.db.secretKeyRef.create }} apiVersion: v1 kind: Secret metadata: @@ -61,10 +60,14 @@ metadata: "helm.sh/resource-policy": "keep" type: Opaque stringData: - {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" ) }} + # Potential existing secret with user requested login details to inject into chart + {{- $secretInj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" }} + # Potential previously created version of this secret. Read in to prevent helm creating a new password string every deploy + {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace .Values.db.secretKeyRef.name }} + {{- $injData := ( get $secretInj "data" ) }} {{- $secretData := ( get $secretObj "data" ) }} - {{- $user := (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }} - {{- $password := (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }} + {{- $user := (( get $injData "POSTGRES_USER" ) | b64dec ) | default (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }} + {{- $password := (( get $injData "POSTGRES_PASSWORD" ) | b64dec ) | default (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }} {{- $host := print "postgres." .Release.Namespace ".svc.cluster.local:5432" }} {{- $url := print "postgresql://" $user ":" $password "@" $host }} {{- $configDbUrl := ( get $secretData .Values.db.secretKeyRef.key ) | default ( print $url "/config-db" ) }} @@ -74,8 +77,6 @@ stringData: POSTGRES_HOST: {{ $host | quote }} {{ .Values.db.secretKeyRef.key }}: {{ $configDbUrl | quote }} -{{- end }} - --- {{- end }} diff --git a/chart/values.yaml b/chart/values.yaml index 6470a058b..7bc04b95c 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -20,14 +20,14 @@ image: scrapeRuleConfigMaps: - config-db-rules db: - # Setting this to true will create a postgres stateful set for config-db to connect to. - enabled: true + # Setting create to true will create a postgres stateful set for config-db to connect to. + # If create=true, the secretKeyRef will be created by helm with the specified name and key + # Optionally a secret named 'postgres-connection' with POSTGRES_USER and POSTGRES_PASSWORD to set the created username and password, otherwise a random password will be created for a 'postgres' user + # If create=false, a prexisting secret containing the URI to an existing postgres database must be provided + # The URI must be in the format 'postgresql://"$user":"$password"@"$host"/"$database"' + create: true secretKeyRef: - create: true - # Setting the name of the secret will disable secret creation in this chart and look for an existing secret (whose name is specified in this field) to mount. - # When setting this up in a fresh environment as a standalone app, it's best to leave the value empty. - name: - # This is the key that either the secret will create(if secretRefKey is empty) or this is the key it'll look for in the secret(if secretRefKey is mentioned). + name: config-db-postgres # The name of the key is mandatory to set. key: DB_URL storageClass: