Support enabling etcd authentication

[justinwalz]

Refer: https://etcd.io/docs/v3.6/op-guide/authentication/rbac/

Summary

My long-term goal is to support operator managed role and user via EtcdRole and EtcdUser.

However, the current state of the project doesn't have supporting functionality yet, as auth isn't managed even for the root user. This issue is the first step in that direction.

The operator should support enabling etcd's built-in authentication system on managed clusters. This includes creating the root user/role, calling auth enable, and managing the root credentials in a Kubernetes Secret.

Approaches

Option A: Auth field on EtcdCluster spec

We add directly to the existing spec. We have an enabled toggle for off/on, and a reference to a kubernetes secret that contains the admin credentials. If this already exists, the operator should use it. If not, the operator should create and use it.

apiVersion: operator.etcd.io/v1alpha1
kind: EtcdCluster      
metadata:
  name: my-cluster
spec:               
  size: 3    
  auth:
    enabled: true                                                                                          
    secretRef:
      name: my-cluster-root-credentials  

When spec.auth.enabled=true, the operator would follow the Enabling authentication steps... creating the root user, and enabling auth.

Option B: Separate EtcdAuth CRD

We introduce a standalone CRD that targets an existing cluster. Here there is no enabled boolean. If the resource exists, we enable auth. If not, we take no action.

apiVersion: operator.etcd.io/v1alpha1                                                                      
kind: EtcdAuth
metadata: 
  name: my-cluster-auth
spec:                                                                                                      
  clusterRef:                                                                                              
    name: my-cluster                                                                                       
  secretRef:
    name: my-cluster-root-credentials

A dedicated controller would handle the auth enable flow independently, with similar logic to above. This might be nice for separation of concerns, but has some ordering dependency as the cluster must be healthy before auth can be enabled, then the cluster controller needs to also read the EtcdAuth resource to get it's own credentials after auth has been enabled.

Secret Ref

Both approaches use a Kubernetes Secret to store the root password.
If the user wants to pre-create the secret, they can, and the operator will just use the existing secret.
If the user doesn't care, then the operator will autogenerate a password and create the secret.
All subsequent calls will just use the secret content.

The username is always root.

Notes

Existing functionality needs to take auth status into account.

When bootstrapping a cluster prior to auth being enabled, etcd client shouldn't be created with any auth info. Once auth is enabled, all clients must be created with the auth info.

From prior experience, the AuthStatus call doesn't return a clean response. My current approach is

• try AuthStatus() with an unauthenticated client
  • If succeeds, then the result is also false, auth is not enabled.
  • If it fails with "user name is empty", then auth is likely enabled. Optionally (defensive) confirm by creating an authenticated client and calling AuthStatus() again.

[ahrtr]

My immediate feeling is that it's users' responsibility to manage roles, users, auth enable or disable etc, because they are runtime data of etcd cluster.

The problem is that if users enable auth on the etcd cluster, then etcd-operator won't be able to talk to the etcd cluster. So we have to extend the EtcdCluster CRD to let users to configure auth info for etcd-operator to use to talk to the etcd cluster. From this perspective, Option A makes sense. But etcd-operator shouldn't manage/create all etcd's roles and users.

From prior experience, the AuthStatus call doesn't return a clean response. My current approach is

FYI. This has been resolved on etcd side in v3.7. etcd-io/etcd#20802

[jberkus]

Agreed that auth itself needs to be part of CRD cluster config, but roles/users shouldn't be. Although ... will the operator need to seed an initial user role, so that the user can log in?

[ahrtr]

will the operator need to seed an initial user role, so that the user can log in?

I think so, otherwise we will run into a chicken-egg dead lock.

[ahrtr]

Contribution is welcome!