Operator Investigation | Run W&B pods as nonRoot and unprivileged

[abhinavg6]

IMPORTANT: This task is only to investigate what is possible and to identify what needs to be changed.

Today:

W&B requires root privileges to run. In (shared) enterprise environments that is not allowed.

Examples:

Starting wandb-app as nonRoot and unprivileged results in the following:

❯ k logs wandb-app-7447fdd8b6-hnb6x 
Defaulted container "app" out of: app, init-db (init)
*** Killing all processes...
Traceback (most recent call last):
  File "/sbin/my_init", line 475, in <module>
    main(args)
  File "/sbin/my_init", line 352, in main
    export_envvars()
  File "/sbin/my_init", line 125, in export_envvars
    with open("/etc/container_environment/" + name, "w") as f:
PermissionError: [Errno 13] Permission denied: '/etc/container_environment/LANG'

or

❯ k logs wandb-app-59b7745dd5-xc4ks
Defaulted container "app" out of: app, init-db (init)
*** Killing all processes...
Traceback (most recent call last):
  File "/sbin/my_init", line 475, in <module>
    main(args)
  File "/sbin/my_init", line 350, in main
    write_envvars_to_file()
  File "/sbin/my_init", line 91, in write_envvars_to_file
    os.makedirs("/etc/original_variables")
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/etc/original_variables'

Expectation:

The following SecurityContext should be possible to configure and all W&B pods run without issues:

runAsNonRoot: true
allowPrivilegeEscalation: false 
privileged: false
runAsUser: 2000
runAsGroup: 2000
readOnlyRootFilesystem: true 
capabilities: 
  drop: 
    - ALL

The result of this tasks should be a list of changes that are required to be implemented to make the above security context possible.

[amanpruthi]

1. Update config.go to Use Non-Root Writable Paths

In config.go file, ensure that paths used by the application point to directories where the non-root user (UID: 2000) has write access.

Link : https://github.com/wandb/core/blob/81affcd2206ce554cb12c35317c75632abc140e2/services/local/util/config.go#L34

var DefaultConfig = LocalConfig{
    ContainerEnvironmentPath:   "/home/wandb/container_environment",
    EnvironmentDefaultsPath:    "/home/wandb/environment_defaults/env.txt",
    OriginalVariablesPath:      "/home/wandb/original_variables/env.txt",
    UserSettingsErrorCachePath: "/home/wandb/user_settings_error_cache.json",
}

2. Update Dockerfile to Ensure Permissions for Non-Root User

Ensure the /home/wandb directory is owned by the non-root user (UID: 2000)
Link: https://github.com/wandb/core/blob/81affcd2206ce554cb12c35317c75632abc140e2/onprem/local/Dockerfile

RUN chown -R 2000:2000 /home/wandb
RUN chmod -R 755 /home/wandb

[abhinavg6]

@flamarion and @mkaesz-wandb to discuss if this is needed