diff --git a/CHANGELOG.md b/CHANGELOG.md index d991da02fa..72284ad5bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,12 @@ and this project adheres to ### Changed +- On Linux, database connections now fail fast when their network path dies + (e.g. a TCP reset during a node-pool upgrade) instead of blocking on a dead + socket for up to ~15 minutes. `DATABASE_TCP_USER_TIMEOUT` (ms) defaults to + `DATABASE_TIMEOUT + 5s`; set it to `0` to restore the previous behaviour. No + effect on non-Linux platforms. + [#4855](https://github.com/OpenFn/lightning/pull/4855) - Consolidate email format validation onto a single canonical validator (Zod v4 regex) applied uniformly across user creation, credential transfer, and both collaborator add/invite flows. Fixes a silent inconsistency where diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 4861f89b04..25ae499857 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -183,58 +183,59 @@ For SMTP, the following environment variables are required: ### Other config -| **Variable** | Description | -| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `ADAPTORS_PATH` | Where you store your locally installed adaptors | -| `ALLOW_SIGNUP` | Set to `true` to enable user access to the registration page. Set to `false` to disable new user registrations and block access to the registration page.
Default is `true`. | -| `CORS_ORIGIN` | A list of acceptable hosts for browser/cors requests (',' separated) | -| `DISABLE_DB_SSL` | In production, the use of an SSL connection to Postgres is required by default.
Setting this to `"true"` allows unencrypted connections to the database. This is strongly discouraged in a real production environment. | -| `EMAIL_ADMIN` | This is used as the sender email address for system emails. It is also displayed in the menu as the support email. | -| `EMAIL_SENDER_NAME` | This is displayed in the email client as the sender name for emails sent by the application. | -| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_CHANNEL_NAME` | The name of the Postgresql channel that is used when Erlang node discovery via Postgres is enabled. Defaults to `lightning-cluster` if not set. | -| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED` | If set to `true`, Lightning will use Postgres to discover Erlang nodes. This strategy will be used in addition to other strategies that are in use. Default value is `false` | -| `IDLE_TIMEOUT` | The number of seconds that must pass without data being received before the Lightning web server kills the connection. | -| `IS_RESETTABLE_DEMO` | If set to `yes`, it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy _most_ of what you have in your database! | -| `K8S_HEADLESS_SERVICE` | This environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're _not_ using Kubernetes, the "gossip" strategy is used to establish clusters. | -| `LISTEN_ADDRESS` | The address the web server should bind to. Defaults to `127.0.0.1` to block access from other machines. | -| `LOG_LEVEL` | How noisy you want the logs to be (e.g., `debug`, `info`) | -| `METRICS_RUN_PERFORMANCE_AGE_SECONDS` | The oldest a run can be to be included in Run performance metrics. | -| `METRICS_RUN_QUEUE_AGE_SECONDS` | The polling period for run queue metrics. | -| `METRICS_STALLED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it is considered stalled. | -| `METRICS_UNCLAIMED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it counts towards an impeded project. | -| `MIX_ENV` | Your mix env, likely `prod` for deployment | -| `NODE_ENV` | Node env, likely `production` for deployment | -| `ORIGINS` | The allowed origins for web traffic to the backend | -| `PER_WORKFLOW_CLAIM_LIMIT` | The maximum number of runs per workflow to consider during run claiming. This prevents any single workflow from dominating the processing queue while ensuring fairness across workflows.
Default is `50`. | -| `CLAIM_WORK_MEM` | PostgreSQL `work_mem` setting for the run claim query. Helps optimize complex sorting operations. Set to a valid PostgreSQL memory value (e.g., `32MB`, `64MB`, `1GB`). Set to empty string to disable.
Default: disabled in dev/test, `32MB` in production. | -| `PORT` | The port your Phoenix app runs on | -| `PROMEX_DATASOURCE_ID` | The datasource that PromEx will use if configured to push initial dashboards to Grafana. Defaults to an empty string. | -| `PROMEX_ENABLED` | Enables PromEx tracking and publishing of metrics if set to 'true' or 'yes'. Defaults to false. | -| `PROMEX_ENDPOINT_SCHEME` | The scheme needed when connecting to the Promex Endpoint. Defaults to https. | -| `PROMEX_EXPENSIVE_METRICS_ENABLED` | Certain metrics may be expensive to generate if Lightning is under load. If set to 'true', or 'yes' these metrics will be enabled. Defaults to 'false'. | -| `PROMEX_GRAFANA_HOST` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | -| `PROMEX_GRAFANA_PASSWORD` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | -| `PROMEX_GRAFANA_USER` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | -| `PROMEX_METRICS_ENDPOINT_AUTHORIZATION_REQUIRED` | If set to 'true' or 'yes', the PromEx endpoint on Lightning will require consumers to provide credentials for authorization. Defaults to 'true'. | -| `PROMEX_METRICS_ENDPOINT_TOKEN` | A Bearer token that the consumer of the promEx endpoint must provide in the Authorization header. Defaults to a random series of bytes. | -| `PROMEX_UPLOAD_GRAFANA_DASHBOARDS_ON_START` | Instructs PromEx to upload iniital dashboards to a Grafana instance if set to 'true' or 'yes'. Defaults to false. | -| `PRIMARY_ENCRYPTION_KEY` | A base64 encoded 32 character long string.
See [Encryption](#encryption). | -| `QUEUE_RESULT_RETENTION_PERIOD_MINUTES` | The number of minutes to keep completed (successful) `ObanJobs` in the queue (not to be confused with runs and/or history) | -| `SCHEMAS_PATH` | Path to the credential schemas that provide forms for different adaptors | -| `ADAPTORS_REGISTRY_JSON_PATH` | Path to adaptor registry file. When provided, the app will attempt to read from it then later fallback to the internet | -| `SECRET_KEY_BASE` | A secret key used as a base to generate secrets for encrypting and signing data. | -| `SENTRY_DSN` | If using Sentry for error monitoring, your DSN | -| `UI_METRICS_ENABLED` | Enable serverside tracking of certain metrics related to the UI. This s temporary functionality. Defaults to `false`. | -| `URL_HOST` | The host used for writing URLs (e.g., `demo.openfn.org`) | -| `URL_PORT` | The port, usually `443` for production | -| `URL_SCHEME` | The scheme for writing URLs (e.g., `https`) | -| `USAGE_TRACKER_HOST` | The host that receives usage tracking submissions
(defaults to https://impact.openfn.org) | -| `USAGE_TRACKING_DAILY_BATCH_SIZE` | The number of days that will be reported on with each run of `UsageTracking.DayWorker`. This will only have a noticeable effect in cases where there is a backlog or where reports are being generated retroactively (defaults to 10). | -| `USAGE_TRACKING_ENABLED` | Enables the submission of anonymized usage data to OpenFn (defaults to `true`) | -| `USAGE_TRACKING_RESUBMISSION_BATCH_SIZE` | The number of failed reports that will be submitted on each resubmission run (defaults to 10) | -| `USAGE_TRACKING_RUN_CHUNK_SIZE` | The size of each batch of runs that is streamed from the database when generating UsageTracking reports (default 100). Decreasing this may decrease memory consumption when generating reports. | -| `USAGE_TRACKING_UUIDS` | Indicates whether submissions should include cleartext UUIDs or not. Options are `cleartext` or `hashed_only`, with the default being `hashed_only`. | -| `REQUIRE_EMAIL_VERIFICATION` | Indicates whether user email addresses should be verified. Defaults to `false`. | +| **Variable** | Description | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ADAPTORS_PATH` | Where you store your locally installed adaptors | +| `ALLOW_SIGNUP` | Set to `true` to enable user access to the registration page. Set to `false` to disable new user registrations and block access to the registration page.
Default is `true`. | +| `CORS_ORIGIN` | A list of acceptable hosts for browser/cors requests (',' separated) | +| `DATABASE_TCP_USER_TIMEOUT` | Milliseconds a database connection may have unacknowledged data in flight before the OS drops it, so a connection whose network path has died fails fast instead of blocking. **Linux-only** (ignored on other platforms). On Linux it defaults to `DATABASE_TIMEOUT` + 5000ms (i.e. `20000` with the default 15s query timeout), so a slow-but-live query times out at the app layer first and this only fires on a dead socket. Set explicitly to override, or to `0` to disable. A value below `DATABASE_TIMEOUT` logs a warning at boot. | +| `DISABLE_DB_SSL` | In production, the use of an SSL connection to Postgres is required by default.
Setting this to `"true"` allows unencrypted connections to the database. This is strongly discouraged in a real production environment. | +| `EMAIL_ADMIN` | This is used as the sender email address for system emails. It is also displayed in the menu as the support email. | +| `EMAIL_SENDER_NAME` | This is displayed in the email client as the sender name for emails sent by the application. | +| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_CHANNEL_NAME` | The name of the Postgresql channel that is used when Erlang node discovery via Postgres is enabled. Defaults to `lightning-cluster` if not set. | +| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED` | If set to `true`, Lightning will use Postgres to discover Erlang nodes. This strategy will be used in addition to other strategies that are in use. Default value is `false` | +| `IDLE_TIMEOUT` | The number of seconds that must pass without data being received before the Lightning web server kills the connection. | +| `IS_RESETTABLE_DEMO` | If set to `yes`, it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy _most_ of what you have in your database! | +| `K8S_HEADLESS_SERVICE` | This environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're _not_ using Kubernetes, the "gossip" strategy is used to establish clusters. | +| `LISTEN_ADDRESS` | The address the web server should bind to. Defaults to `127.0.0.1` to block access from other machines. | +| `LOG_LEVEL` | How noisy you want the logs to be (e.g., `debug`, `info`) | +| `METRICS_RUN_PERFORMANCE_AGE_SECONDS` | The oldest a run can be to be included in Run performance metrics. | +| `METRICS_RUN_QUEUE_AGE_SECONDS` | The polling period for run queue metrics. | +| `METRICS_STALLED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it is considered stalled. | +| `METRICS_UNCLAIMED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it counts towards an impeded project. | +| `MIX_ENV` | Your mix env, likely `prod` for deployment | +| `NODE_ENV` | Node env, likely `production` for deployment | +| `ORIGINS` | The allowed origins for web traffic to the backend | +| `PER_WORKFLOW_CLAIM_LIMIT` | The maximum number of runs per workflow to consider during run claiming. This prevents any single workflow from dominating the processing queue while ensuring fairness across workflows.
Default is `50`. | +| `CLAIM_WORK_MEM` | PostgreSQL `work_mem` setting for the run claim query. Helps optimize complex sorting operations. Set to a valid PostgreSQL memory value (e.g., `32MB`, `64MB`, `1GB`). Set to empty string to disable.
Default: disabled in dev/test, `32MB` in production. | +| `PORT` | The port your Phoenix app runs on | +| `PROMEX_DATASOURCE_ID` | The datasource that PromEx will use if configured to push initial dashboards to Grafana. Defaults to an empty string. | +| `PROMEX_ENABLED` | Enables PromEx tracking and publishing of metrics if set to 'true' or 'yes'. Defaults to false. | +| `PROMEX_ENDPOINT_SCHEME` | The scheme needed when connecting to the Promex Endpoint. Defaults to https. | +| `PROMEX_EXPENSIVE_METRICS_ENABLED` | Certain metrics may be expensive to generate if Lightning is under load. If set to 'true', or 'yes' these metrics will be enabled. Defaults to 'false'. | +| `PROMEX_GRAFANA_HOST` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | +| `PROMEX_GRAFANA_PASSWORD` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | +| `PROMEX_GRAFANA_USER` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | +| `PROMEX_METRICS_ENDPOINT_AUTHORIZATION_REQUIRED` | If set to 'true' or 'yes', the PromEx endpoint on Lightning will require consumers to provide credentials for authorization. Defaults to 'true'. | +| `PROMEX_METRICS_ENDPOINT_TOKEN` | A Bearer token that the consumer of the promEx endpoint must provide in the Authorization header. Defaults to a random series of bytes. | +| `PROMEX_UPLOAD_GRAFANA_DASHBOARDS_ON_START` | Instructs PromEx to upload iniital dashboards to a Grafana instance if set to 'true' or 'yes'. Defaults to false. | +| `PRIMARY_ENCRYPTION_KEY` | A base64 encoded 32 character long string.
See [Encryption](#encryption). | +| `QUEUE_RESULT_RETENTION_PERIOD_MINUTES` | The number of minutes to keep completed (successful) `ObanJobs` in the queue (not to be confused with runs and/or history) | +| `SCHEMAS_PATH` | Path to the credential schemas that provide forms for different adaptors | +| `ADAPTORS_REGISTRY_JSON_PATH` | Path to adaptor registry file. When provided, the app will attempt to read from it then later fallback to the internet | +| `SECRET_KEY_BASE` | A secret key used as a base to generate secrets for encrypting and signing data. | +| `SENTRY_DSN` | If using Sentry for error monitoring, your DSN | +| `UI_METRICS_ENABLED` | Enable serverside tracking of certain metrics related to the UI. This s temporary functionality. Defaults to `false`. | +| `URL_HOST` | The host used for writing URLs (e.g., `demo.openfn.org`) | +| `URL_PORT` | The port, usually `443` for production | +| `URL_SCHEME` | The scheme for writing URLs (e.g., `https`) | +| `USAGE_TRACKER_HOST` | The host that receives usage tracking submissions
(defaults to https://impact.openfn.org) | +| `USAGE_TRACKING_DAILY_BATCH_SIZE` | The number of days that will be reported on with each run of `UsageTracking.DayWorker`. This will only have a noticeable effect in cases where there is a backlog or where reports are being generated retroactively (defaults to 10). | +| `USAGE_TRACKING_ENABLED` | Enables the submission of anonymized usage data to OpenFn (defaults to `true`) | +| `USAGE_TRACKING_RESUBMISSION_BATCH_SIZE` | The number of failed reports that will be submitted on each resubmission run (defaults to 10) | +| `USAGE_TRACKING_RUN_CHUNK_SIZE` | The size of each batch of runs that is streamed from the database when generating UsageTracking reports (default 100). Decreasing this may decrease memory consumption when generating reports. | +| `USAGE_TRACKING_UUIDS` | Indicates whether submissions should include cleartext UUIDs or not. Options are `cleartext` or `hashed_only`, with the default being `hashed_only`. | +| `REQUIRE_EMAIL_VERIFICATION` | Indicates whether user email addresses should be verified. Defaults to `false`. | ### AI Chat diff --git a/lib/lightning/config/bootstrap.ex b/lib/lightning/config/bootstrap.ex index 39c0a4cd2b..fc62893ee8 100644 --- a/lib/lightning/config/bootstrap.ex +++ b/lib/lightning/config/bootstrap.ex @@ -466,13 +466,53 @@ defmodule Lightning.Config.Bootstrap do end database_url = env!("DATABASE_URL", :string, nil) + database_timeout = env!("DATABASE_TIMEOUT", :integer, 15_000) + + # A socket-level timeout so a connection whose network path has died fails + # fast rather than blocking on a dead socket. It does not abort a live + # query; it only closes a connection whose peer has stopped acknowledging + # data. + # + # `tcp_user_timeout` caps how long transmitted data may go unacknowledged + # before the socket is dropped. It is a Linux-only raw socket option + # (IPPROTO_TCP=6, TCP_USER_TIMEOUT=18, value in ms) and defaults to + # `DATABASE_TIMEOUT + 5s`, so the application query timeout fires first on a + # connection that is merely slow. Set `DATABASE_TCP_USER_TIMEOUT` to + # override, or `0` to disable it. + linux? = match?({:unix, :linux}, :os.type()) + + tcp_user_timeout = + case env!("DATABASE_TCP_USER_TIMEOUT", :integer, nil) do + nil -> database_timeout + 5_000 + ms when ms <= 0 -> nil + ms -> ms + end + + if tcp_user_timeout && tcp_user_timeout < database_timeout do + IO.warn( + "DATABASE_TCP_USER_TIMEOUT (#{tcp_user_timeout}ms) is below " <> + "DATABASE_TIMEOUT (#{database_timeout}ms). On Linux this can drop a " <> + "slow-but-live connection — e.g. a large write whose data is slow to " <> + "be acknowledged — before the query timeout fires. Set it above " <> + "DATABASE_TIMEOUT.", + [] + ) + end + + db_socket_options = + if linux? && tcp_user_timeout do + [{:raw, 6, 18, <>}] + else + [] + end config :lightning, Lightning.Repo, url: database_url, pool_size: env!("DATABASE_POOL_SIZE", :integer, 10), - timeout: env!("DATABASE_TIMEOUT", :integer, 15_000), + timeout: database_timeout, queue_target: env!("DATABASE_QUEUE_TARGET", :integer, 50), - queue_interval: env!("DATABASE_QUEUE_INTERVAL", :integer, 1000) + queue_interval: env!("DATABASE_QUEUE_INTERVAL", :integer, 1000), + socket_options: db_socket_options port = env!( @@ -525,9 +565,11 @@ defmodule Lightning.Config.Bootstrap do disable_db_ssl = env!("DISABLE_DB_SSL", &Utils.ensure_boolean/1, false) + # appends rather than overwrites db_socket_options, so the IPv6 toggle and + # the socket timeout both apply config :lightning, Lightning.Repo, url: database_url, - socket_options: maybe_ipv6 + socket_options: maybe_ipv6 ++ db_socket_options if disable_db_ssl do config :lightning, Lightning.Repo, ssl: false diff --git a/test/lightning/config/bootstrap_test.exs b/test/lightning/config/bootstrap_test.exs index 17ecd6b826..bcd2006337 100644 --- a/test/lightning/config/bootstrap_test.exs +++ b/test/lightning/config/bootstrap_test.exs @@ -127,6 +127,124 @@ defmodule Lightning.Config.BootstrapTest do assert endpoint_idle_timeout() == 75_000 end + test "Repo socket options (tcp_user_timeout, IPv6)" do + # tcp_user_timeout is a Linux-only raw socket option; assert its presence + # only where it is actually applied. + linux? = match?({:unix, :linux}, :os.type()) + + # default: tcp_user_timeout defaults to DATABASE_TIMEOUT + 5s on Linux + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE" + }) + + default_opts = repo_opt(:socket_options) + + if linux? do + assert {:raw, 6, 18, <<20_000::32-native>>} in default_opts + else + assert default_opts == [] + end + + # the default tracks an overridden DATABASE_TIMEOUT + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DATABASE_TIMEOUT" => "30000" + }) + + if linux? do + assert {:raw, 6, 18, <<35_000::32-native>>} in repo_opt(:socket_options) + end + + # explicit override + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DATABASE_TCP_USER_TIMEOUT" => "25000" + }) + + override_opts = repo_opt(:socket_options) + + if linux? do + assert {:raw, 6, 18, <<25_000::32-native>>} in override_opts + end + + # the prod block also sets socket_options for the IPv6 toggle; it appends + # the resilience options rather than clobbering them, so ECTO_IPV6 and the + # tcp_user_timeout option must coexist + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "ECTO_IPV6" => "true" + }) + + ipv6_opts = repo_opt(:socket_options) + assert :inet6 in ipv6_opts + + if linux? do + assert {:raw, 6, 18, <<20_000::32-native>>} in ipv6_opts + end + + # 0, or any non-positive value, disables the socket timeout entirely + # rather than wrapping into a nonsensical unsigned value + for disabled <- ["0", "-1"] do + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DATABASE_TCP_USER_TIMEOUT" => disabled + }) + + refute Enum.any?( + repo_opt(:socket_options), + &match?({:raw, 6, 18, _}, &1) + ) + end + end + + test "warns when DATABASE_TCP_USER_TIMEOUT is below DATABASE_TIMEOUT" do + low = + ExUnit.CaptureIO.capture_io(:stderr, fn -> + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DATABASE_TCP_USER_TIMEOUT" => "5000" + }) + end) + + assert low =~ "DATABASE_TCP_USER_TIMEOUT" + + ok = + ExUnit.CaptureIO.capture_io(:stderr, fn -> + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DATABASE_TCP_USER_TIMEOUT" => "20000" + }) + end) + + refute ok =~ "DATABASE_TCP_USER_TIMEOUT" + end + + test "Repo SSL is on by default and disabled by DISABLE_DB_SSL" do + # SSL defaults on for managed Postgres; DISABLE_DB_SSL opts out + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE" + }) + + assert repo_opt(:ssl) == true + assert repo_opt(:ssl_opts) == [verify: :verify_none] + + reconfigure(%{ + "SECRET_KEY_BASE" => "Foo", + "DATABASE_URL" => "ecto://USER:PASS@HOST/DATABASE", + "DISABLE_DB_SSL" => "true" + }) + + assert repo_opt(:ssl) == false + end + test "prod endpoint URL defaults" do reconfigure(%{ "SECRET_KEY_BASE" => "Foo", @@ -951,6 +1069,12 @@ defmodule Lightning.Config.BootstrapTest do end end + defp repo_opt(key) do + :lightning + |> get_env(Lightning.Repo) + |> Keyword.get(key) + end + defp reconfigure(envs) do Process.put({Config, :config}, []) Process.put({Config, :imports}, [])