diff --git a/website/src/content/cookbook/vpc-air-gapped.mdx b/website/src/content/cookbook/vpc-air-gapped.mdx index 843aba522d..db26b1e0ed 100644 --- a/website/src/content/cookbook/vpc-air-gapped.mdx +++ b/website/src/content/cookbook/vpc-air-gapped.mdx @@ -107,10 +107,10 @@ If you ship software that runs inside your customers' VPCs, the same setup turns | Backend | Use when | Status | | --- | --- | --- | | [File System](/docs/self-hosting/filesystem) (RocksDB-based) | Single-node deployments, including air-gapped installs | Production-ready, single node only | -| [PostgreSQL](/docs/self-hosting/postgres) | Multi-node deployments | Recommended for multi-node today, but experimental | +| [PostgreSQL](/docs/self-hosting/postgres) | Multi-node and multi-region deployments | Production-ready for multi-node | | FoundationDB | Largest production deployments | [Enterprise](/sales) | -For multi-node deployments, run two or more engine nodes behind a load balancer and add NATS for pub/sub, which replaces the default PostgreSQL `LISTEN`/`NOTIFY` path at high throughput. Neither is needed for a single-node file system install. See the [Production Checklist](/docs/self-hosting/production-checklist). +For multi-node deployments, run two or more engine nodes behind a load balancer, all sharing one PostgreSQL instance. The built-in PostgreSQL pub/sub is sufficient for most deployments; very high-throughput deployments can add NATS as a dedicated pub/sub layer. Neither is needed for a single-node file system install. See the [Production Checklist](/docs/self-hosting/production-checklist). ## Perimeter Checklist diff --git a/website/src/content/docs/self-hosting/configuration.mdx b/website/src/content/docs/self-hosting/configuration.mdx index 0196ff24ad..837f4c9cbc 100644 --- a/website/src/content/docs/self-hosting/configuration.mdx +++ b/website/src/content/docs/self-hosting/configuration.mdx @@ -76,5 +76,5 @@ Use `samples: 1` for a uniform random pick that skips slot reads. Use `samples > ## Related - SQLite actor startup picks the storage schema by probing the actor KV subspace for legacy v1 data. Existing v1 data stays on v1, and actors without v1 data start on v2. -- [PostgreSQL](/docs/self-hosting/postgres): Configure the experimental PostgreSQL backend -- [File System](/docs/self-hosting/filesystem): Configure file system storage for development +- [PostgreSQL](/docs/self-hosting/postgres): Configure the PostgreSQL backend for multi-node and multi-region deployments +- [File System](/docs/self-hosting/filesystem): Configure file system storage for single-node deployments diff --git a/website/src/content/docs/self-hosting/docker-compose.mdx b/website/src/content/docs/self-hosting/docker-compose.mdx index 5d77eae8af..b7f98a2464 100644 --- a/website/src/content/docs/self-hosting/docker-compose.mdx +++ b/website/src/content/docs/self-hosting/docker-compose.mdx @@ -181,9 +181,9 @@ Create `rivet-config.json` in the same directory as your `docker-compose.yaml`. ### Postgres Setup - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ```yaml services: diff --git a/website/src/content/docs/self-hosting/docker-container.mdx b/website/src/content/docs/self-hosting/docker-container.mdx index cbbd39cb6b..914fe7be70 100644 --- a/website/src/content/docs/self-hosting/docker-container.mdx +++ b/website/src/content/docs/self-hosting/docker-container.mdx @@ -141,9 +141,9 @@ Create `rivet-config.json` in your working directory. See the [Configuration](/d ### Postgres Setup - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ```bash # Create network diff --git a/website/src/content/docs/self-hosting/filesystem.mdx b/website/src/content/docs/self-hosting/filesystem.mdx index 07db518a53..b7a8b86ce6 100644 --- a/website/src/content/docs/self-hosting/filesystem.mdx +++ b/website/src/content/docs/self-hosting/filesystem.mdx @@ -4,9 +4,9 @@ description: "The file system backend stores all data on the local disk. This is skill: true --- - -For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based); for multi-node deployments, PostgreSQL is the recommended backend today but remains experimental as we evaluate the best fit for scalability and performance, and Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. - + +The file system backend (RocksDB-based) is recommended for single-node Rivet deployments. For multi-node and multi-region deployments, use the [PostgreSQL backend](/docs/self-hosting/postgres). Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ## Configuration @@ -47,4 +47,4 @@ The file system backend is ideal for: - Testing and prototyping - Air-gapped environments without database infrastructure -If you need a production-ready Rivet deployment today, use this backend for smaller single-node setups; for multi-node deployments, PostgreSQL is the recommended backend today though still experimental, and Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. +Use this backend for single-node setups. For multi-node and multi-region deployments, use the [PostgreSQL backend](/docs/self-hosting/postgres). Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. diff --git a/website/src/content/docs/self-hosting/foundationdb.mdx b/website/src/content/docs/self-hosting/foundationdb.mdx index e98bf8f887..a91882c9f5 100644 --- a/website/src/content/docs/self-hosting/foundationdb.mdx +++ b/website/src/content/docs/self-hosting/foundationdb.mdx @@ -25,9 +25,9 @@ Its strict serializability guarantees, fault tolerance, and ability to scale lin | | RocksDB (File System) | PostgreSQL | FoundationDB | |---|---|---|---| -| **Scalability** | Single node | Primary/replica failover | Linear horizontal scaling | +| **Scalability** | Single node | Multi-node and multi-region | Linear horizontal scaling | | **Fault tolerance** | None | Primary/replica failover | Automatic recovery with no data loss | -| **Production readiness** | Development and small deployments | Experimental for multi-node | Battle-tested at global scale | +| **Production readiness** | Single-node deployments | Production-ready for multi-node | Battle-tested at global scale | ## Getting Started diff --git a/website/src/content/docs/self-hosting/index.mdx b/website/src/content/docs/self-hosting/index.mdx index 116371878c..177490a616 100644 --- a/website/src/content/docs/self-hosting/index.mdx +++ b/website/src/content/docs/self-hosting/index.mdx @@ -118,9 +118,9 @@ Rivet has 3 core components: Rivet supports multiple storage backends: -- **File System**: Recommended for smaller single-node deployments today -- **PostgreSQL**: Recommended for multi-node deployments today, but still experimental -- **FoundationDB**: Best scalability and performance for production-ready deployments ([requires enterprise](/sales)) +- **File System**: Recommended for single-node deployments +- **PostgreSQL**: Recommended for production multi-node and multi-region deployments +- **FoundationDB**: Best scalability and performance for the largest deployments ([requires enterprise](/sales)) ## Deployment Platforms diff --git a/website/src/content/docs/self-hosting/kubernetes.mdx b/website/src/content/docs/self-hosting/kubernetes.mdx index 42afe25056..c1641b2da1 100644 --- a/website/src/content/docs/self-hosting/kubernetes.mdx +++ b/website/src/content/docs/self-hosting/kubernetes.mdx @@ -4,9 +4,9 @@ description: "Deploy production-ready Rivet Engine to Kubernetes with PostgreSQL skill: true --- - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ## Prerequisites diff --git a/website/src/content/docs/self-hosting/postgres.mdx b/website/src/content/docs/self-hosting/postgres.mdx index a6aa9ad0ad..dd05cd9c42 100644 --- a/website/src/content/docs/self-hosting/postgres.mdx +++ b/website/src/content/docs/self-hosting/postgres.mdx @@ -4,9 +4,21 @@ description: "Configure PostgreSQL for self-hosted Rivet deployments." skill: true --- - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the [file system backend](/docs/self-hosting/filesystem) (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + + +## Overview + +PostgreSQL is the storage and coordination backend for self-hosted Rivet deployments that run more than one engine node. Multiple engine nodes can share a single PostgreSQL instance with no extra coordination service to deploy. Rivet handles leader election, failover, and version sequencing internally. + +Use PostgreSQL when you need: + +- **Multiple engine nodes** behind a load balancer for redundancy and horizontal scaling. +- **Multi-region deployments** (deploy one PostgreSQL instance per region, see [Multi-Region](/docs/self-hosting/multi-region)). +- **High availability** with a managed or self-managed primary/replica failover setup. + +For a single-node deployment, the file system backend is simpler and requires no separate database. For the largest, most demanding production workloads, FoundationDB ([enterprise](/sales)) scales further. ## Basic Configuration @@ -28,6 +40,41 @@ RIVET__database__postgres__url="postgresql://user:password@host:5432/database" +## Requirements and Recommendations + +### Version + +Use PostgreSQL 14 or newer. Rivet is tested against PostgreSQL 18, which is recommended for new deployments. + +### Connection Limits + +Each Rivet engine node opens a pool of direct connections to PostgreSQL and can use well over a hundred connections per node under load. PostgreSQL's default `max_connections` of `100` is too low for even a single busy engine node. + +- Set PostgreSQL `max_connections` to comfortably exceed `(number of engine nodes × 150)` plus headroom for backups, monitoring, and your own queries. +- If you use a managed PostgreSQL service, confirm its connection limit is high enough or pick a tier that allows raising it. Connection exhaustion shows up as engine startup failures or stalled requests under load. + + +Do not work around the connection limit with a connection pooler. See [Do Not Use Connection Poolers](#do-not-use-connection-poolers) below. + + +### Resources + +PostgreSQL is the system of record for the entire deployment, so size it accordingly: + +- Give PostgreSQL dedicated CPU, memory, and fast disk (SSD/NVMe with high IOPS). Avoid co-locating it with other heavy workloads. +- Rivet generates steady write and row-turnover on its internal tables. Keep autovacuum enabled and healthy so dead tuples do not accumulate. + +### High Availability and Backups + +A single PostgreSQL instance is a single point of failure for your whole deployment. + +- Configure a standby replica with automatic failover (managed services such as Amazon RDS, Cloud SQL, and Azure Database provide this). +- Enable automated backups and point-in-time recovery, and periodically test restoring from them. + +### Multi-Region + +Deploy one PostgreSQL instance per region or datacenter. Engine nodes connect to the PostgreSQL instance in their own region. See [Multi-Region](/docs/self-hosting/multi-region) for the full topology. + ## Managed Postgres Compatibility Some hosted PostgreSQL platforms require additional configuration due to platform-specific restrictions. @@ -43,8 +90,7 @@ Use direct connection (not connection pooler). { "database": { "postgres": { - "url": "postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require", - "unstable_disable_lock_customization": true + "url": "postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" } } } @@ -52,7 +98,6 @@ Use direct connection (not connection pooler). ```bash Environment-variables RIVET__database__postgres__url="postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" -RIVET__database__postgres__unstable_disable_lock_customization=true ``` @@ -71,8 +116,7 @@ Use direct connection on port `5432` (not connection pooler). { "database": { "postgres": { - "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable", - "unstable_disable_lock_customization": true + "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" } } } @@ -80,7 +124,6 @@ Use direct connection on port `5432` (not connection pooler). ```bash Environment-variables RIVET__database__postgres__url="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" -RIVET__database__postgres__unstable_disable_lock_customization=true ``` @@ -96,7 +139,6 @@ Download the root certificate from your Supabase dashboard and specify its path. "database": { "postgres": { "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require", - "unstable_disable_lock_customization": true, "ssl": { "root_cert_path": "/path/to/supabase-ca.crt" } @@ -107,7 +149,6 @@ Download the root certificate from your Supabase dashboard and specify its path. ```bash Environment-variables RIVET__database__postgres__url="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require" -RIVET__database__postgres__unstable_disable_lock_customization=true RIVET__database__postgres__ssl__root_cert_path="/path/to/supabase-ca.crt" ``` @@ -198,26 +239,11 @@ Do not use: ## Troubleshooting -### Permission Denied Errors - -If you see errors like: +### Too Many Connections -``` -ERROR: permission denied to set parameter "deadlock_timeout" -ERROR: current transaction is aborted, commands ignored until end of transaction block -``` +Errors like `FATAL: sorry, too many clients already` or engine nodes failing to start under load mean PostgreSQL's `max_connections` is too low. Raise it to account for every engine node (see [Connection Limits](#connection-limits)). Do not add a connection pooler to work around this. -Add `unstable_disable_lock_customization: true` to your configuration: - -```json -{ - "database": { - "postgres": { - "url": "postgresql://...", - "unstable_disable_lock_customization": true - } - } -} -``` +### Connection Refused or TLS Errors -This disables Rivet's attempt to set `lock_timeout = 0` and `deadlock_timeout = 10ms`. Since `lock_timeout` defaults to `0` in PostgreSQL, skipping these settings is safe. Deadlock detection will use the default `1s` timeout instead of `10ms`. +- Confirm the engine connects directly to PostgreSQL and not through a pooler (PgBouncer, Supavisor, RDS Proxy). Rivet requires direct connections. +- For TLS errors, verify `sslmode` matches your server and, for custom certificate authorities, that `ssl.root_cert_path` points to the correct CA certificate. See [SSL/TLS Support](#ssltls-support). diff --git a/website/src/content/docs/self-hosting/production-checklist.mdx b/website/src/content/docs/self-hosting/production-checklist.mdx index 6f5547edb9..201124b8fb 100644 --- a/website/src/content/docs/self-hosting/production-checklist.mdx +++ b/website/src/content/docs/self-hosting/production-checklist.mdx @@ -8,9 +8,9 @@ skill: true We recommend passing this page to your coding agent to verify your configuration before deploying. - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + Also review the [general production checklist](/docs/general/production-checklist). @@ -29,20 +29,24 @@ Also review the [general production checklist](/docs/general/production-checklis - **Configure autoscaling for the Rivet Engine.** Set target CPU utilization to 70% and memory to 80% to ensure headroom for traffic spikes. In Kubernetes, this is configured via a Horizontal Pod Autoscaler (HPA). - **Use 2+ engine nodes for redundancy.** Running a single engine node is a single point of failure. Deploy at least two engine instances behind a load balancer. -- **RocksDB only supports a single node.** Do not run multiple RocksDB nodes. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). For multi-node deployments, PostgreSQL is the recommended backend today, though it remains experimental as we evaluate the best fit for scalability and performance. +- **RocksDB only supports a single node.** Do not run multiple RocksDB nodes. The file system backend (RocksDB-based) is for single-node deployments. For multi-node and multi-region deployments, use the PostgreSQL backend. - **Validate the rate limit on your serverless actor host.** Actor start requests are sent from your engine instances, so they all originate from a small set of IPs. Per-IP rate limits on the actor host will throttle the engine before they would throttle end-user traffic. Size the limit to your peak actor create and wake rate, and configure platform max concurrency (e.g. on GCP Cloud Run) to match your expected concurrent actor count. ## PostgreSQL -- **PostgreSQL is recommended for multi-node deployments, but remains experimental.** Validate the deployment carefully before rollout. -- **Configure automated backups.** Set up regular backups for your PostgreSQL database to prevent data loss. -- **Configure failover.** Set up a standby replica with automatic failover to ensure high availability. -- **Use FoundationDB for the most scalable production-ready deployments.** FoundationDB provides the best performance, scalability, and uptime for Rivet. Contact [enterprise support](https://rivet.dev/sales) for FoundationDB guidance. +- **Use PostgreSQL for multi-node and multi-region deployments.** Multiple engine nodes can share one PostgreSQL instance; no extra coordination service is required. See [PostgreSQL](/docs/self-hosting/postgres). +- **Raise `max_connections`.** Each engine node opens well over a hundred connections under load. Size `max_connections` to at least `(number of engine nodes × 150)` plus headroom. PostgreSQL's default of `100` is too low. See [Connection Limits](/docs/self-hosting/postgres#connection-limits). +- **Do not use a connection pooler.** Rivet requires direct connections. Do not put PgBouncer, Supavisor, or RDS Proxy in front of PostgreSQL. +- **Give PostgreSQL dedicated resources.** Provision dedicated CPU, memory, and fast disk, and keep autovacuum healthy. PostgreSQL is the system of record for the whole deployment. +- **Configure automated backups.** Set up regular backups and point-in-time recovery, and test restoring from them. +- **Configure failover.** Set up a standby replica with automatic failover to ensure high availability. A single instance is a single point of failure. +- **Use one PostgreSQL instance per region.** For multi-region deployments, deploy a separate PostgreSQL instance in each region. +- **Use FoundationDB for the largest deployments.** Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) for FoundationDB guidance. ## NATS -- **Use NATS for pub/sub (recommended).** By default, Rivet uses PostgreSQL `LISTEN`/`NOTIFY` for pub/sub which has limited throughput. NATS significantly improves performance for high-traffic deployments. This is not needed if using RocksDB. See [Configuration](/docs/self-hosting/configuration). -- **Deploy 2+ NATS replicas.** Run at least two NATS replicas for high availability. +- **Use NATS for very high-throughput pub/sub (optional).** By default, Rivet uses PostgreSQL for pub/sub, which is sufficient for most deployments. For very high-traffic deployments, NATS can be used as a dedicated pub/sub layer. This is not needed if using RocksDB. See [Configuration](/docs/self-hosting/configuration). +- **Deploy 2+ NATS replicas.** If you use NATS, run at least two replicas for high availability. ## Monitoring diff --git a/website/src/content/docs/self-hosting/railway.mdx b/website/src/content/docs/self-hosting/railway.mdx index a4e71a6bf2..4de305c510 100644 --- a/website/src/content/docs/self-hosting/railway.mdx +++ b/website/src/content/docs/self-hosting/railway.mdx @@ -12,9 +12,9 @@ import { TutorialVideo } from '@/components/TutorialVideo'; - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ## Quick Deploy diff --git a/website/src/content/docs/self-hosting/render.mdx b/website/src/content/docs/self-hosting/render.mdx index 718095f3ac..4a502e0757 100644 --- a/website/src/content/docs/self-hosting/render.mdx +++ b/website/src/content/docs/self-hosting/render.mdx @@ -1,12 +1,12 @@ --- title: "Render Deployment" -description: "Deploy Rivet Engine to Render with managed PostgreSQL and automatic HTTPS, using the experimental PostgreSQL backend." +description: "Deploy Rivet Engine to Render with managed PostgreSQL and automatic HTTPS." skill: true --- - -PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment. - + +PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. + ## Prerequisites