Add MySQL database adapter and shared pool mode via USE switching#283
Draft
QuirkQ wants to merge 15 commits intobasecamp:mainfrom
Draft
Add MySQL database adapter and shared pool mode via USE switching#283QuirkQ wants to merge 15 commits intobasecamp:mainfrom
USE switching#283QuirkQ wants to merge 15 commits intobasecamp:mainfrom
Conversation
Clean-room implementation of MySQL support for tenant database lifecycle operations. Extracts a with_server_connection helper to avoid repeating temporary connection setup, uses connection quoting for all SQL values, queries information_schema.schemata for exact existence checks instead of LIKE, and rescues ActiveRecord wrapper exceptions instead of adapter-specific ones for mysql2/trilogy parity. Adds database name length validation (64-char MySQL limit) and per-tenant host routing to BaseConfig.
Three new error classes (TenantSwitchError, TenantResetError, TenantSwitchInTransactionError) for shared pool runtime failures, and four config methods on BaseConfig (shared_pool?, fallback_database, build_shared_pool_config, validate_shared_pool) to detect, validate, and build the concrete db config for shared pool mode. No runtime behavior changes. Non-shared-pool code paths are unaffected.
Today every tenant maps 1:1 to a Rails connection pool. At high tenant counts (5,000+) this causes heavy pool churn, memory pressure, and latency spikes from constant pool recreation. This commit is the architectural pivot for shared pool support. It introduces pool_shard_for, a single decision point that splits logical tenant identity from physical pool identity. In shared pool mode, all tenants map to one synthetic shard key (:__tenanted_shared_pool__) per role instead of one pool per tenant. In non-shared mode, the tenant string is returned directly — zero overhead, zero behavioral change. Every pool flow now routes through this: connection_pool computes the physical shard once and threads it through retrieve_connection_pool and _create_tenanted_pool. destroy_tenant in shared mode drops the database but never removes the shared pool (since other tenants still need it). Without adapter callbacks (next commit), the shared pool connects to the fallback database and stays there. The pool routing layer is testable independently of the switching layer.
Introduces the connection-level tenant switching layer for shared pool mode. This is Layer 1 of the dual-layer safety model — it handles first checkouts and connection teardown, while Layer 2 (tenant context reconciliation, coming in the next commit) will cover sticky leases and nested with_tenant. On checkout, the module resolves the current tenant from the connection class stored in the adapter config, switches the MySQL session to the tenant's database via USE, and wraps the pool's query cache in a NamespaceStore that prefixes cache keys with the current tenant database name. On checkin, it resets the connection back to the fallback database and clears tenant state so idle connections never carry stale tenant associations. Failed USE statements always discard the connection via throw_away! and raise typed errors. Tenant switches inside an open transaction raise TenantSwitchInTransactionError. Non-shared-pool connections see a single boolean check and early return with zero overhead.
This completes the dual-layer safety model for shared pool tenant switching. Layer 1 (the SharedPool adapter module's checkout/checkin callbacks) handles first checkouts and connection teardown. But it can't cover sticky leases — when a connection is already leased and with_tenant changes context, the pool's ||= short-circuits checkout entirely, so no callback fires. Layer 2, added here, fills that gap. with_tenant now calls ensure_shared_pool_tenant_switch at two points: inside connected_to before yielding to user code (switches a sticky lease to the new tenant), and in a method-level ensure after connected_to pops the shard stack (reconciles back to the outer tenant on nested exit). A single :after :set_current_tenant callback covers the current_tenant= path for persistent changes. All three call sites use the same method, which reads current_tenant as ground truth and no-ops when the connection is already correct. No captured state, no restore tracking, no callback ordering dependencies. The Railtie initializer includes SharedPool into both MySQL adapter classes via load hooks, activating the checkout/checkin callbacks registered by that module's included block.
…eation race condition Two critical bugs in the tenanted gem's shared pool mode are fixed here. The first bug causes SharedPool checkout/checkin callbacks to fire on temporary and task connections that are not true shared-pool runtime connections. The SharedPool module is included into all MySQL2 and Trilogy adapters globally via Railtie load hooks, and its callbacks guard on shared_pool? which only checks @config[:shared_pool] == true. The problem is that shared_pool: true propagates through config inheritance: BaseConfig -> new_tenant_config -> TenantConfig, and from there into with_server_connection which builds a HashConfig via configuration_hash.except(:database). These temporary configs carry shared_pool: true but never receive :tenanted_connection_class_name, which is only set by build_shared_pool_config for actual runtime pool connections. When the checkout callback fires on a temporary connection, shared_pool? returns true and then @config.fetch(:tenanted_connection_class_name) raises KeyError. This affects tenant_exist?, tenants, create_tenant, destroy_tenant, and all DatabaseTasks flows in shared pool mode. The fix tightens the shared_pool? guard to also require the presence of :tenanted_connection_class_name in the config hash, which is the definitive marker of a properly-constructed shared pool connection. The second bug is a race condition in MySQL tenant creation. Two sub-issues: acquire_ready_lock for MySQL was a no-op that just yielded, unlike the SQLite adapter which uses a file-based mutex. This means two concurrent threads or processes can both pass the database_exist? check and attempt to create the same database. Compounding this, the rescue clause in create_tenant unconditionally calls adapter.drop_database on any error, even if the current process did not create the database. So if thread A creates the database, thread B's create_database fails, thread B's rescue drops thread A's database, and then thread A's in-progress migration fails because its database was pulled out from under it. The fix has two parts: acquire_ready_lock now uses MySQL advisory locking via GET_LOCK/RELEASE_LOCK scoped to the tenant database name with a 30-second timeout, providing cross-process coordination on the same MySQL server. And created_db = true is moved to immediately after adapter.create_database succeeds, with the rescue gated on if created_db, so we only clean up databases we actually created.
Root cause: Rails' AbstractAdapter#query_cache (query_cache.rb:217-226) has a special path for pinned connections accessed from a non-owner thread — it returns pool.query_cache directly (a raw per-thread Store from QueryCacheRegistry). This bypasses the NamespaceStore wrapper that SharedPool#attach_query_cache_namespace sets on @query_cache during checkout. Without namespacing, if the cross-thread switches tenants (e.g., a Capybara server thread handling requests for different tenants during a system test), the bare SQL key could match a cached result from a different tenant's query. Fix: Override query_cache in SharedPool to intercept the raw Store coming from the cross-thread path and wrap it in a NamespaceStore. The guard !cache.is_a?(NamespaceStore) ensures the normal path (where @query_cache is already a NamespaceStore) passes through untouched. The wrapper is lightweight (2 instance variables, pure delegation) and the namespace proc reads tenant_database dynamically, keeping keys properly prefixed even as the connection switches tenants. Scope: This primarily affects test environments (transactional fixtures, system tests) where connections are pinned and shared across threads.
The 64-character database name length guard in BaseConfig#database_for ran before test_workerize appended the _<worker_id> suffix. A database name at or near 64 characters would pass validation and then exceed the MySQL limit at runtime during parallel test execution. The fix reorders the two operations so the length check runs against the final database name including any test worker suffix.
The rescue clauses in tenant_databases and database_exist? caught both NoDatabaseError and StatementInvalid, returning empty array or false. StatementInvalid wraps nearly any MySQL error including permission denied, connection failures, and syntax errors, so a misconfigured user lacking SHOW DATABASES privileges would silently return empty results instead of surfacing the problem. StatementInvalid is removed from both rescue clauses, keeping only NoDatabaseError as the legitimate not-found case. Both rescued paths now log a warning with the error message so they don't go unnoticed. All logging throughout the gem is standardized from Rails.logger to ActiveRecord::Base.logger with nil-safe navigation. Rails.logger is not guaranteed to be available during early boot, in rake tasks, or when using ActiveRecord outside of Rails. ActiveRecord::Base.logger is the correct choice for an ActiveRecord extension and is consistent with how ActiveRecord itself logs internally. Seven occurrences are updated across mysql.rb, sqlite.rb, and tenant.rb. The existing ActiveRecord::Base.logger.info call in destroy_tenant gains a nil guard it was missing. The log_tenant_tag method adds a respond_to?(:tagged) check and falls through to yield when the logger is nil or not a TaggedLogging instance, preventing the block from being silently swallowed.
The server connection helper used during tenant creation was replacing
ActiveRecord::Base's global connection pool, leaking a database-less
config into unrelated queries for the duration of the create+migrate
critical section. It now uses an isolated ConnectionHandler that is
discarded after each use.
Advisory lock release failures in ensure could mask a successful tenant
creation and cause the caller to drop a perfectly good database. The
release is now rescued and logged, since MySQL automatically frees
advisory locks when the connection closes.
Both host_for and database_for used sprintf, which treats any literal
percent sign as a format directive. They now use gsub to substitute only
the %{tenant} placeholder.
The shared-pool prepared-statements guard only rejected the boolean true,
but Active Record treats most non-"false" values as enabled. It now uses
the same type_cast_config_to_boolean that Active Record uses internally.
…dation
Prevent with_server_connection from attempting to connect to a literal %{tenant}.db.example.com hostname when called on a BaseConfig with host templating. This made tenants, with_each_tenant, and migrate-all workflows silently broken for host-templated setups. Now raises TenantConfigurationError with a clear message explaining that
host-templated configs cannot enumerate databases from a single server.
Also move shared-pool validation to boot time (after_initialize) so misconfigs like wrong adapter, missing untenanted_database, or prepared_statements: true are caught immediately instead of at first tenant connection. The existing runtime check in build_shared_pool_config is kept as defense-in-depth.
…mplates
The switch from sprintf to gsub in database_for and host_for silently regressed template typo safety. A misspelled token like %{tenant_name} instead of %{tenant} no longer raised a KeyError — it passed through unchanged, quietly routing every tenant to the same database or host. Add post-substitution checks that raise
TenantConfigurationError if any %{...} token survives after %{tenant} replacement, catching typos at the point of use rather than letting them cause silent misrouting.
acquire_ready_lock built lock names as "tenanted:#{database_path}", but database names are allowed up to 64 characters, producing lock names up to 73 characters. MySQL's GET_LOCK has a 64-character limit and silently truncates longer names, which could cause unrelated tenants to collide on the same lock. Replace the raw database path
with a SHA256 digest, keeping the name deterministic and always within the 64-character limit.
The prepared_statements check in validate_shared_pool rejected configs that simply omitted the setting, even though mysql2 and trilogy both default to false. Since the validation already requires one of those two adapters, an unset value is safe. Skip the check when prepared_statements is nil and only raise when it's explicitly set to a truthy value.
…ments in shared pool mode Every shared-pool query currently issues two USE roundtrips — one on checkin (reset to fallback) and one on checkout (switch to tenant). Since switch_tenant_database already has a fast-path no-op when the connection is already on the right tenant, we can skip the checkin reset and let consecutive same-tenant checkouts resolve with just two string comparisons and zero IO. The checkout callback still validates the tenant on every checkout, so tenant safety is maintained. Defaults to true to preserve existing behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a draft/proof-of-concept for MySQL support in
activerecord-tenanted, including a shared pool mode that replaces the per-tenant connection pool model with a single shared pool per role. Tenant switching happens at the MySQL session level viaUSE <db>on checkout and checkin.The motivation is straightforward: at high tenant cardinality (thousands of tenants), the per-tenant pool model causes excessive memory usage, file descriptor pressure, and latency spikes from frequent pool creation and teardown. A single shared pool avoids all of that.
This approach also provides a much cleaner migration path for applications coming from
ros-apartment, which uses the same session-levelUSEdatabase switching model. Apps moving toactiverecord-tenantedwould get a familiar runtime model without needing to restructure their MySQL databases into separate connection pools per tenant.No tests are included yet. This is a proof of concept to see if there's community interest. If there is, I'd be happy to add comprehensive tests following the existing scenario-based patterns and discuss splitting this into smaller, focused PRs (for example, the MySQL adapter could land as a standalone PR since it's independently useful, with shared pool mode layered on top).
What's in the diff
The MySQL database adapter
The
DatabaseAdapters::MySQLclass is heavily based on @andrewmarkle's work in #246. He introduced the adapter pattern and the core implementation. I built on top of his PR with some changes: extracting the repeated server-connection setup into awith_server_connectionhelper, switching fromDatabaseTasks.with_temporary_connectionto a throwawayConnectionHandler(the former replacesActiveRecord::Base's global connection pool for the duration of the block, which would break concurrent queries), usingconn.quoteinstead of string interpolation forSHOW DATABASES LIKE, adding proper advisory locking via MySQL'sGET_LOCK/RELEASE_LOCK, and filling in thetest_workerizeimplementation with a double-suffix guard.The adapter is registered for both
mysql2andtrilogy.The core idea: splitting logical tenant identity from physical pool identity
The key architectural decision is introducing
pool_shard_foras a single decision point. In the existing per-tenant model, each tenant name maps 1:1 to a Rails shard key and therefore to a distinct connection pool. In shared pool mode, all tenants map to one synthetic shard key (:__tenanted_shared_pool__) per role. The logical tenant stays in theconnected_tostack exactly as before, but the physical pool lookup uses the sentinel key.This works because
activerecord-tenantedalready overridesconnection_poolon the Tenant module and controls the entire pool lookup path. Rails' internal pool machinery is never patched. The change is just which shard key gets passed toretrieve_connection_poolandconnection_handler.establish_connection.For pool creation in shared mode, I call
connection_handler.establish_connectiondirectly with the sentinel shard rather than going through the class-levelestablish_connection, because the class-level method hardcodesshard: current_shard(which would be the tenant name string, not the sentinel). The pool connects to the configureduntenanted_database(typicallyinformation_schema) and tenant switching happens at runtime via adapter callbacks.Tenant switching: adapter callbacks and context reconciliation
The primary switching mechanism is straightforward. The
SharedPoolmodule is included intoMysql2AdapterandTrilogyAdaptervia Railtie load hooks. It registers:after checkoutand:after checkincallbacks. On checkout, it switches the connection to the current tenant's database viaUSEand attaches a tenant-namespaced query cache wrapper. On checkin, it resets to the fallback database. This covers the normal request lifecycle where a connection is checked out, used for one tenant, and returned.On top of that there's a second layer of reconciliation in the
Tenantmodule.with_tenantcallsensure_shared_pool_tenant_switchinsideconnected_tobefore the user block, and again in a method-levelensureafterconnected_topops the shard stack. There's also an:after :set_current_tenantcallback forcurrent_tenant=.In practice,
with_tenantdefaults toprohibit_shard_swapping: true, which means Rails will raise if you try to nestwith_tenantcalls — so the normal flow is one tenant per request and the checkout callback handles it entirely. The reconciliation layer is a safety net for the edge cases whereprohibit_shard_swapping: falseis passed explicitly (currently onlydestroy_tenantandwithout_tenantin the testing helper do this), and forcurrent_tenant=which does a persistent switch without going through the pool checkout path.Both layers call
switch_tenant_database, which no-ops when the connection is already on the right tenant (two string comparisons). No conflict, no double-switching.Safety guarantees
Failed
USEstatements discard the connection entirely viathrow_away!and raise a typed error (TenantSwitchErrororTenantResetError). No reuse of unknown session state. Tenant switching inside an open transaction raisesTenantSwitchInTransactionErrorbecause switching would corrupt the transaction's session state.Query cache keys are tenant-namespaced via a
NamespaceStorewrapper that prefixes keys with the current tenant database name. This prevents cross-tenant cache hits when two tenants execute the same SQL on the same shared connection. The wrapper uses a proc for the namespace (not a fixed string) because tenant switching can happen after the wrapper is created.Every shared-pool method is guarded with
return unless shared_pool?. Non-shared-pool connections see a single hash lookup and early return. Zero overhead for existing SQLite or per-tenant MySQL deployments.Other changes worth mentioning
destroy_tenantin shared pool mode drops the database but never removes the shared pool (since it's shared across all tenants).create_tenantbootstraps the shared pool on the first tenant and reuses it for all subsequent tenants. I also fixed a bug increate_tenantwhere the rescue block would calldrop_databaseeven if the database creation itself hadn't succeeded yet.database_forwas switched fromsprintftogsubwith unrecognized token detection, andhost_forwas added for per-tenant host routing in non-shared-pool mode. The MySQL 64-character database name limit is validated on the full interpolated name.A few
Rails.loggercalls were changed toActiveRecord::Base.logger&.for consistency and to avoid nil errors when the logger isn't set up.Configuration
Shared pool mode is opt-in. Without
shared_pool: true, behavior is identical to today's per-tenant pool model.Configuration is validated at boot: the adapter must be
mysql2ortrilogy,untenanted_databasemust be present,prepared_statementsmust not be explicitly set totrue(prepared statement caches are tied to a specific database — both mysql2 and trilogy already default tofalse), and host templating with%{tenant}is rejected because a single pool implies a single host.Status and next steps
This is a draft. I'm opening it to gauge interest and get early feedback on the approach. If there's appetite for this:
I'd add unit and integration tests following the existing scenario patterns. The MySQL adapter could probably land as its own PR first since it's independently useful for any MySQL-backed tenanted deployment. Shared pool mode would layer on top. Happy to iterate on any of the design decisions above.