Database task roadmap (epic)

[flavorjones]

This gem only implements the bare minimum database rake tasks to get started:

• db:migrate (implicit create)
• db:drop
• linking of everything to db:prepare

There are many more rake tasks that this gem will need to implement. This issue is meant to track the overall work, since I imagine the list will not get completed in a single PR (and may never get completed, let's be honest).

An open question for me is whether Rails could be presenting rake tasks so that gems can more easily override them. I've got a branch (TODO) that wraps up the tasks in this gem in PORO, and that's a pattern that I think might be more easily extensible if it was adopted upstream.

Here are the full set of database rake tasks implemented by Rails as of ~v8.0, along with some notes on how they're implemented and my thoughts on whether we need to implement it or not.

• [~] db:_dump
  • forwards to "db:schema:dump"
• [~] db:_dump:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • forwards to "db:schema:dump:dbname"
• db:abort_if_pending_migrations
  • uses DatabaseTasks.with_temporary_pool_for_each to iterate over db configs
• db:abort_if_pending_migrations:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
• db:charset
  • only seems to work against "primary"
  • error caused by nil returned from configs_for(env_name: env_name, name: db_name)
  • maybe we don't need to support this immediately?
• db:check_protected_environments
  • loops over configs_for(env_name: environment)
  • we probably do not want or need to check all of the tenant databases. maybe check just the first tenant?
• db:collation
  • only seems to work against "primary"
  • error caused by nil returned from configs_for(env_name: env_name, name: db_name)
  • maybe we don't need to support this immediately?
• db:create
  • forwards to ActiveRecord::Tasks::DatabaseTasks.create_current
• db:create:all
  • forwards to ActiveRecord::Tasks::DatabaseTasks.create_all
• db:create:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: name)
• db:drop
  • forwards to "db:drop:_unsafe"
• db:drop:_unsafe
  • forwards to ActiveRecord::Tasks::DatabaseTasks.drop_current
• db:drop:all
  • forwards to ActiveRecord::Tasks::DatabaseTasks.drop_all
• db:drop:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: name)
• [~] db:encryption:init
  • orthogonal to tenanting
• db:environment:set
  • uses ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool which raises NoTenantError if tenanted is primary
• [~] db:fixtures:identify
  • orthogonal to tenanting
• db:fixtures:load
  • I think the current behavior is the desired behavior: use ARTENANT (fallback to default tenant) as the target database
• db:forward with support for STEP
  • migration
  • calls ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool.migration_context.forward(step)
• [~] db:load_config
  • orthogonal to tenanting
• db:migrate with support for VERSION and SCOPE
  • migration
  • calls ActiveRecord::Tasks::DatabaseTasks.migrate_all
• db:migrate:down with support for VERSION
  • migration
  • calls ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool.migration_context.run(:down, target_version)
• db:migrate:down:__dbname__
  • migration
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
  • calls pool.migration_context.run(:down, ActiveRecord::Tasks::DatabaseTasks.target_version)
• db:migrate:__dbname__
  • migration
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.migrate
• db:migrate:redo with support for STEP and VERSION
  • migration
  • forwards to either:
    • "db:migrate:down" + "db:migrate:up"
    • "db:rollback" + "db:migrate"
• db:migrate:redo:__dbname__
  • migration
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • forwards to either:
    • "db:migrate:down:dbname" + "db:migrate:up:dbname"
    • "db:rollback:dbname" + "db:migrate:dbname"
• [~] db:migrate:reset
  • invokes "db:drop", "db:create", "db:schema:dump", "db:migrate"
• db:migrate:status
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each
  • calls ActiveRecord::Tasks::DatabaseTasks.migrate_status
• db:migrate:status:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.migrate_status
• db:migrate:up with support for VERSION
  • migration
  • calls ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool.migration_context.run(:up, target_version)
• db:migrate:up:__dbname__
  • migration
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
  • calls pool.migration_context.run(:up, ActiveRecord::Tasks::DatabaseTasks.target_version)
• db:prepare
  • forwards to ActiveRecord::Tasks::DatabaseTasks.prepare_all
• db:purge (see Known Issues below)
  • forwards to ActiveRecord::Tasks::DatabaseTasks.purge_current
• db:purge:all (see Known Issues below)
  • forwards to ActiveRecord::Tasks::DatabaseTasks.purge_all
• db:reset
  • forwards to "db:drop", "db:create", "db:schema:dump", "db:migrate"
• db:reset:all
  • forwards to "db:drop", "db:setup"
• db:reset:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • forwards to "db:drop:dbname", "db:setup:dbname"
• db:rollback with support for STEP
  • migration
  • calls ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool.migration_context.rollback(step)
• db:rollback:__dbname__
  • migration
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name)
  • calls pool.migration_context.rollback(step)
• db:schema:cache:clear
  • iterates with ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env).each
  • we can probably skip implementing this since the gem needs the schema cache to operate, but whatever, maybe it's useful to force a re-dump
• db:schema:cache:dump
  • iterates with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each
  • we can probably skip implementing this since the gem automatically dumps the schema cache for tenanted databases
• db:schema:dump
  • forwards to ActiveRecord::Tasks::DatabaseTasks.dump_all
  • I think we can skip this? Check that the gem always dumps the schema.
• db:schema:dump:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • iterates with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.dump_schema
  • I think we can skip this? Check that the gem always dumps the schema.
• db:schema:load
  • forwards to ActiveRecord::Tasks::DatabaseTasks.load_schema_current(nil, ENV["SCHEMA"])
• db:schema:load:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.load_schema
  • I think we can skip this since the schema is automatically applied at tenant creation time
• db:seed
  • forwards to ActiveRecord::Tasks::DatabaseTasks.load_seed
  • I think the current behavior is the desired behavior: use ARTENANT (fallback to default tenant) as the target database
• db:seed:replant
  • forwards to :load_config, :truncate_all, :seed
• db:setup
  • forwards to "db:create", :environment, "db:schema:load", :seed
• db:setup:all
  • forwards to "db:create", :environment, "db:schema:load", :seed
• db:setup:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • forwards to "db:create:dbname", :environment, "db:schema:load:dbname", "db:seed"
• db:test:load_schema
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: "test")
  • calls ActiveRecord::Tasks::DatabaseTasks.load_schema
• db:test:load_schema:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: "test", name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.load_schema
• db:test:prepare
  • forwards to "db:test:load_schema"
• db:test:prepare:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • forwards to "db:test:load_schema:dbname"
• db:test:purge
  • implemented with ActiveRecord::Base.configurations.configs_for(env_name: "test")
  • calls ActiveRecord::Tasks::DatabaseTasks.purge(db_config)
• db:test:purge:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: "test", name: name)
  • calls ActiveRecord::Tasks::DatabaseTasks.purge(db_config)
• db:truncate_all
  • forwards to ActiveRecord::Tasks::DatabaseTasks.truncate_all
• db:version
  • implemented with ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env)
• db:version:__dbname__
  • created by ActiveRecord::Tasks::DatabaseTasks.for_each(databases)
  • implemented with ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: name)

[miguelmarcondesf]

Hey @flavorjones, very cool that you compiled all the work still needed!
What do you expect from this roadmap?
Could people look at something from the list and open a PR related to the work described?
I ask because some colleagues were also interested in contributing

[flavorjones]

Hi @miguelmarcondesf, thanks for asking. I feel like this is going to be a lot of work, and that makes me want to

1. ruthlessly prioritize
2. try really hard to find shortcuts that will re-use the existing Rails rake tasks

I haven't been able to see any great shortcuts clearly, so I would like to ruthlessly prioritize.

I think the most important missing rake task features are around migration support:

• db:migrate support for VERSION and SCOPE
• db:migrate:up and db:migrate:down and db:rollback
• db:migrate:redo

It would also be helpful if someone who's not me would validate that the following existing rake tasks in this gem do what's expected:

• db:migrate
• db:prepare
• db:drop
• db:reset

I personally don't use most of the rest of the tasks, and so I don't personally have strong feelings about whether or when the long tail of database tasks gets implemented.

[lipedjow]

upvote for db:rollback support

[kyletully]

Just wanted to say thank you ahead of time... And for all of your amazing work thus far. This is really quite something. 🖤

Unfortunately the general scope and complexity of this goes well over my head at the moment, but it's cool to watch some real software engineering take place. 💪😎