Question about active_record-tenanted removed on Fizzy

[lairtonmendes]

Hi @flavorjones

I noticed that active_record-tenanted was removed from Fizzy in PR #1558.
Could you share a brief summary of the reasoning behind that decision?

Also:
• Do you still have plans to make active_record-tenanted production-ready?
• Is there any intention to reintroduce it into Fizzy in the future?

Thanks in advance, any high-level context would already be very helpful. We’re currently building a product we plan to launch in the second quarter from next year, and sharing the challenges you ran into could be extremely valuable. So far we haven’t faced major issues, but we may not yet be operating at the scale where those problems start to appear.

[flavorjones]

@lairtonmendes The short answer is that activerecord-tenanted was working very well in Fizzy, and I consider it stable enough to run in production.

But for Fizzy we had additional requirements: we wanted to have sqlite replication to hot read replicas around the globe with automatic failover between data centers; and that replication/failover functionality was not where we needed it to be for a global product launch.

Specifically, the constraint that sqlite writes can only happen on the primary/writer node required tight integration between the database and the routing tier, and also required some changes to Solid Queue to implement passive/active status. And: we mostly had it all working! But like any distributed system, there were interesting failure modes that we simply needed wall clock time to work through, and the product release couldn't wait.

@kevinmcconnell and I are planning to write a blog post about this when we come up for air with a bit more detail and also our plans to continue working on this infrastructure in a future project. Stay tuned!