RFD-0000 - Enhanced Event Data for Derivative Microservices

[joecorall]

This RFD proposes enhancements to Islandora's event system, addressing incomplete event data and lack of auditing/retries/orchestration for events. It outlines two solutions: passing complete event data via an X-Islandora-Event header and replacing the current Java-based stack with a Drupal-native queue system using Symfony Messenger.

You can read it easily here: https://github.com/Islandora-Labs/rfds/tree/RFD-0000/0000 and make comment to the README/RFD like any GitHub PR code review using inline comments and approving/disapproving the PR using GitHub reviews and/or comments.

This is the first RFD we're doing as a community, and I sort of just did this since it felt like the best way to get this conversation started. This came about from the multiple requests in Slack for the "create a PDF of all children images for paged content items" that we have working in Lehigh's Islandora repository. This RFD describes the technical blockers I've made local workaround for at Lehigh that need broader community solutions for to make the feature possible/maintainable for other Islandora community members.

FWIW here is another open source community's process for this sort of thing: https://github.com/folio-org/rfcs. For technical decisions, i'd prefer to be less formal than that spec and just ensure broad community participation/awareness, and leverage GitHub's approval workflow to confirm decisions.

[aOelschlager]

I think this would be a big improvement to the current stack.

[Natkeeran]

This addresses several challenges with respect to the current microservices workflow:

• stack complexity
• Ability to monitor/track tasks via Drupal UI

A good improvement to the stack.

[joshdentremont]

This sounds awesome. I'm currently looking into adding cover page generation to our site, so I would be in favour of both the short term and long term proposals. I'd also be happy to contribute to testing and documenting both processes.

[rosiel]

i agree with Annie, Nat, and Josh. This would be a good improvement (both in the short and long terms).

[mitchmac]

@joecorall couple of initial questions (apologies if I missed these in the RFD!):

• It seems like the default Doctrine transport could be swapped if someone found the need, does that lineup with what you're thinking?
• Would Symfony Messenger's worker be the default queue worker? If so, is the noted challenge "Queue worker reliability considerations (cron vs. dedicated workers)" summed up a bit here (https://symfony.com/doc/current/messenger.html#deploying-to-production)?

[joecorall]

@mitchmac

It seems like the default Doctrine transport could be swapped if someone found the need, does that lineup with what you're thinking?

Sure, but I think we should focus community support on a single solution that supports big and small. I think doctrine does that.

Would Symfony Messenger's worker be the default queue worker? If so, is the noted challenge "Queue worker reliability considerations (cron vs. dedicated workers)" summed up a bit here

TBD but probably. I think we'll find the best path during implementation

[kayakr]

@joecorall I'm keen to see this happen, having experimented with SM over at Islandora/documentation#1627 (comment). The RFD mentions islandora_paged_content https://github.com/Islandora-Labs/rfds/pull/1/files#diff-2f46a53c449991737dc8c3c625d117c972f2e9989b88e051bf3ef1f0d7f602eaR280 - is there working code for the proposed extension points?

[joecorall]

@kayakr - we're almost done with the short term solutions. Having heard no -1's on this proposal I think we can probably merge this PR and begin work on the long term solutions