Add DataPass pull poller (API-6754)#6
Open
Un3x wants to merge 4 commits into
Open
Conversation
The column was originally speculative — anything-external-could-go-here. V1 has exactly one external source (DataPass), so the name should reflect that. Drops the abstraction in favor of clarity. The unique index follows the column automatically (Postgres rename_column). Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
0143b5a to
14ba365
Compare
Periodically pull validated authorization_requests from DataPass and
idempotently create ProactiviteRequest rows. Replaces functionally the
cancelled webhook plan from API-6716.
The adapter mirrors the CNOUS layout already in place:
Datapass::Configuration env-driven (auth/api URLs, client creds,
definition_id slug); empty strings are
treated as unset so a copied .env.template
doesn't crash boot
Datapass::HttpConnection Faraday wrapper; 5xx/401/connect →
TransientError, 4xx → PermanentError
Datapass::{Transient,Permanent}Error
separated retry/discard surfaces
Datapass::Authenticate OAuth2 client_credentials → bearer token,
never logged (R-005)
Datapass::ListValidatedAuthorizations
paginated GET, client-side filter on
definition_id + last_validated_at ≥ since;
missing last_validated_at on incremental
runs is dropped with a Sentry report (we
can't tell if it's newer than the
watermark)
Datapass::CreateProactiviteRequests
idempotent insert per public_id, relying
on the unique constraint as the source of
truth so the exists?-then-create! race
cannot deadlock the poller; entries are
walked in last_validated_at order so the
watermark advance can be sealed at the
first validation failure (failed rows
pin the watermark, get retried each pull,
Sentry alerts the operator)
Datapass::AdvanceWatermark monotonic, no-rewind update
Datapass::PullEnrollments organizer wiring the four interactors
State lives in a singleton DatapassPullWatermark row (check constraint
id = 1, ActiveRecord accessor #current). First-run last_validated_at is
nil so the backlog of currently-validated authorizations is ingested on
deploy.
Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
Register the pull as an in-app GoodJob cron entry (daily 01:00 UTC, overridable via DATAPASS_PULL_CRON). PullDatapassEnrollmentsJob calls the Datapass::PullEnrollments organizer with the current watermark; TransientError retries 5× with polynomial backoff, PermanentError discards and reports to Sentry. The same pipeline is reachable from the CLI via Datapass::PullEnrollmentsCli + rake datapass:pull — useful for manual replay and for the carve-out where API-6726 / API-6750 settles on an external scheduler instead of the in-app cron. CLI prints a one-line summary (count + watermark) and exits 0/1/2 for success/permanent/transient. .env.template documents the new vars: separate DATAPASS_AUTH_URL and DATAPASS_API_BASE_URL (CNOUS convention — safer if auth host diverges from API host), client creds, optional cron override, optional definition_id slug override. Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
14ba365 to
5633ad4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
authorization_requestsfrom DataPass and idempotently createsProactiviteRequestrows.Datapass::mirror the CNOUS layout: configuration value object, Faraday HTTP wrapper with TransientError/PermanentError translation, four interactors organized byDatapass::PullEnrollments, a single-rowDatapassPullWatermarkmodel,PullDatapassEnrollmentsJob, adatapass:pullrake + CLI for manual replay or external schedulers.after_create_commitcallback enqueuesProcessProactiviteRequestJob(no double-wiring), and theexternal_request_idcolumn is renamed todatapass_authorization_idto reflect the now-single external source.DATAPASS_PULL_CRONoverridable). First run watermark isnil, so the backlog of currently-validated CNOUS authorizations gets ingested on deploy.Architecture
Decisions (see plan)
external_request_id→datapass_authorization_id(no second column).DATAPASS_AUTH_URLandDATAPASS_API_BASE_URLenv vars (mirror CNOUS).data['niveau_bourse_min'] || data['echelon_bourse']for the scholarship-level mapping (DataPass field name unsettled).0 1 * * *; carve-out flagged if API-6726 / API-6750 mandates an external scheduler —rake datapass:pull+ CLI already supports it.nilso the existing CNOUS habilitation backlog is ingested.Test plan
bundle exec rspec— 169 examples / 0 failuresbundle exec rubocop— 67 files / 0 offensesbundle exec brakeman— 0 warningsDatapass::Authenticateproves the bearer token never reachesRails.logger🤖 Generated with Claude Code