Skip to content

Add DataPass pull poller (API-6754)#6

Open
Un3x wants to merge 4 commits into
mainfrom
feature/api-6754-pull-datapass-api-poller-des-habilitations-proactivite
Open

Add DataPass pull poller (API-6754)#6
Un3x wants to merge 4 commits into
mainfrom
feature/api-6754-pull-datapass-api-poller-des-habilitations-proactivite

Conversation

@Un3x
Copy link
Copy Markdown
Collaborator

@Un3x Un3x commented May 18, 2026

Summary

  • Periodic GoodJob cron pulls validated authorization_requests from DataPass and idempotently creates ProactiviteRequest rows.
  • New layers under Datapass:: mirror the CNOUS layout: configuration value object, Faraday HTTP wrapper with TransientError/PermanentError translation, four interactors organized by Datapass::PullEnrollments, a single-row DatapassPullWatermark model, PullDatapassEnrollmentsJob, a datapass:pull rake + CLI for manual replay or external schedulers.
  • The model's existing after_create_commit callback enqueues ProcessProactiviteRequestJob (no double-wiring), and the external_request_id column is renamed to datapass_authorization_id to reflect the now-single external source.
  • Daily cron at 01:00 UTC (DATAPASS_PULL_CRON overridable). First run watermark is nil, so the backlog of currently-validated CNOUS authorizations gets ingested on deploy.
  • Replaces functionally the cancelled webhook plan (API-6716).

Architecture

GoodJob cron (daily 01:00 UTC)
  └─> PullDatapassEnrollmentsJob#perform
        └─> Datapass::PullEnrollments  (organizer)
              ├─> Authenticate                  (OAuth2 client_credentials → bearer)
              ├─> ListValidatedAuthorizations   (paginate + filter on definition_id + last_validated_at >= since)
              ├─> CreateProactiviteRequests     (find_or_create_by datapass_authorization_id)
              └─> AdvanceWatermark              (last_validated_at = max(entries))

Decisions (see plan)

  • Rename external_request_iddatapass_authorization_id (no second column).
  • Separate DATAPASS_AUTH_URL and DATAPASS_API_BASE_URL env vars (mirror CNOUS).
  • Fallback data['niveau_bourse_min'] || data['echelon_bourse'] for the scholarship-level mapping (DataPass field name unsettled).
  • Daily cron 0 1 * * *; carve-out flagged if API-6726 / API-6750 mandates an external scheduler — rake datapass:pull + CLI already supports it.
  • First-run watermark nil so the existing CNOUS habilitation backlog is ingested.
  • Watermark advances past every entry the poller saw (incl. per-row validation failures, which Sentry-report instead of pinning the cycle).

Test plan

  • bundle exec rspec — 169 examples / 0 failures
  • bundle exec rubocop — 67 files / 0 offenses
  • bundle exec brakeman — 0 warnings
  • R-005 regression spec on Datapass::Authenticate proves the bearer token never reaches Rails.logger
  • Replay safety: re-running the organizer with the same DataPass entries creates no duplicate rows and enqueues no duplicate jobs
  • Failure paths: 5xx → TransientError → job retries; 4xx → PermanentError → job discards + Sentry; mid-pagination crash leaves the watermark untouched
  • Real DataPass call — blocked on API-6753 (ops provisioning)

🤖 Generated with Claude Code

The column was originally speculative — anything-external-could-go-here.
V1 has exactly one external source (DataPass), so the name should
reflect that. Drops the abstraction in favor of clarity.

The unique index follows the column automatically (Postgres rename_column).

Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
@Un3x Un3x force-pushed the feature/api-6754-pull-datapass-api-poller-des-habilitations-proactivite branch from 0143b5a to 14ba365 Compare May 18, 2026 13:10
Un3x added 2 commits May 18, 2026 17:33
Periodically pull validated authorization_requests from DataPass and
idempotently create ProactiviteRequest rows. Replaces functionally the
cancelled webhook plan from API-6716.

The adapter mirrors the CNOUS layout already in place:

  Datapass::Configuration       env-driven (auth/api URLs, client creds,
                                definition_id slug); empty strings are
                                treated as unset so a copied .env.template
                                doesn't crash boot
  Datapass::HttpConnection      Faraday wrapper; 5xx/401/connect →
                                TransientError, 4xx → PermanentError
  Datapass::{Transient,Permanent}Error
                                separated retry/discard surfaces
  Datapass::Authenticate        OAuth2 client_credentials → bearer token,
                                never logged (R-005)
  Datapass::ListValidatedAuthorizations
                                paginated GET, client-side filter on
                                definition_id + last_validated_at ≥ since;
                                missing last_validated_at on incremental
                                runs is dropped with a Sentry report (we
                                can't tell if it's newer than the
                                watermark)
  Datapass::CreateProactiviteRequests
                                idempotent insert per public_id, relying
                                on the unique constraint as the source of
                                truth so the exists?-then-create! race
                                cannot deadlock the poller; entries are
                                walked in last_validated_at order so the
                                watermark advance can be sealed at the
                                first validation failure (failed rows
                                pin the watermark, get retried each pull,
                                Sentry alerts the operator)
  Datapass::AdvanceWatermark    monotonic, no-rewind update
  Datapass::PullEnrollments     organizer wiring the four interactors

State lives in a singleton DatapassPullWatermark row (check constraint
id = 1, ActiveRecord accessor #current). First-run last_validated_at is
nil so the backlog of currently-validated authorizations is ingested on
deploy.

Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
Register the pull as an in-app GoodJob cron entry (daily 01:00 UTC,
overridable via DATAPASS_PULL_CRON). PullDatapassEnrollmentsJob calls
the Datapass::PullEnrollments organizer with the current watermark;
TransientError retries 5× with polynomial backoff, PermanentError
discards and reports to Sentry.

The same pipeline is reachable from the CLI via Datapass::PullEnrollmentsCli
+ rake datapass:pull — useful for manual replay and for the carve-out
where API-6726 / API-6750 settles on an external scheduler instead of
the in-app cron. CLI prints a one-line summary (count + watermark) and
exits 0/1/2 for success/permanent/transient.

.env.template documents the new vars: separate DATAPASS_AUTH_URL and
DATAPASS_API_BASE_URL (CNOUS convention — safer if auth host diverges
from API host), client creds, optional cron override, optional
definition_id slug override.

Signed-off-by: Thomas COMES <thomas.comes.tech@gmail.com>
@Un3x Un3x force-pushed the feature/api-6754-pull-datapass-api-poller-des-habilitations-proactivite branch from 14ba365 to 5633ad4 Compare May 18, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant