Description:
Following investigation of the data destruction (DD) process, several QC improvements are needed. The DD order of operations has been updated so that all delete operations must succeed before a participant is marked as data-destroyed. This issue tracks the remaining follow-up work tied to the April release.
Background
The previous DD process could mark a participant as dataHasBeenDestroyed = true even if sub-collection deletions failed partway through. Error handling has been improved, but we need a recurring audit and cleanup of legacy records.
Tasks
•Monthly audit function: Build a Cloud Function triggered by Cloud Scheduler to run monthly. The function audits all participants where dataHasBeenDestroyed == true and reports any orphaned documents across all DD-related collections, pathology report storage files, and residual stub fields.
•Include a positive check that ensures all expected stub fields do exist. Note that we'll need to determine which stub fields to check and whether all stub fields are always present, depending on each participant's journey in Connect before data destruction.
•GCP error alerting: Set up alerting (email to ConnectCC@nih.gov) when errors occur during the DD process. Evaluate a modular approach if this pattern will be reused across other processes, such as DHQ credentialing.
Stub record updates:
•Remove authenticationEmail (421823980) from stub retention — confirmed not needed; participant can still log in to MyConnect.
•Add dateRevokedHIPAA to the stub record.
•Add dateOfDataDestroyed to the stub record (must be dependent on the success of all delete operations).
--
Note: Reasons for refusal/withdrawal are top-level (not nested). New reason CIDs must be individually added to the stub definition going forward. Amelia to update the Stub Variables list.
Audit log output: Monthly audit logs posted to Box for Analytics team review. Analytics to communicate any inconsistencies to the team.
SOP updates: Document DD error alerting and monthly audit in the Data Destruction SOP. Evaluate whether the quarterly BQ1 manual check by analytics staff can be replaced by the automated monthly audit.
Description:
Following investigation of the data destruction (DD) process, several QC improvements are needed. The DD order of operations has been updated so that all delete operations must succeed before a participant is marked as data-destroyed. This issue tracks the remaining follow-up work tied to the April release.
Background
The previous DD process could mark a participant as dataHasBeenDestroyed = true even if sub-collection deletions failed partway through. Error handling has been improved, but we need a recurring audit and cleanup of legacy records.
Tasks
•Monthly audit function: Build a Cloud Function triggered by Cloud Scheduler to run monthly. The function audits all participants where dataHasBeenDestroyed == true and reports any orphaned documents across all DD-related collections, pathology report storage files, and residual stub fields.
•Include a positive check that ensures all expected stub fields do exist. Note that we'll need to determine which stub fields to check and whether all stub fields are always present, depending on each participant's journey in Connect before data destruction.
•GCP error alerting: Set up alerting (email to ConnectCC@nih.gov) when errors occur during the DD process. Evaluate a modular approach if this pattern will be reused across other processes, such as DHQ credentialing.
Stub record updates:
•Remove authenticationEmail (421823980) from stub retention — confirmed not needed; participant can still log in to MyConnect.
•Add dateRevokedHIPAA to the stub record.
•Add dateOfDataDestroyed to the stub record (must be dependent on the success of all delete operations).
--
Note: Reasons for refusal/withdrawal are top-level (not nested). New reason CIDs must be individually added to the stub definition going forward. Amelia to update the Stub Variables list.
Audit log output: Monthly audit logs posted to Box for Analytics team review. Analytics to communicate any inconsistencies to the team.
SOP updates: Document DD error alerting and monthly audit in the Data Destruction SOP. Evaluate whether the quarterly BQ1 manual check by analytics staff can be replaced by the automated monthly audit.