Skip to content

Implement CSV import #118

Merged
notjackl3 merged 10 commits intomainfrom
backend/import-csv
Mar 23, 2026
Merged

Implement CSV import #118
notjackl3 merged 10 commits intomainfrom
backend/import-csv

Conversation

@1michhu1
Copy link
Copy Markdown
Contributor

@1michhu1 1michhu1 commented Mar 18, 2026

Implemented CSV import based on the format of the Blank Membership List for Blueprint.

Response shape:

{
  status: "success" | "partial_success" | "failed",
  summary: {
    totalRows: int,
    parsedSucceeded: int,
    parseFailed: int,
    dbSucceeded: int,
    dbFailed: int
  },
  parseErrors: [
    {
      rowIndex: int,
      column: string* (see Blank Membership List for possible columns),
      message: string (ie. 'Missing required volunteer name')
    }
  ],
  dbErrors: [
    {
      rowIndex: number;
      message: string;
      pg_code?: string (PostgreSQL error code);
    }
  ]
}

Important points about response shape and implementation

  • import_csv response returns whether some, all or none of the imported csv rows were created based on the status (success, partial_success, failed)
  • This function tries to insert rows on a best effort basis. There are two types of errors which describe why a volunteer could not be inserted:
    1. ParseError: Either Papaparse failed to parse the csv row, or a value(s) in a row did not align with the expected formatting from the Blank Membership List. This is listed in summary.parseErrors
    2. DbError: A row fails to be upserted into Roles, Cohorts, Volunteers, VolunteerRoles or VolunteerCohorts. This could be due to a check constraint violation or some other database error listed in summary.dbErrors. To note, the RPC always attempts to upsert into all of these tables to avoid duplicate key errors.
  • rowIndex in errors corresponds to the actual index of that volunteer in the raw csv. rowIndex skips the header and starts counting from the first data row
  • The function does not attempt to insert rows that failed the parsing stage
  • The package Papaparse was installed to simplify csv parsing
  • Created Roles default to is_active = true
  • The RPC
    • upsert_volunteer_with_roles_and_cohorts was created to easily rollback db changes on errors. This is because cleanup on up to 5 tables would otherwise be difficult
    • The RPC tries to match csv volunteers to rows in the Volunteers table by email and name. csv data will overwite Volunteer table data on email and name match
  • A migration file was manually added to account for changes in the remote schema. I believe it would be helpful to add supabase db pull --db-url [db-url with password] to our workflow and readme in order to sync up schema changes.
  • Local database schema changes have not yet been pushed to the remote

Other Notes:

  • Another test suite unrelated to this feature was edited to fix a minor database cleanup bug
  • A folder for test data was added to /tests
  • Currently, the supabase

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
trcc Ignored Ignored Mar 23, 2026 3:20pm

@1michhu1 1michhu1 added the backend this ticket is part of the backend label Mar 19, 2026
@1michhu1 1michhu1 linked an issue Mar 19, 2026 that may be closed by this pull request
@1michhu1 1michhu1 marked this pull request as ready for review March 19, 2026 02:56
@notjackl3 notjackl3 self-requested a review March 23, 2026 14:49
@notjackl3 notjackl3 force-pushed the backend/import-csv branch from 36d1847 to c7b16af Compare March 23, 2026 15:09
Copy link
Copy Markdown
Contributor

@notjackl3 notjackl3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@notjackl3 notjackl3 force-pushed the backend/import-csv branch from c7b16af to 83f0af4 Compare March 23, 2026 15:20
@notjackl3 notjackl3 merged commit 29bb8cb into main Mar 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend this ticket is part of the backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BE 2.0 - Implement import_csv function

2 participants