Implement API Endpoints for bulk-import of Schools (#622)

## Status

- Related to
RaspberryPiFoundation/experience-cs#1309

## Points for consideration:

- 🧠 Review - Reviewers may find it easier to review commit-by-commit
- ❓ Exclude - do we want to keep the wrapper scripts and test data?
- 📖 Documentation - should we migrate the documentation for CSMs out of
here?
- 🔒 Security - calling these endpoints should only be possible for users
with `editor-admin` or `experience-cs-admin` roles. All other roles and
none should return `401`.
- 🐌 Performance - The schools are created asynchronously through
GoodJob, so immediate performance of the endpoints should not be an
issue, however this code does make an N+1 query to `userinfo` to look up
school owners by email - there is no bulk option for this.

## What's changed?

- Introduction of two new endpoints to allow import and validation of
schools via CSV files:
  - `/api/schools/import` - for POSTing CSV
  - `/api/school_import_jobs` - for tracking job status
- Both endpoints require a user to have the `editor-admin` or
`experience-cs-admin` roles.
- An `ImportSchoolsJob` GoodJob job type that will execute the import
asynchronously
- A `SchoolImportResult` record type that tracks the results of
`ImportSchoolsJob`s.

## Steps to perform after deploying to production

There is a database migration involved in this to add the table that
tracks `SchoolImportResult`.

## Tasks Outstanding

- [x] Improve consistency of naming (I have used `import_schools` in
some places and `school_import` in others).
- [x] Check that we validate all of the data correctly
- [x] Stress-test with some very large imports to get a sense of upper
bounds.
- [x] Write a tool to make it easy to get an `editor-api` OAuth token
- [x] Merge the branch that adds UI in the `/admin` path
- [x] Remove the wrapper scripts in favour of web-based UI
- [x] Remove test data commit.
- [x] Rewrite the CSM guide to refer to the new Admin UI.

## Follow-On Work

~This is not the final form of this feature. In a follow-up task in the
next sprint we intend to add a UI under the `/admin` route to make it
easier for non-developers to access this endpoint. The goal in the first
instance is to develop and test the endpoints separately from the UI
that accesses them.~

This PR now contains all the work necessary to make a complete feature,
including web-based UI.


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Adds CSV-based bulk school import with admin UI, async GoodJob
processing, job status API, and persisted results (with CSV export),
restricted to editor/experience CS admins.
> 
> - **API/Services**:
> - `POST /api/schools/import` to start imports; `GET
/api/school_import_jobs/:id` to track status.
> - CSV parsing/validation via `School::ImportBatch` with structured
`SchoolImportError` codes.
> - **Background Processing**:
> - `SchoolImportJob` (queue: `import_schools_job`) creates/verifies
schools, assigns owner role, and saves results to `SchoolImportResult`.
>   - GoodJob queues updated in `config/initializers/good_job.rb`.
> - **Admin UI**:
> - New `admin/school_import_results` pages (index/show/new) for upload,
status, results, and CSV download.
> - Custom Administrate fields: `StatusField`, `ResultsSummaryField`,
`UserInfoField` with batched user lookups.
> - **Data Model**:
>   - New `SchoolImportResult` model/table (+ migration) and dashboard.
> - **Authorization**:
> - CanCan abilities grant `School#import` and `:school_import_job` read
to editor/experience CS admins.
> - **Utilities/Config**:
> - `UserInfoApiClient` adds `find_user_by_email` and improved
transforms.
>   - Locale for missing CSV error; routes wired for admin/API.
> - **Docs/Tests**:
>   - CSM guide and CSV template added under `docs/import/`.
>   - Specs for import service, job, API endpoints, and admin pages.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
6eacf9a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Author: fspeirs

app/controllers/admin/school_import_results_controller.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+require 'csv'
+
+module Admin
+  class SchoolImportResultsController < Admin::ApplicationController
+    def index
+      search_term = params[:search].to_s.strip
+      resources = Administrate::Search.new(
+        SchoolImportResult.all,
+        dashboard_class,
+        search_term
+      ).run
+      resources = apply_collection_includes(resources)
+      resources = order.apply(resources)
+      resources = resources.page(params[:_page]).per(records_per_page)
+
+      # Batch load user info to avoid N+1 queries
+      user_ids = resources.filter_map(&:user_id).uniq
+      RequestStore.store[:user_info_cache] = fetch_users_batch(user_ids)
+
+      page = Administrate::Page::Collection.new(dashboard, order: order)
+
+      render locals: {
+        resources: resources,
+        search_term: search_term,
+        page: page,
+        show_search_bar: show_search_bar?
+      }
+    end
+
+    def show
+      respond_to do |format|
+        format.html do
+          render locals: {
+            page: Administrate::Page::Show.new(dashboard, requested_resource)
+          }
+        end
+        format.csv do
+          send_data generate_csv(requested_resource),
+                    filename: "school_import_#{requested_resource.job_id}_#{Date.current.strftime('%Y-%m-%d')}.csv",
+                    type: 'text/csv'
+        end
+      end
+    end
+
+    def new
+      @error_details = flash[:error_details]
+      render locals: {
+        page: Administrate::Page::Form.new(dashboard, SchoolImportResult.new)
+      }
+    end
+
+    def create
+      if params[:csv_file].blank?
+        flash[:error] = I18n.t('errors.admin.csv_file_required')
+        redirect_to new_admin_school_import_result_path
+        return
+      end
+
+      # Call the same service that the API endpoint uses, ensuring all validation is applied
+      result = School::ImportBatch.call(
+        csv_file: params[:csv_file],
+        current_user: current_user
+      )
+
+      if result.success?
+        flash[:notice] = "Import job started successfully. Job ID: #{result[:job_id]}"
+        redirect_to admin_school_import_results_path
+      else
+        # Display error inline on the page
+        flash.now[:error] = format_error_message(result[:error])
+        @error_details = extract_error_details(result[:error])
+        render :new, locals: {
+          page: Administrate::Page::Form.new(dashboard, SchoolImportResult.new)
+        }
+      end
+    end
+
+    private
+
+    def default_sorting_attribute
+      :created_at
+    end
+
+    def default_sorting_direction
+      :desc
+    end
+
+    def format_error_message(error)
+      return error.to_s unless error.is_a?(Hash)
+
+      error[:message] || error['message'] || 'Import failed'
+    end
+
+    def extract_error_details(error)
+      return nil unless error.is_a?(Hash)
+
+      error[:details] || error['details']
+    end
+
+    def generate_csv(import_result)
+      CSV.generate(headers: true) do |csv|
+        # Header row
+        csv << ['Status', 'School Name', 'School Code', 'School ID', 'Owner Email', 'Error Code', 'Error Message']
+
+        results = import_result.results
+        successful = results['successful'] || []
+        failed = results['failed'] || []
+
+        # Successful schools
+        successful.each do |school|
+          csv << [
+            'Success',
+            school['name'],
+            school['code'],
+            school['id'],
+            school['owner_email'],
+            '',
+            ''
+          ]
+        end
+
+        # Failed schools
+        failed.each do |school|
+          csv << [
+            'Failed',
+            school['name'],
+            '',
+            '',
+            school['owner_email'],
+            school['error_code'],
+            school['error']
+          ]
+        end
+      end
+    end
+
+    def fetch_users_batch(user_ids)
+      return {} if user_ids.empty?
+
+      users = UserInfoApiClient.fetch_by_ids(user_ids)
+      return {} if users.nil?
+
+      users.index_by { |user| user[:id] }
+    rescue StandardError => e
+      Rails.logger.error("Failed to batch fetch user info: #{e.message}")
+      {}
+    end
+  end
+end

app/controllers/api/school_import_jobs_controller.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Api
+  class SchoolImportJobsController < ApiController
+    before_action :authorize_user
+
+    def show
+      authorize! :read, :school_import_job
+      @job = find_job
+
+      if @job.nil?
+        render json: SchoolImportError.format_error(:job_not_found, 'Job not found'), status: :not_found
+        return
+      end
+
+      @status = job_status(@job)
+      @result = SchoolImportResult.find_by(job_id: @job.active_job_id) if @job.succeeded?
+      render :show, formats: [:json], status: :ok
+    end
+
+    private
+
+    def find_job
+      job = GoodJob::Job.find_by(active_job_id: params[:id])
+
+      # Verify this is an import job (security check)
+      return nil unless job && job.job_class == SchoolImportJob.name
+
+      job
+    end
+
+    def job_status(job)
+      return 'discarded' if job.discarded?
+      return 'succeeded' if job.succeeded?
+      return 'failed' if job_failed?(job)
+      return 'running' if job.running?
+      return 'scheduled' if job_scheduled?(job)
+
+      'queued'
+    end
+
+    def job_failed?(job)
+      job.finished? && job.error.present?
+    end
+
+    def job_scheduled?(job)
+      job.scheduled_at.present? && job.scheduled_at > Time.current
+    end
+  end
+end

app/controllers/api/schools_controller.rb

@@ -4,6 +4,7 @@ module Api
   class SchoolsController < ApiController
     before_action :authorize_user
     load_and_authorize_resource
+    skip_load_and_authorize_resource only: :import
 
     def index
       @schools = School.accessible_by(current_ability)
@@ -47,6 +48,29 @@ def destroy
       end
     end
 
+    def import
+      authorize! :import, School
+
+      if params[:csv_file].blank?
+        render json: { error: SchoolImportError.format_error(:csv_file_required, 'CSV file is required') },
+               status: :unprocessable_entity
+        return
+      end
+
+      result = School::ImportBatch.call(
+        csv_file: params[:csv_file],
+        current_user: current_user
+      )
+
+      if result.success?
+        @job_id = result[:job_id]
+        @total_schools = result[:total_schools]
+        render :import, formats: [:json], status: :accepted
+      else
+        render json: { error: result[:error] }, status: :unprocessable_entity
+      end
+    end
+
     private
 
     def school_params

app/dashboards/school_import_result_dashboard.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require 'administrate/base_dashboard'
+
+class SchoolImportResultDashboard < Administrate::BaseDashboard
+  ATTRIBUTE_TYPES = {
+    id: Field::String,
+    job_id: StatusField,
+    user_id: UserInfoField,
+    results: ResultsSummaryField,
+    created_at: Field::DateTime,
+    updated_at: Field::DateTime
+  }.freeze
+
+  COLLECTION_ATTRIBUTES = %i[
+    job_id
+    user_id
+    results
+    created_at
+  ].freeze
+
+  SHOW_PAGE_ATTRIBUTES = %i[
+    id
+    job_id
+    user_id
+    results
+    created_at
+    updated_at
+  ].freeze
+
+  FORM_ATTRIBUTES = [].freeze
+
+  COLLECTION_FILTERS = {}.freeze
+
+  def display_resource(school_import_result)
+    "Import Job #{school_import_result.job_id}"
+  end
+end

app/fields/results_summary_field.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require 'administrate/field/base'
+
+class ResultsSummaryField < Administrate::Field::Base
+  def to_s
+    "#{successful_count} successful, #{failed_count} failed"
+  end
+
+  def successful_count
+    data['successful']&.count || 0
+  end
+
+  def failed_count
+    data['failed']&.count || 0
+  end
+
+  def total_count
+    successful_count + failed_count
+  end
+end

app/fields/status_field.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require 'administrate/field/base'
+
+class StatusField < Administrate::Field::Base
+  def to_s
+    job_status
+  end
+
+  def job_status
+    return 'unknown' if data.blank?
+
+    job = GoodJob::Job.find_by(active_job_id: data)
+    return 'not_found' unless job
+
+    determine_job_status(job)
+  end
+
+  def status_class
+    case job_status
+    when 'succeeded', 'completed' then 'status-completed'
+    when 'failed', 'discarded' then 'status-failed'
+    when 'running' then 'status-running'
+    when 'queued', 'scheduled' then 'status-queued'
+    else 'status-unknown'
+    end
+  end
+
+  private
+
+  def determine_job_status(job)
+    return 'discarded' if job.discarded?
+    return 'succeeded' if job.succeeded?
+    return 'failed' if job.finished? && job.error.present?
+    return 'running' if job.running?
+    return 'scheduled' if scheduled?(job)
+
+    'queued'
+  end
+
+  def scheduled?(job)
+    job.scheduled_at.present? && job.scheduled_at > Time.current
+  end
+end