Paginate superuser users/projects lists with server-side search by sakibsadmanshajib · Pull Request #4474 · OpenFn/lightning

sakibsadmanshajib · 2026-02-26T16:43:58Z

Description

This PR adds server-side pagination, sorting, and search to the superuser users/projects settings list views, and moves those list operations out of in-memory filtering into SQL-backed queries.

Closes #2913

Validation steps

mix format
mix test test/lightning/accounts_test.exs test/lightning/projects_test.exs test/lightning_web/live/user_live_test.exs test/lightning_web/live/project_live_test.exs --seed 0
mix test test/lightning_web/live/audit_live_test.exs --seed 0

Additional notes for the reviewer

Added trigram indexes for users/projects search paths in migration 20260226162000_add_admin_search_trgm_indexes.exs.
LiveView list state is now URL-param driven (page, filter, sort, dir) for both users and projects.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

I have used Claude Code
I have used another model
I have not used AI

Pre-submission checklist

I have performed an AI review of my code (we recommend using /review with Claude Code)
I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
I have updated the changelog.
I have ticked a box in "AI usage" in this PR

Copilot

Pull request overview

This PR improves superuser admin list performance by moving /settings/users and /settings/projects filtering/sorting/pagination from in-memory operations to database-backed queries with URL-param-driven state, and adds trigram indexes to support efficient search.

Changes:

Added Accounts.list_users_for_admin/1 and Projects.list_projects_for_admin/1 using Repo.paginate/2 with normalized params for safe filtering/sorting.
Updated Users/Projects LiveViews to treat URL params (page, filter, sort, dir, page_size) as the source of truth and to use Scrivener pages in table rendering.
Added pg_trgm + GIN trigram indexes for key search columns, plus tests covering pagination and param safety.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/lightning_web/live/user_live_test.exs	Adds LiveView pagination navigation coverage for users index.
test/lightning_web/live/project_live_test.exs	Adds LiveView pagination navigation coverage for projects index.
test/lightning/projects_test.exs	Adds coverage for admin projects search/sort and invalid param fallback.
test/lightning/accounts_test.exs	Adds coverage for admin users search/sort and invalid param fallback.
priv/repo/migrations/20260226162000_add_admin_search_trgm_indexes.exs	Adds `pg_trgm` extension and trigram indexes for admin search paths.
lib/lightning_web/live/user_live/table_component.ex	Switches users table component to Scrivener-page-driven rendering + URL patching.
lib/lightning_web/live/user_live/index.html.heex	Passes normalized `table_params` down to the table component.
lib/lightning_web/live/user_live/index.ex	Normalizes/validates table URL params (sort/dir/page/page_size/filter).
lib/lightning_web/live/user_live/components.ex	Updates users table component API to accept `page` + `pagination_path`.
lib/lightning_web/live/project_live/index.html.heex	Switches projects table to use Scrivener page entries + pagination URL.
lib/lightning_web/live/project_live/index.ex	Adds param normalization and uses `Projects.list_projects_for_admin/1` for server-side table data.
lib/lightning/projects.ex	Implements paginated/sortable/searchable admin projects query.
lib/lightning/accounts.ex	Implements paginated/sortable/searchable admin users query.
docs/plans/2026-02-26-superuser-users-projects-pagination-search-design.md	Documents design/approach and rollout considerations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/lightning_web/live/user_live/table_component.ex

lib/lightning/accounts.ex

midigofrank

Hey @sakibsadmanshajib , great job here. The superuser interface needs some love, thank you for picking this up.

I have left some comments:

I don't see a need for indexes on those tables. They're usually tiny
Consider using using an embedded schema to have a single place for validations, like: https://github.com/OpenFn/lightning/blob/main/lib/lightning/workorders/search_params.ex
With the embedded schema in place, you can try to refactor the list_users function, like: https://github.com/OpenFn/lightning/blob/main/lib/lightning/invocation.ex#L482

priv/repo/migrations/20260226162000_add_admin_search_trgm_indexes.exs

lib/lightning_web/live/user_live/index.ex

lib/lightning/accounts.ex

queries/live views refactored to use them

midigofrank

Hey @sakibsadmanshajib , sorry I took so long to get back to you. You're on the right track. I have left inline comments, but here is the tldr:

Use standard field names, like those in https://github.com/OpenFn/lightning/blob/main/lib/lightning/workorders/search_params.ex . Use search_term instead of filter, sort --> sort_by, dir --> sort_direction
Boolean state fields like role, enabled, and support_user don't need to be in the sort list, these are better handled as toggles in the form. We can track that work in a separate follow-up issue.
Ecto changesets can handle all the normalizations you need, so the custom normalize_value function isn't necessary.
Context functions should receive the struct directly rather than raw map params
I've suggested some renaming — admin conflicts with the ProjectUser role, so we'll want to resolve that

Please go through the comments and apply the same changes to the projects context. Note that I haven't reviewed the LiveViews yet, II'll do that in the next round once the contexts are updated. In the meantime, it's worth looking at how the workorders page uses the filter changeset (https://github.com/OpenFn/lightning/tree/main/lib/lightning_web/live/run_live) and applying the same pattern to these pages.

Thanks again for picking this up, don't hesitate to reach out if anything is unclear.

midigofrank · 2026-03-18T05:32:56Z

lib/lightning/accounts/admin_search_params.ex

You don't need to stringify and normalize while using ecto changesets. The changesets will do this for you, look at https://github.com/OpenFn/lightning/blob/main/lib/lightning/workorders/search_params.ex. You can read more about Ecto changesets in https://hexdocs.pm/ecto/Ecto.Changeset.html

Also, could you use the same field names as the workorders search params? The filter --> search_term, sort --> sort_by, dir --> sort_direction

midigofrank · 2026-03-18T05:33:30Z

lib/lightning/projects/admin_search_params.ex

You don't need to normalize these values while using Ecto Changeset

midigofrank · 2026-03-18T05:38:48Z

lib/lightning/accounts.ex

+  server-side filtering and sorting.
+  """
+  @spec list_users_for_admin(map()) :: Scrivener.Page.t()
+  def list_users_for_admin(params \\ %{}) do


Suggested change

def list_users_for_admin(params \\ %{}) do

def list_all_users(%AdminSearchParams{} = params) do

The context function should take the typed params instead of building it inside the body. This way we establish a "contract" with the callers.

We should also rename the function, to list_all_users or list_users_for_superuser. admin conflicts with the ProjectUser role

midigofrank · 2026-03-18T06:16:10Z

lib/lightning/accounts.ex

+    do: order_by(query, [u], asc_nulls_last: u.scheduled_deletion)
+
+  defp order_admin_users(query, "scheduled_deletion", "desc"),
+    do: order_by(query, [u], desc_nulls_first: u.scheduled_deletion)


Suggested change

do: order_by(query, [u], desc_nulls_first: u.scheduled_deletion)

do: order_by(query, [u], desc_nulls_last: u.scheduled_deletion)

Given null are non deleted users, we should show them last

midigofrank · 2026-03-18T06:21:43Z

lib/lightning/accounts.ex

+    )
+  end
+
+  defp order_admin_users(query, "enabled", "asc"),


I don't think we need to sort by a boolean. This should be toggle somewhere on the form. Let's skip it for now

midigofrank · 2026-03-18T06:23:47Z

lib/lightning/accounts.ex

+
+    User
+    |> filter_admin_users(params.filter)
+    |> order_admin_users(params.sort, params.dir)


Suggested change

|> order_admin_users(params.sort, params.dir)

|> sort_users_query(params.sort, params.dir)

midigofrank · 2026-03-18T06:24:55Z

lib/lightning/accounts.ex


+  defp filter_admin_users(query, ""), do: query
+
+  defp filter_admin_users(query, filter) do


Suggested change

defp filter_admin_users(query, filter) do

defp search_users_query(query, search_term) do

midigofrank · 2026-03-18T06:26:48Z

lib/lightning/accounts.ex

+  defp order_admin_users(query, "scheduled_deletion", "desc"),
+    do: order_by(query, [u], desc_nulls_first: u.scheduled_deletion)
+
+  defp order_admin_users(query, "role", dir) do


I also don't see the need to sort by role. There are only 2 types of users, :user and :superuser. This should be a toggle

midigofrank · 2026-03-18T06:29:18Z

lib/lightning/accounts/admin_search_params.ex

+  @primary_key false
+
+  @default_sort "email"
+  @allowed_sorts ~w(first_name last_name email role enabled support_user scheduled_deletion)


Suggested change

@allowed_sorts ~w(first_name last_name email role enabled support_user scheduled_deletion)

@allowed_sorts ~w(first_name last_name email scheduled_deletion)

midigofrank · 2026-03-18T06:39:34Z

lib/lightning/accounts.ex

+  defp order_admin_users(query, "enabled", "desc"),
+    do: order_by(query, [u], asc: u.disabled)
+
+  defp order_admin_users(query, "scheduled_deletion", "asc"),


Suggested change

defp order_admin_users(query, "scheduled_deletion", "asc"),

defp sort_users_query(query, "scheduled_deletion", "asc"),

Copilot AI review requested due to automatic review settings February 26, 2026 16:43

github-project-automation bot added this to Core Feb 26, 2026

github-project-automation bot moved this to New Issues in Core Feb 26, 2026

Copilot started reviewing on behalf of sakibsadmanshajib February 26, 2026 16:44 View session

sakibsadmanshajib added 8 commits February 26, 2026 11:48

feat: add paginated admin users query

17d082f

feat: add paginated admin projects query

0679a9d

perf: add trigram indexes for admin users/projects search

4c85c38

feat: server-side pagination and search for admin users list

fb9006d

feat: server-side pagination and search for admin projects list

4681703

test: cover admin list param fallback and safety

7990c77

chore: format admin pagination changes

1ee5649

docs: update changelog for superuser list pagination

fcf4b25

sakibsadmanshajib force-pushed the paginate-superuser-list branch from a7f7859 to fcf4b25 Compare February 26, 2026 16:48

Copilot AI reviewed Feb 26, 2026

View reviewed changes

lib/lightning_web/live/user_live/table_component.ex Outdated Show resolved Hide resolved

lib/lightning/accounts.ex Show resolved Hide resolved

fix: address PR review feedback on admin list queries

67ca8db

midigofrank requested changes Mar 3, 2026

View reviewed changes

priv/repo/migrations/20260226162000_add_admin_search_trgm_indexes.exs Outdated Show resolved Hide resolved

lib/lightning_web/live/user_live/index.ex Show resolved Hide resolved

lib/lightning/accounts.ex Show resolved Hide resolved

github-project-automation bot moved this from New Issues to In review in Core Mar 3, 2026

sakibsadmanshajib added 3 commits March 7, 2026 13:33

index migration removed, embedded schemas added, and the admin list

5ab22f0

queries/live views refactored to use them

add new schema

a560814

fix credo violations

d84db8b

sakibsadmanshajib requested a review from midigofrank March 13, 2026 18:56

midigofrank requested changes Mar 18, 2026

View reviewed changes

	def list_users_for_admin(params \\ %{}) do
	def list_all_users(%AdminSearchParams{} = params) do

	do: order_by(query, [u], desc_nulls_first: u.scheduled_deletion)
	do: order_by(query, [u], desc_nulls_last: u.scheduled_deletion)

	\|> order_admin_users(params.sort, params.dir)
	\|> sort_users_query(params.sort, params.dir)


		defp filter_admin_users(query, ""), do: query

		defp filter_admin_users(query, filter) do

	defp filter_admin_users(query, filter) do
	defp search_users_query(query, search_term) do

	@allowed_sorts ~w(first_name last_name email role enabled support_user scheduled_deletion)
	@allowed_sorts ~w(first_name last_name email scheduled_deletion)

	defp order_admin_users(query, "scheduled_deletion", "asc"),
	defp sort_users_query(query, "scheduled_deletion", "asc"),

Conversation

sakibsadmanshajib commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation steps

Additional notes for the reviewer

AI Usage

Pre-submission checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

midigofrank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

midigofrank left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sakibsadmanshajib commented Feb 26, 2026 •

edited

Loading