From 19848b5d75abe756583b1d6e9eea67d83f84ca29 Mon Sep 17 00:00:00 2001 From: gregcorbett Date: Mon, 3 Nov 2025 14:51:57 +0000 Subject: [PATCH 1/3] Add paging documentation - original text sourced from the EGI wiki, https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API - modified slightly when reproduced here and through code review Co-authored-by: rowan04 --- api/read/optional_cursor_paging.md | 42 ++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 api/read/optional_cursor_paging.md diff --git a/api/read/optional_cursor_paging.md b/api/read/optional_cursor_paging.md new file mode 100644 index 0000000..19a93d9 --- /dev/null +++ b/api/read/optional_cursor_paging.md @@ -0,0 +1,42 @@ +# GOCDB Programmatic Interface - Optional Cursor Paging + +Cursor paging allows clients to iterate through massive result-sets which are +split into one or more smaller pages. This allows result-sets to be processed +that would otherwise be too large or memory-expensive to process in a single +query. + +Paging may not always be necessary, we suggest you first try to execute your +query without paging. If the query fails due to breaching of the execution +time or memory limit, then try adding `next_cursor=0`. + +Most of the API methods now support optional `next_cursor` and `prev_cursor` +URL parameters. + +For backward compatibility, pagination is disabled by default and will only be +enabled when passing one of the new cursor query parameters. + +If neither the `next_cursor` or `prev_cursor` URL parameters are specified, the +method behaves as before by fetching/rendering the requested result-set in one +query, and will not add the `meta` element as detailed below. + +A non-paged query can therefore fail/timeout if either the execution time or +memory limits are exceeded (e.g. if you make a request for every downtime +without narrowing the search with extra filter parameters). + +Paged XML results contain the `meta` element, which contains links to the +current/self, next, previous and first page of results. + +When using paging, you need to move forward through the results by reading the +next links until the `count` element hits zero which indicates no more +results are available at that particular moment in time. + +To start at the beginning of a result-set, specify `next_cursor=0`. + +The max page size has been initially set to 100 but this is subject to change/ +refinement. NB: the max page size value is provided in the `max_page_size` +element. + +The `count` element shows how many results are on the current page (note, if +this figure is less than `max_page_size` then you have come to the end of +the results, and at that moment in time, the next page will show no results +with a count of zero). From 1072118839dd7a134cbcbd4722608643d94f4afe Mon Sep 17 00:00:00 2001 From: gregcorbett Date: Mon, 3 Nov 2025 17:14:28 +0000 Subject: [PATCH 2/3] Update docs to point at new cursor documentation. --- api/read/get_cert_status_changes/index.md | 4 ++-- api/read/get_cert_status_date/index.md | 4 ++-- api/read/get_downtime/index.md | 4 ++-- api/read/get_downtime_nested_services/index.md | 4 ++-- api/read/get_downtime_to_broadcast/index.md | 4 ++-- api/read/get_ngi/index.md | 4 ++-- api/read/get_project_contacts/index.md | 4 ++-- api/read/get_roc_contacts/index.md | 4 ++-- api/read/get_service/index.md | 8 ++------ api/read/get_service_group/index.md | 4 ++-- api/read/get_service_group_role/index.md | 4 ++-- api/read/get_site/index.md | 4 ++-- api/read/get_site_contacts/index.md | 4 ++-- api/read/get_user/index.md | 4 ++-- 14 files changed, 28 insertions(+), 32 deletions(-) diff --git a/api/read/get_cert_status_changes/index.md b/api/read/get_cert_status_changes/index.md index d1e04c6..3b50cd6 100644 --- a/api/read/get_cert_status_changes/index.md +++ b/api/read/get_cert_status_changes/index.md @@ -20,8 +20,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_cert_status_date/index.md b/api/read/get_cert_status_date/index.md index 8878444..a9b8bfa 100644 --- a/api/read/get_cert_status_date/index.md +++ b/api/read/get_cert_status_date/index.md @@ -19,8 +19,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_downtime/index.md b/api/read/get_downtime/index.md index ad0e99f..c19c926 100644 --- a/api/read/get_downtime/index.md +++ b/api/read/get_downtime/index.md @@ -26,8 +26,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_downtime_nested_services/index.md b/api/read/get_downtime_nested_services/index.md index 3987761..900e7e5 100644 --- a/api/read/get_downtime_nested_services/index.md +++ b/api/read/get_downtime_nested_services/index.md @@ -26,8 +26,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_downtime_to_broadcast/index.md b/api/read/get_downtime_to_broadcast/index.md index 654794c..3a33981 100644 --- a/api/read/get_downtime_to_broadcast/index.md +++ b/api/read/get_downtime_to_broadcast/index.md @@ -20,8 +20,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_ngi/index.md b/api/read/get_ngi/index.md index 4fbf99e..579ffb6 100644 --- a/api/read/get_ngi/index.md +++ b/api/read/get_ngi/index.md @@ -20,8 +20,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_project_contacts/index.md b/api/read/get_project_contacts/index.md index 00ee873..e53ffb6 100644 --- a/api/read/get_project_contacts/index.md +++ b/api/read/get_project_contacts/index.md @@ -18,8 +18,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_roc_contacts/index.md b/api/read/get_roc_contacts/index.md index 349b936..1bca024 100644 --- a/api/read/get_roc_contacts/index.md +++ b/api/read/get_roc_contacts/index.md @@ -18,8 +18,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_service/index.md b/api/read/get_service/index.md index bf2ec1c..a8dc6a9 100644 --- a/api/read/get_service/index.md +++ b/api/read/get_service/index.md @@ -27,12 +27,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). - -or - -This method does not support paging. +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_service_group/index.md b/api/read/get_service_group/index.md index 8a40a57..d243490 100644 --- a/api/read/get_service_group/index.md +++ b/api/read/get_service_group/index.md @@ -21,8 +21,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_service_group_role/index.md b/api/read/get_service_group_role/index.md index fe396d4..1cf237a 100644 --- a/api/read/get_service_group_role/index.md +++ b/api/read/get_service_group_role/index.md @@ -20,8 +20,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_site/index.md b/api/read/get_site/index.md index 05a5f75..03ba618 100644 --- a/api/read/get_site/index.md +++ b/api/read/get_site/index.md @@ -26,8 +26,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_site_contacts/index.md b/api/read/get_site_contacts/index.md index 74e113b..80a8f8c 100644 --- a/api/read/get_site_contacts/index.md +++ b/api/read/get_site_contacts/index.md @@ -23,8 +23,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output diff --git a/api/read/get_user/index.md b/api/read/get_user/index.md index 044b5db..743a5c1 100644 --- a/api/read/get_user/index.md +++ b/api/read/get_user/index.md @@ -22,8 +22,8 @@ ### Paging -This method optionally supports paging, to iterate over all results, -[see here for info](https://wiki.egi.eu/wiki/GOCDB/notifications#Optional_Cursor_Paging_on_Read_API). +This method optionally supports paging to iterate over all results, +[see here for info](../optional_cursor_paging.md). ### Example Output From 42f76c6ff2071d436a4bf4dd60c7ea1bd134aba2 Mon Sep 17 00:00:00 2001 From: gregcorbett Date: Tue, 4 Nov 2025 13:08:55 +0000 Subject: [PATCH 3/3] Add summary of paging parameters and meta tags Co-authored-by: rowan04 --- api/read/optional_cursor_paging.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/api/read/optional_cursor_paging.md b/api/read/optional_cursor_paging.md index 19a93d9..0e4d171 100644 --- a/api/read/optional_cursor_paging.md +++ b/api/read/optional_cursor_paging.md @@ -1,5 +1,7 @@ # GOCDB Programmatic Interface - Optional Cursor Paging +## Overview + Cursor paging allows clients to iterate through massive result-sets which are split into one or more smaller pages. This allows result-sets to be processed that would otherwise be too large or memory-expensive to process in a single @@ -40,3 +42,23 @@ The `count` element shows how many results are on the current page (note, if this figure is less than `max_page_size` then you have come to the end of the results, and at that moment in time, the next page will show no results with a count of zero). + +## Summary + +### Paging parameters explained + +| parameter | definition | +| - | - | +| `next_cursor` | A cursor that points to the start of the next page of results. | +| `prev_cursor` | A cursor that points to the start of the previous page of results. | + +### `meta` tags explained + +| tag | definition | +| - | - | +| `self` | The link to the current page of results. | +| `next` | The link to the start of the next page of results. | +| `previous` | The link to the start of the previous page of results. | +| `start` | The link to the start of the first page of results. | +| `count` | Shows how many results are on the current page (note, if this figure is less than `max_page_size` then you have come to the end of the results, and at that moment in time, the next page will show no results with a count of zero). | +| `max_page_size` | The maximum number of results shown on a page. |