Skip to content

Conversation

@Mayureshpawar29
Copy link
Contributor

@Mayureshpawar29 Mayureshpawar29 commented Dec 2, 2025

Description

  • Added support for JSON object configurations in the nextPage field for pagination. Previously, nextPage only supported string URLs. Now it can also accept a JSON object with endpoint, method, body, and headers fields.

  • Many APIs require POST requests with dynamic request bodies for pagination.
    The old string-only approach couldn't handle these cases.
    Before (only strings worked):
    nextPage: '.data | "https://api.example.com/page2"'
    After (strings AND objects work):

# Still works - strings
nextPage: '.data | "https://api.example.com/page2"'

# NEW - objects for complex pagination
nextPage: |
  .data | fromjson | {
    endpoint: "https://api.example.com/graphql",
    method: "POST",
    body: ({query: "...", variables: {after: .cursor}} | tojson),
    headers: {"Authorization": "Bearer token"}
  }

  • Fully backwards compatible - existing string-based configs still work

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation and I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • I have checked downstream dependencies (e.g. ExternalTaskSensors) by searching for DAG name elsewhere in the repo

@Mayureshpawar29 Mayureshpawar29 changed the title Handled next page case if object is passed in the next_page instead of the url Feat: Next page to accept object Dec 3, 2025
@Mayureshpawar29 Mayureshpawar29 marked this pull request as ready for review December 3, 2025 06:43
prasadlohakpure
prasadlohakpure previously approved these changes Dec 3, 2025
}
}
}
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be safe - if something other than a string or object is passed to nextPage (like a boolean, null, number, array, or anything else), we break the pagination loop instead of continuing with invalid data.

hladush
hladush previously approved these changes Dec 3, 2025
Copy link
Contributor

@hladush hladush left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like feature

@Mayureshpawar29 Mayureshpawar29 merged commit 6a2b606 into main Dec 4, 2025
6 checks passed
@Mayureshpawar29 Mayureshpawar29 deleted the http-next-pag-fix branch December 4, 2025 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants