Skip to content

Ability to paginate through a dataset query in parallel #289

@davecap

Description

@davecap

Paginating through datasets can be slow especially for large datasets. Adding a parallel query capability through requests-futures or similar would be useful since you can opt-in to the behavior if you have the dependency installed.

The implementation would need to handle the following (the same way the current client code handles them):

  • Parsing the JSON response
  • Handling error responses
  • Handling 429 rate limits

Ideally it will be transparent to the end-user. They can just iterate through a query as usual but behind the scenes it will pull multiple pages in parallel, with a configurable number of "workers".

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions