Skip to content

Parallelization for SWIFTGalaxies #19

@kyleaoman

Description

@kyleaoman

There's an obvious use-case for tools to iterate over SWIFTGalaxy objects in parallel. This was omitted from the initial SWIFTGalaxies PR for simplicity but implementing some parallelization tools on top of SWIFTGalaxies should be high on the to-do list. Some initial thoughts carried over from the SWIFTGalaxies development process:

  • SWIFTGalaxies.split(n) to return a list of n SWIFTGalaxies instances each with a fraction approx 1/n of the ensemble of target regions (where each region could contain many target galaxies). This can easily be farmed out to multiple processes (maybe a SWIFTGalaxies.sub(n, N) approach makes more sense so that each process gets a copy-on-write instance and then picks out its own share?). Some cookbook examples useful here.
  • Could combine map with split (or sub) to offer some parallelization built in.
  • Consider analogous tools to other tools from the multiprocessing in addition to map. Things like some kind of queue, or asynchronous access to results as they arrive.
  • Beware nested parallelism.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions