Skip to content

Make mosaicking faster/memory efficient through healpix grid + Make it compatible to reduce truncated data#41

Open
AkashA98 wants to merge 3 commits intoparsotat:develfrom
AkashA98:skyproject
Open

Make mosaicking faster/memory efficient through healpix grid + Make it compatible to reduce truncated data#41
AkashA98 wants to merge 3 commits intoparsotat:develfrom
AkashA98:skyproject

Conversation

@AkashA98
Copy link

There are two main changes that are addressed by this PR:

  1. By default, BatAnalysis does sky project onto a global WCS grid and then does mosaicking, but this consumes a lot of memory, storage space, and compute time. So switch the mosaicking step to use global healpix grids, so that it becomes both memory/storage and time efficient. For comparison, a single mosaic that previously used to take 10+min of compute time and 10 GB of storage space now takes ~3 min to run and 1 GB of storage.
  2. Previously, BatAnalysis (and batsurvey) used to skip truncated data sets. This PR addresses this by making batsurvey routines independent and customizable (can also be used for regular data if the user wants more freedom).

@parsotat
Copy link
Owner

@AkashA98 while I am reviewing these improvements, can you change the PR such that the changes be merged into the devel branch instead of main?

@AkashA98 AkashA98 changed the base branch from main to devel March 12, 2026 19:05
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the purpose of this file? Is it used in the code that is included in the PR?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is the old version of the code, this whole directory can be removed.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do it, if you can confirm that this is just the old version of the code

Returns:
"""

heatools.ftcoco(
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like BatAnalysis to use astropy tools as much as possible and only use heasoftpy where it is actually needed.

}


def convert_met_to_utc(times, invert=False):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function doesnt include the effect of the clock offset, which is why the batlib functions that deal with MET/UTC end up calling the swiftbat package since that is taken care of there.

else:
self.all_logs["batcelldetect"].append(batcelldetect_log)

def ftcopy(self, infile: str, outfile: str):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is another example of a ftools script that we can try to convert to pure python to reduce dependencies on heasoftpy

return new_tab


def post_process_catalogs(files, outfile=None, mode="merge"):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be converted to be astropy based

self.__dict__.update(content)
print("Load successful.")

def _save(self, file):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want this to be _save? or save? ie should it be private or not? In line with BatSurvey it seems like it should be public

)


class MosaicProjections:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this inherit from reprojectSurveys class? I have the MosaicBatSurvey class which inherits from the BatSurvey class since they share many properties (except for one being a mosaic) so I would naively expect the same here

shutil.rmtree(self._local_pfile_dir)
return out

def load_default_params(self):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default parameters can be obtained eg:

input_dict = batsurvey_gti.default_params.copy()

recalc=False,
persistent_source_sep=10 * u.arcmin,
snr=5,
plot_dest_dir=None,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to keep with the structure of the rest of the code, plotting shouldn't be something that gets set in the __init__ method there should be separate methods that get called to explicitly do the plotting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants