Skip to content

Reduce memory consuption by doing in place calculations #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
hugobuddel wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Reduce memory consuption by doing in place calculations #310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81233aa
In place convolution to reduce memory consumption
hugobuddel Dec 16, 2023
ae69082
In place median to reduce memory consumption.
hugobuddel Dec 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions scopesim/effects/psf_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,13 @@ def get_bkg_level(obj, bg_w):
mask = np.zeros_like(obj, dtype=bool)
if bg_w > 0:
mask[:, bg_w:-bg_w, bg_w:-bg_w] = True
bkg_level = np.ma.median(np.ma.masked_array(obj, mask=mask),
axis=(2, 1)).data

# Using overwrite_input=True reduces memory consumption.
# Nevertheless, the computation of the median here can be
# responsible for more than 40% of the memory consumption.
bkg_masked_array = np.ma.masked_array(obj, mask=mask)
bkg_level_temp = np.ma.median(bkg_masked_array, axis=(2, 1), overwrite_input=True)
bkg_level = bkg_level_temp.data

else:
raise ValueError("Unsupported dimension:", obj.ndim)
Expand Down
12 changes: 10 additions & 2 deletions scopesim/effects/psfs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,12 @@ def apply_to(self, obj, **kwargs):
# kernel /= np.sum(kernel)
# kernel[kernel < 0.] = 0.

image = obj.hdu.data.astype(float)
# This line copied the entire image, while below obj.hdu.data
# is updated anyway. So better do all modifications in place.
#image = obj.hdu.data.astype(float)
if obj.hdu.data.dtype != float:
obj.hdu.data = obj.hdu.data.astype(float)
image = obj.hdu.data

# subtract background level before convolving, re-add afterwards
bkg_level = pu.get_bkg_level(image, self.meta["bkg_width"])
Expand All @@ -104,7 +109,10 @@ def apply_to(self, obj, **kwargs):
image[iplane,] - bkg_level[iplane,],
kernel[iplane,], mode=mode)

obj.hdu.data = new_image + bkg_level
# This line also copied all the data, better to do the
# modifications in place.
#obj.hdu.data = new_image + bkg_level
obj.hdu.data -= bkg_level
Comment on lines +114 to +115
Copy link
Contributor

@teutoburg teutoburg Dec 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This used to be an addition and is now a subtraction? Also, what happened to new_image?

Copy link
Collaborator Author

@hugobuddel hugobuddel Dec 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh sorry for wasting your time, I didn't run the test suite :-(

Copy link
Contributor

@teutoburg teutoburg Dec 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happens to me as well sometimes 🙃


# ..todo: careful with which dimensions mean what
d_x = new_image.shape[-1] - image.shape[-1]
Expand Down