Reduce memory usage #24
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
User-visible strings should be quoted with " and ' is for internal strings.
The max_itr is now more explicit and the convergence criteria are only checked in one place instead of in the while condition and after the loop. The loop now actually stops after doing max_itr iterations, where previously it would do iterations numbered 0 to max_itr, i.e. one iteration too many.
This will construct a large matrix that converges slowly. The matrix is 12000x12000 single precision numbers, in total 576 MB.
The self.original and self.aggregates are never modified but kept the same as the original user input. The `m` and `aggregates` variables are a copy of the user input and may be modified during the iterations. The `iteration` method is responsible for converting the user input to a numpy array of floats. The `ipfn_np` function only checks whether that condition holds but does no conversion. This allows the `ipfn_np` to be used directly for user that want absolute minimum memory usage without any extra copy of the input array. Instead of always converting to float64, the input arrays are now only checked if they are of floating point type. This means that a single precision float32 array will be kept that way, allowing for a smaller memory footprint, if the problem allows the lower precision.
…matrices This save some memory in case of more than two steps.
Since the previous commit we don't need an integer loop index `inc` anymore, so simply using zip over `aggregates` and `dimensions` is more clear.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a performance test script and some memory optimizations for
ipfn_np.On my test system 10 iterations for a 12,000x12,000 matrix go from 36 seconds to 21 s. The peak memory usage goes down from 6.2 GB to 1.8 GB.
It is split in multiple commits, so that each individual change is easy to review.
Changes are on top of the vortechbv/ipfn:cleanup branch, so it probably makes sense to merge #23 first.