What is even worth doing?

[MatthewCaseres]

Trying to think through some work items that are interesting, a lot of work can be done but it isn't clear what should be done. In no order.

Lifelib

• make a reasonable contribution to lifelib
  • Basic structure of actuarial programs, introduction to caching, graph capture.
  • https://github.com/benfred/py-spy should allow for some analysis of seriatim vs. array based methods.
    • The issue is probably related to just function call overall. We save so many recursive calls with the array, how to communicate this.
  • finish up analysis of kahn's algorithm for optimizing memory. I don't think trying to publish a library or something like that is necessary, just a solid analysis.
  • Using autodiff with a decorator.

Maybe even make a PDF document. Keep it short and rigorous.

So the pitch is to contribute a couple of high quality educational notebooks to the lifelib projects so that people know some basic information about building their own simple modeling systems in Python.

Website

The fact that star counts have not been updating is pretty bad. We need to nuke the source code and make it as concise and maintainable as possible.

Simulating universal life data

This also has potential as a lifelib contribution. We need the simulated data to do reproducible research.

Benchmarks

The benchmarks are fine but I don't think there are any enhancements that need to be prioritized.

Chainladder

Raising issues and not fixing is in poor taste. Should probably come back around and solve the issues I raised.

[lewisfogden]

I'm working on an approach to vectorize access to multi-dimensional tables (e.g. with keys of age, duration, sex) - I could put together a problem statement for this and some examples?

It's key blocker for me in terms of using Python in production (most of our tables have 2 or 3 keys, included banded keys).

(non-vertorized tables work fine using pandas MultiIndexes, but vectorised Indexes only work correctly in pandas for a single index)

[lewisfogden]

Thanks, I'm aiming to spend a week or so trialling heavytables out and checking it can cope with all my use cases. I've got a Table library in heavylight at the moment but it's pandas series based and a bit iffy, so might merge the two (although heavytables is probably useful unbundled too).

Happy to join forces, I am already using heavylight (unfortunately don't know who else is) so I need to be careful about breaking changes (and put in warnings etc if they are).

Worth putting together a roadmap/idea list?

Alongside yours, I'd have

• pip/pypi install package
• user examples installable from package (included in pip)
• documentation / API written
• HOWTOs for new users
• on tests - excel models and heavylight equivalent models demonstrating same results - people trust Excel
• More of a package/suite approach - not a full blown UI
• Controlled way of running production jobs

[MatthewCaseres]

Here is the source code for the Class to inherit from https://github.com/actuarialopensource/heavylight/blob/main/src/heavylight/heavylight.py

You can see that the only public facing methods RunModel and ToDataFrame have not changed. Naming conventions for which methods are cached have not changed either.

Happy things first, scroll to the bottom to see a list of breaking changes, let me know if there are any deal breakers.

Previous TODOs are done + implementation details

• Implemented code coverage and automated testing
  • 100% code coverage (Still need to test more things, like t comes first in ToDataFrame for example)
• All of the caches can be cleared and the model can be rerun
  • This is necessary for memory optimization which uses a warm-up run
• Implemented memory saving mechanism which is showing over 99% reduction in cache size
• There is now a decorator and a class to inherit from. The class iterates through its methods and applies the decorator. In theory you can just use the decorator though, but not sure if I will document this or just leave it as an implementation detail.
• There is an intermediate dictionary representation of stored results which is easy to load into a DataFrame or perform transformations on
  • Can easily aggregate/round/select from cached NumPy arrays before loading them into a DataFrame.

Plans for the weekend

• Raise error if user runs twice without resetting the caches
• Implement and test a simpler non-optimizable cache to be used in situations where the user does not need additional performance
  • May be two classes, OptimizableModel and Model, using two different decorators.
• Build out the documentation site, may focus on documenting the non-optimizable cache first because it is simpler to explain.

I'm going to have to give an extra effort at work and school starting Monday so the plan is to have a PR on Sunday night and then you can think on it and give feedback. Unsure how far I will get.

The points on your roadmap are noted and I have no disagreements, but can't get them done by Sunday.

Notes on testing of models

Providing Excel reference implementations

I agree that reference implementations in Excel are good if we are building out some standard models lifelib style.

Grinding out coverage to 100%

My view of testing is really about being able to write tests in PyTest. Basically I am going to define __setitem__ on your _Cache'd method. And then I am going to write a unit test like

model.pols_if[1] = 1
model.pols_death[1] = .5
assert model.pols_if(2) == .5

This is just because there is a lot of interconnectedness in the methods and being able to inject values into the cache will allow us to isolate this. Specifically I am trying to ensure we are able to grind out code coverage to 100% on any models with unit tests. Obviously we can integration test, but as soon as we swap out a mortality table, the integration tests break. To unit test, we inject values in the cache.

This is implemented in fork - https://github.com/actuarialopensource/heavylight/blob/a7f6b702528da0e9583cb514dcfbbba3f9ed90c9/src/heavylight/cache_graph.py#L102

Breaking changes

• Removal of tables class
  • This code is not related to the caching of methods
• Removal of verbose
  • Seeing a lot more of this than I am used to seeing in other open source projects. I think you are using it to debug while developing but maybe using the IDE's debugger will be a suitable replacement?
• Not dynamically assigning object properties based on what the user passed in
  • The user will super init
• methods must have a single parameter and it must be named t for them to be called on RunModel.
  • I know you are not requiring t to be the param name, but if we are going to be iterating over timesteps and calling this method, I think it is reasonable to ask for

[lewisfogden]

Thank you - will have a look over. Main comment is that single parameter methods won't work for some of my production use cases, so maybe that needs to be optional (i.e. I've set it so if one parameter then it will loop it, maybe the constraint is it only loops it if it is labelled t?)

I'm tempted to say we should treat this as a community driven experimental model (or multiple models) and not necessarily heavylight v2. (Aim of heavylight is to give a simple way to generate projections, which IT security departments can check easily).

I've been working on testing of heavytables which I've merged into heavylight (and get rid of the original tables class as it wasn't great) - maybe we should keep these separate though? (Found one problem bug, with false positives on string lookup)

There are a lot of good ideas in cashflower too (https://github.com/acturtle/cashflower/tree/main by @zchmielewska et al.).

[MatthewCaseres]

I've set it so if one parameter then it will loop it, maybe the constraint is it only loops it if it is labelled t?

That is how it currently works in my codebase. Sorry if unclear.

I'm tempted to say we should treat this as a community driven experimental model (or multiple models) and not necessarily heavylight v2.

In this case I guess I would take my stuff, put it in it's own memory_optimized.py file, and then send that over?

Aim of heavylight is to give a simple way to generate projections, which IT security departments can check easily

Unsure what security concerns I've introduced?

I've been working on testing of heavytables which I've merged into heavylight (and get rid of the original tables class as it wasn't great) - maybe we should keep these separate though? (Found one problem bug, with false positives on string lookup)

Can you set up some automated testing? Or I guess if my PR goes through then you will have it. It usually catches a couple bugs and makes you feel safe when refactoring code.

my main takeaway

I was wanting to rewrite some of the existing code because of things that bother me like do_run or dynamic assignment of object properties. This is probably not needed as there isn't really anything wrong with it, it just doesn't follow a certain style that I like, and that is fine. no attempts to delete existing source code, additive changes only.

My main complaint about cashflower is that it is opinionated, describing should files be named, how should assumptions be accessed. But I do like their aggregation type. I implement it by letting a user define a function.

https://cashflower.acturtle.com/model.html#aggregation-type

[lewisfogden]

I'm a bit of a noob at writing testing (and github in general), and it's something I need to work on - I think I approved your PR already?

My main concern is you spend lots of time writing code and I have to say no to it, so probably just need to do a bit of planning. Like the idea of getting rid of do_run and maybe just defaulting proj_len to 0 to achieve the same thing.

Security is just about size of code - anything I bring into my org needs to be scanned and have due dilligence, so less code = easier to check for issues. Obviously heavylight is a bit easier as I've written it, but modelx/lifelib/cashflower are all big libraries with a lot of code I'd need to check.

I've got a branch with the integrated Tables in, am working on documentation (and found a flaw in my string lookup approach In [10] here):
https://github.com/lewisfogden/heavylight/blob/dev_heavytable_integration/examples/notebook/table_documentation.ipynb

[lewisfogden]

This chain is getting a little hard to follow from all the ideas. @MatthewCaseres - I've added you to the repo (and set up a project) - does that work for you? Let's get issues/ideas noted out as separate items.

[MatthewCaseres]

Sure.