Runs hang at reform simulation

[rickecon]

Running the example script and other simulations of mine, I am able to run the baseline SS and TPI successfully, but the simulations hang up in the reform simulation after the tax function estimation, after making the reform SS and TPI directories and after stating "Using previous solutions for SS". I think this is probably a memory issue. In other runs, I think we have seen that the memory footprint passed to each node doubles in size for the reforms. We need to fix this. In the meantime, we might need to roll back the version.

cc: @jdebacker

[jdebacker]

@rickecon What version of OG-Core are you using for the runs that hang?

Do we think this is related to the USA calibration or something more fundamental in OG-Core?

[rickecon]

@jdebacker. Good question. The table below shows the run statuses and package version numbers. In the OG-Core case, I built the package in the environment from my local branch. I don't remember if I did that for the ogusa environment. I just ran the OG-Core example file run_ogcore_example.py and the whole thing (baseline and reform) ran in about 45 minutes.

Run Script	Model	ogcore Version	ogusa Version	Run Status
run_ogcore_example.py	OG-Core	0.14.10		success
run_ogusa.py	OG-USA	0.14.10	0.3.1	failure

[jdebacker]

@rickecon I can confirm the same hanging behavior. If I let it go > 14 hours I noticed it did eventually start on the reform SS solution, but it was only through 3 iterations at that time.

I have tried different configurations to see if it's the parametrization or what. All of this was with the OG-USA master and OG-Core 0.14.10:

Change to run_ogusa.py	Run Status
Set reform_use_baseline_solution to False (default is True)	Hangs at reform SS
Do not change parameters from p	Hangs at reform SS
Do not change parameters from p and set reform_use_baseline_solution to False	Hangs at reform SS
Try closing and deleting client and instantiating a new client before reform solution	Hangs at reform SS

|

A few notes:

• I think the main difference in the OG-Core vs OG-USA example scripts performance is that OG-Core uses J=7 and OG-USA uses J=10. I would expect about a 100% slow down, but it appears to be more than that. Memory overhead? Not sure.
• There does seem to be something off with how OG-Core performs when baseline=False. It invariably seems much slower than the baseline, even when the same parameters are used in the baseline and reform (but the slowness increases considerably with J=10 as compared to J=7).