Figure out high-volume analysis #39
Description
What
Improving pipeline to handle more files.
Why
The current version of the code can only process ~2k files of the 1.5 million we have on AWS. This restricts the scope of the analysis and interferes with the regression that is it's key component (it needs more Block Groups to use more variables).
How (optional)
Not sure yet. I'll update this as soon as I have a plan.
Deliverables
- Additional programs in/improvements to the pipeline that allow us to process 1k+ Census Block Groups.