Cache data locally after download

[vincentarelbundock]

Allow users to pass a flag that will cache the data locally after it is downloaded. Goals:

1- Reproducibility. Even if the online version disappears, the user will still have a local copy
2- Quicker download on second request.

Should be accompanied by a standalone cleaning script if applicable.

[leeper]

This is a great idea. Are you thinking an R.cache dependency or
something else?

Thomas J. Leeper
http://www.thomasleeper.com

On Wed, Mar 5, 2014 at 6:29 AM, Vincent Arel-Bundock <
notifications@github.com> wrote:

Allow users to pass a flag that will cache the data locally after it is
downloaded. Goals:

1- Reproducibility. Even if the online version disappears, the user will
still have a local copy
2- Quicker download on second request.

Should be accompanied by a standalone cleaning script if applicable.

Reply to this email directly or view it on GitHubhttps://github.com//issues/8
.

[christophergandrud]

+1

[vincentarelbundock]

No need for a dependency. Just let user specify a local path where to save stuff.

[antagomir]

Note that CRAN has reservations for writing on local files: "Packages should not write in the users’ home filespace, nor anywhere else on the file system apart from the R session’s temporary directory (or during installation in the location pointed to by TMPDIR: and such usage should be cleaned up). Installing into the system’s R installation (e.g., scripts to its bin directory) is not allowed." http://cran.r-project.org/web/packages/policies.html

But perhaps this is not a problem, if the package does not do this by default, but only after users active choice. Of course there should be an opportunity for packages that are designed for data access. But need to be careful how to implement that. My suggestion is to cache to local tmpdir.

[christophergandrud]

Just added a file argument to get_data. It allows the user to specify an RData file to save the data frame into. I'll push it up in a few minutes.

(I don't think this is a problem for CRAN as I have a number of packages that do this, the default is always set to not write the file.)

I do like the cache idea though. This allows the user to not have to think about their file paths, which can differ when they run the function on different computers, for example.

[vincentarelbundock]

Cool. I think there should also be a way to replicate the file processing (e.g. create unique country names, drop duplicate country years, etc.). So we'd need to save a R script as well.

[christophergandrud]

@vincentarelbundock currently it saves the downloaded and processed data frame as the last step before return(data) at the end of the function. Does this make sense or should we try something else?

[vincentarelbundock]

That's cool, but in my view it's not enough. If you're given only the raw and the processed data, you still might have to reverse engineer to find out what the transformations were. Was that log(variable) or log(variable+1)? Which duplicate countries did we drop, exactly?

I think what makes the most sense would be related to my issue here: #10

You would have 2 files:

• database_political_institutions.yaml (download url, bibtex cite, etc.)
• database_political_institutions.R (cleaning script with all transformations)

And a standardized function:

• get_data(): Parse .yaml file, download data if not already cached, and run R script.
• If flagged for caching, then copy yaml, raw data, processed data and R script to specified path.

[antagomir]

I agree that in principle it would be useful to have both raw data and the full script available in a reproducible format. If this gets too messy, however, it would be useful to at least store the package version and function name. This is already rather useful as such, and virtually sufficient for full reproducibility.

[vincentarelbundock]

I think your current setup is going to be much messier in the long run. "Silos" for each datasets are the only way I can think of to cleanly separate the cleaning scripts. That's a real simple way to scale the project...

[christophergandrud]

The function calls + package version is effectively a recipe. So for example, having an R script that went:

## Using psData 1.0
library(psData)

# Download data
Main <- get_dpi(vars = 'checks', sha1 = 'foo', file = 'Checks.rda')

# Set as psData
MainP <- as.panel(Main, 'Country', 'Year')

# Transform
AS OF YET NOT CREATED TRANSFORMATION FUNCTIONS

DROP OBSERVATIONS, ETC

It would definitely be helpful to add the package version and call to the metadata in as.panel (we might be able to do it a step earlier at get_dpi). Maybe it's possible to also output a log of the whole process. But I think fundamentally, this is reproducible silos and recipes. Am I missing something?

Note also, something that hasn't been started yet: @briatte did a good job of creating a way of running basic data sets through get_data to get a basic data frame. But we also need to have a system for data that requires more manipulation to get it into the panel-series data frame format. These were the 'variable builders' in the original version of psData. These would also effectively be recipes, if I understand you correctly.

Thoughts?

[vincentarelbundock]

Here's a repo I quickly put together as a proof of concept to show what I mean more concretely.

https://github.com/vincentarelbundock/dpi

I think you may be right. I guess I'm just suggesting that a slightly different repo structure with explicit silos would be cleaner, more scalable, and much easier to contribute to.

In my experience, generic functions called things like "get_data" tend to try to do too much, and end up being unwieldy. Better to provide some basic functionalities, and cleanly delineate which transforms were operated on each dataset.

Even if there is some code duplication, it is better for transparency to have a file called "database_political_institutions.R" that is more or less self-contained and can show the user exactly what is done in the pre-processing stage.