It seems like osf.io has modest limits on how many calls we can make to the server--I frequently hit these limits when updating the data using osfclient, in which case the update/download fails when starting some file upload and I simply have to rerun and hope it works. osfclient users are are of this issue, but the package is no longer maintained, so using the cli is going to be fragile moving forward. Potential options to consider:
- use osfclient's python API (though I am still worried about issues here)
- use zip/tar on data (so that the number of file uploads/downloads is smaller)
- use osf download links only
- bypass osfclient API and directly use JSON API
It seems like osf.io has modest limits on how many calls we can make to the server--I frequently hit these limits when updating the data using osfclient, in which case the update/download fails when starting some file upload and I simply have to rerun and hope it works. osfclient users are are of this issue, but the package is no longer maintained, so using the cli is going to be fragile moving forward. Potential options to consider: