More details for rclone setup

[eRJe79]

Hi,

I've been playing around with your script and trying to set it up now with OneDrive from Microsoft.

I'm failing to understand the setup in the .env. Could you give some more pointers?

[peterrus]

Sure! What can I help you with exactly?

If we take this example: https://github.com/peterrus/restic-wrapper/blob/master/configs/example-job-with-rclone.env
And have a look at this part of the rclone documentation: https://rclone.org/onedrive/#standard-options

The values in the .env that are prefixed with RCLONE_CONFIG are just read by rclone directly and technically have 'nothing to do' with restic_wrapper). I have named the backend OFFSITE so rclone will know about a backend called offsite. Small sidenote: The env var names you find in the linked rclone documentation are used by another mechanism to configure rclone. Rclone apparently has two schemes. For restic_wrapper I am using this scheme: https://rclone.org/docs/#config-file.

You can see a bunch of options for the OneDrive backend like client_id and region. In the .env linked above they should map to export RCLONE_CONFIG_OFFSITE_CLIENT_ID and export RCLONE_CONFIG_OFFSITE_REGION.

Once you have set all the required options for the OneDrive backend you can test if everything is working by sourcing the env file and running rclone manually.

Testing if the rclone backend is working can be done by (from a bash/zsh shell at least):

source ./your-job-that-uses-onedrive.env
rclone lsd offsite:/

You should then get a directory listing off whatever is on your remote OneDrive.

Let me know if anything else is not clear!

[eRJe79]

Thanks for the prompt response!

It's getting a little more clear now that you mentioned that there are two scheme's. Looking at the standard options that you use, I noticed that only RCLONE_ONEDRIVE_REGION is not tagged with Required: false. I think all these options were optional in rclone interactive configuration setup.

That made me realise that the authentication to the OneDrive is already configured within rclone and stored in .config/rclone.conf. I kept thinking I had to set this up within the restic-wrapper. Can you confirm I can leave out all the rclone options and only have to configure:

export RESTIC_REPOSITORY
export RESTIC_PASSWORD
export RESTIC_SOURCE

I'm actually planning to do the opposite of what most people probably do. Instead of backing up to OneDrive, I want to backup to the localhost. So in that case I complete the variables as follows:

export RESTIC_REPOSITORY=/mnt/storage
export RESTIC_PASSWORD=ooohsoosecret
export RESTIC_SOURCE=rclone:MyOneDrive:Documents

(MyOneDrive is the name that I used in rclone and Documents is the folder I want to backup to my localhost:/mnt/storage)

Do you foresee any problem with this approach?

[peterrus]

Ah yes, that might not be very clear in my documentation yet: Technically you shouldn't have to use ~/.config/rclone/rclone.conf as you can substitute that with environment variables that live inside the .env file. This way you have a backup of everything you need to use Restic when you run ./restic_wrapper -a export-config.

That being said: You can just as well configure everything rclone related in the rclone config file and leave that out of the .env file.

Regarding using Restic in the opposite fashion: Honestly I have never tried this nor given it any thought but there should be nothing in restic_wrapper that prevents you from doing so. If Restic can do it, the wrapper should be able to do it.

Let me know if that works for you!

[eRJe79]

OK, read everything again with a fresh mind. I think I understand the concept now of either using the rclone.config (only one connection) or using jobs.env to setup the connection, which would allow me to connect to multiple OneDrive's. I'm going for the latter option as I have two OneDrive account to connect to.

I updated the MyOneDrive.env with the following:

export RESTIC_REPOSITORY=/mnt/storage
export RESTIC_PASSWORD=ooohsoosecret
export RESTIC_SOURCE=rclone:MyOneDrive:Documents

and from the rclone.config I copied:

#[MyOneDrive]
export RCLONE_CONFIG_MYONEDRIVE_TYPE="onedrive"
#export CLONE_MYONEDRIVE_DRIVE_TYPE=personal
export RCLONE_MYONEDRIVE_TOKEN={"access_token":"E--------------------------------
Z+G---------------------------------8+/ZYKI2ZWbjM59---------------------------------bD1EEs
QDpCrxIoq-------------------------UBCRq7BRxv--------------------------- ix2+jtlKIx96pgNP/DiTe
1SZ---------------------------------1jGTi1vA------------------------------------------f0SSqIkqy
2T2/jBmzSx+R---------------------------------Yy0SC---------------------------------eIcTnDe
z75S---------------------------------lcs---------------------------------Ki6OvhxpG9GLXj06vJA/
Lkvxurq---------------------------------ag+V---------------------------------qC8f/C
Lwci-----------hzGx---------------------------------6eP8m-----------------------kw4/AVZyHa
UtA-----------------Nzse3f+---------------------------------eGUve------------6PmYuR
gfB---------------------------------9CMACt---------------------------------4Ws92p
r1iIghjfzHpjJ/4mxiAI=","token_type":"Bearer","refresh_token":"M.R3_BL2.-CZIJEK*Q
i7q---------------------------------ZaD0vIrd---------------------------------BWF
qq!Z---------------------------------uf*YiQ!---------------------------------3JhTvhgTYqi
H*---------------------------------CeF4WN---------------------------------mGbwIfsgh
rZN---------------------------------6h5lbnzfiLqZs---------------------------------DlSdlL
7Wcpr8xer---------------------------------GP","expiry":"2022-09-04T18:55:23.120277888Z"}
export RCLONE_MYONEDRIVE_DRIVE_ID=e------7-----d-----ed

Now when I run ./restic_wrapper -b MyOneDrive -a run -r init, I get the following return:

/root/restic-wrapper/configs/MyOneDrive.env: line 17: export: `RCLONE_ONEDRIVE-RJ_TOKEN={access_token:EwCAA8l6BAAUkj1NuJYtTVha+Mogk+HEiPbQo04AATlShbkBdXAcfrq': not a valid identifier

In fact, I get the same return (not a valid identifier) with any identifier.

[eRJe79]

I did some more testing.

First, Restic Backup does not support to use Rclone OneDrive (directly) as source. The cloud can only be a repository in that way. https://forum.restic.net/t/backing-up-remote-rclone-to-local-disk/1803.

A possible workaround is to use rclone mount and have Restic backup from this mount to the local filesystem. This has some caveats.

1. The OneDrive (or other cloud service) will always be mounted unless a mount is invoked before Restic starts a backup.
2. The mount requires FUSE.
3. The data on the mount is considered unreliable.

My workaround for now is not very sleek but more robust. I use rclone to copy a 1:1 backup to my local drive and then have restic make snapshots of this copy, using your script.

Out of curiosity to test your script the correct way, I set it up the designed way (making snapshots local to OneDrive). I ge the same error messages on the invalid identifiers for the OneDrive section in the .env.

[peterrus]

Apologies for the late reply.

I think your workaround seems like a decent one. I understand it is not ideal but I am afraid this is just a limitation you have to work around. However I wouldn't dismiss mounting onedrive using fuse right away because it might work just fine for your usecase! If restic fails to back a file up because fuse couldn't provide it you should get a failed run and be alerted of that through restic_notify (if configured correctly). That said, I have never used OneDrive myself.

When looking at the .env file you provided I notice two things:

• I suppose that is just a access token for testing right? Because it is now public (not trying to be condescending, just making sure there are no misunderstandings).
• The file is not formatted properly. Env vars can only be one line long (not split over multiple lines). But you can try something like this: https://stackoverflow.com/a/49489260/1417190. Another option would be to convert the multiline json blob in to single line one and then escape it properly before putting it into the .env file. You might have to read up on bash escaping a bit (e.g. https://stackoverflow.com/questions/51485566/escaping-quotes-in-environment-variable-within-a-shell)

If all else fails you could just configure the OneDrive remote in using rclone config and completely leave all that config out of the .env file.

Good luck!