Allow to specify a start date/time to be able to set the intiale date of the oplogs (be integrated with sanpshot bakcup procedure)

[ericfranckx]

Hi,
it should be to be able to have a parameter like -d dd/mm/yyyy-hh:mm:ss ( or now ) that will tell to the the tool from which it should start the first oplogs entry it should take into account.

Example:

• do a snapshot on disk level at 09:30:00
• start the backup.sh tool and tells that I want to be able to recover from the start of my snapshot (and don't any oplogs older then 09:25 for example ) --> snapshot.sh ... -d 09:25:00 (so take the actual day if not set).
  So only oplogs generates since 09:25 will be extract and write in the OS log files and don't need the huge entries in the oplogs before the snapshot --> already include in the snapshot.

Also a retention time parameter like keep (-k) should be nice like you tells that you want to keep minimum xx times (specified like HH:MM:SS).
So again if you do a snapshot every hour, and set it to 1:10, you keep always 1:10 of oplogs and good reply to the last snapshot. (The tool should delete old files or renamed the files that contains the older oplogs entries --> condition : all oplogs in a files should be older than the -k parameter; then good candidate files are automatically renamed with an extra extension like .del and so we can delete the files, copy it to another location (archive it).

Regards,

Eric

[viveksd87]

Hi Eric,
Thank you for your inputs.

In fact we had thought of providing the feature wherein a user could selectively back up documents within a time interval. However, after some detailed discussion within the team we had decided to drop the idea and wait for the feedback to drive that feature.

A major reason that caused the idea to be dropped was, the potential inconsistency it might cause when performing the restore. For example, if your snapshot contains a document to be deleted, and the particular document was created very long back in time (i.e. insert not included in the snapshot), then the delete operation would fail while restoring on an empty standalone mongod.
Also, oplog.rs usually being a capped collection, we assumed it shouldn't be much overhead to perform backup from the start of the collection.

However, if you feel that incorporating this feature will help in making the job easier for devops guys and add value to the tool, please let us know.

Thanks,
Vivek

[ericfranckx]

Hi,
I understand but it is for that this is for specific case (extra options/features ;-) ) certainly helpful with snapshot on storage level.
Also the time of snapshot (example every hour we run a snapshot) should be cover by the oplogs generated OS files , so If my snapshot runs every hour --> and can be restored, I need only updates done between 2 snapshots (delta).
So should be nice to for example:

• I run my snapshot very 2 hours
• I need then to keep oplogs update/info for minimum more than the 2 hours (an keep more the last 2 hours oplogs update are critical)
  --> so I start the backup.sh and it will look to "updates" in the log files older than that and could be deleted.

The snapshot policies and backup.sh policies should be aligned to covers different scenarios.

Also for example in oracle you have the redo logs ... --> same as your files generates by backup.sh but with some additional parameter a child process could copy older files to another location.

Oplogs can becomes big, if you follow some rules of mongodb where if you use Replicaset, MMS backup, you should be able to keep extra updates during a none-sync situation (replica server note available, ...) .

Eric

[ericfranckx]

If 1 generates 10 GB of data in my collection (migration project ) and then start a snapshot at 10:00 AM (for example) on disk, I don't need to find this oplogs entries in the files generates by backup.sh.
I want to be able to tell backup.sh to take into account the time of my snapshots and so will only look to new oplogs entries after the time specified. The plan is to start 1 hour later again a new snapshot at 11:00 AM , but if I have a crash, I need only to restore the snapshot of 10:00 (on my storage I can restore 10 TB in some seconds ... between 1 and 10 seconds) and RE-APPLY only oplogs entries that were generates after 10:00 until the last entry in the generates logs files on OS level.

Correct ?
If I want a more older situation, then I could restore the snapshot of 05:00 AM. But then I don't have the oplogs generates files (by backup.sh) but depends the additional features where like on Oracle you could have the redo logs file, then the archives (offline ) redo logs --> that could be snapshoted also on storage level depending the RTO/RPO a have to support in my SLA.

Regards,

Eric

[viveksd87]

Hi Eric,
From your explanation I understand that you want to take a snapshot, and then perform only incremental backups after the snapshot, so that if anything goes wrong after you have taken the snapshot, you will
1.) restore the snapshot completely.
2.) re-apply the new oplog entries from incremental backup.

Please confirm my understanding.

If my understanding is correct, then I would say that Tayra can do that job for you.
There is a file "timestamp.out" that stores the timestamp of last backed up document.
So once you have terminated the backup process, the entry in "timestamp.out" is updated with the timestamp of last document in the backup. Next time when you start the process, the backup resumes from where it had left based upon the entry from "timestamp.out" file.

Now when performing a restore you could perform complete restore from the backup files generated. Or you could perform selective restore by using the options --sSince and --sUntil provided for restore.sh.

Please let me know if you meant otherwise.

Thanks,
Vivek