Multi-cron job or multi-concurrency jobs

[sierrodc]

I would like to run a single unique activity (executable) in a cluster but with many schedules.
I.e:

• every minute during 11th hour of every day
• every 30minutes for the rest of the day

This is something I cannot write with a single cron expression (let me know if I'm wrong).
What I would like is to have multiple cron expression for a single job.

As an alternative, I can create multiple jobs with differnet cron expression. The problem is that these two jobs are exactly the same and I don't want that at 11:30 the activity is executed twice. (ps: the activity may last 1h, so jobs may overlaps even if I remove hour 11 from the second cron expression).
In this case I would like something like "concurrency group", so that the set { "job_minute", "job_half_hour" } cannot run both together.

[yvanoers]

Interesting. An alternative could be to introduce multiple cron schedules for a single job.
Not sure right now whether that would be easier or more difficult compared to "concurrency groups".

I am curious, though:
What makes it that the job must run every minute on the 11th hour, and afterwards must not run more than twice an hour?

[sierrodc]

It was just an example. But consider this real-live example:
We have a scheduled executable that tries to download a value from a website. We know the information will be published about at 11:10 AM. But simetime it takes time to have the value published. We want it asap to take actions.
A possible solution is:

• try to download the value every minutes from 11:10 to 11:30 (this covers 95% of cases)
• try to download the value every 10 minutes after 11:30 till noon (99% of cases)
• try to download the value once an hour after 12:00 (rare cases, no need to have the value asap, it's already late, but we want it)

Question: we have little/no knowledge about dkron. Do you think "multiple cron" job is an "easy" task to implement? maybe we can try to implement this feature by ourselves.

[yvanoers]

But consider this real-live example:
...

How do you intend to prevent the value from being processed multiple times, sequentially? So, e.g. when the value has been downloaded and processed at 11:15, but the next run at 11:16 does it all over again?
I'm asking because I'm thinking about the possibility of using several jobs with retries and job chaining for your case.

Question: we have little/no knowledge about dkron. Do you think "multiple cron" job is an "easy" task to implement? maybe we can try to implement this feature by ourselves.

From the looks of it it is easier to support multiple schedules than to support group concurrency.
What would roughly be necessary is to change AddJob in scheduler.go so it can add more than one schedule with the same job to the Cron instance. And update the API (and UI) to allow for addition of a multi-valued schedule property.
Doesn't look too hard, but I cannot objectively be the judge of what you would consider "easy".

It would be good to have @vcastellm 's blessing for adding this as a feature before you try, though. It adds complexity to dkron that we may not want.

[vcastellm]

Well, for this complex case it seems that some logic will be needed in the execution side, for this kind of "edge" cases I've always considered the possibility of manipulating runs from the execution itself, for example, you can reschedule the same job after 11:30h to run every 10mins, and so forth, in case the file is not published, in case it gets the file, you can disable your job, all this can be done calling the API.