How to retry/postpone an activity without failing it?

[glukinho]

Hi, first of all thank you for the great package.

My question is: can I postpone an activity to later time without failing it?

I have a workflow which includes uploading a file to an external API, obtaining task_id, and waiting for processing completion by periodically requesting processing status using obtained task_id. This periodic requesting is done as dedicated activity - when it is done, the workflow goes on (downloading result etc).

So CheckForCompletionActivity logic is basically:

• request task status from external API using task_id
• if status is "finished" then finish the activity so the workflow goes on
• if status is "processing" then retry the check later

When I did this with regular Laravel chained jobs I could simply return $this->release(60); inside a job and this job is reprocessed after 60 seconds without errors. But I can't do this with workflow + activity: when I return anything the activity is considered finished and workflow goes on prematurely.

What I can do now is throw ProcessingIsNotDoneYetException from the activity and set $backoff value. But I personally don't like it as this is not an error by nature, nothing is broken, I just need to retry the check later. My logs are full with ActivityFailed events and rendered ProcessingIsNotDoneYetException exceptions which is not error at all, so I have to tune $exceptions->dontReport() values. Also my queue worker has FAIL rows which are not failures...

Looped sleep is not an option of course as I don't want to lock queue worker.

What did I miss? Can I retry/reschedule an activity later without failing it or throwing exceptions?

Thanks a lot!

[glukinho]

Another bad thing about using exception for postponing an activity is I want to really fail (and not retry) an activity if external API gives "failed" status as in this case something is really wrong with the processing. I don't see a nice way to retry an activity in case of ProcessingIsNotDoneYetException and fail without retrying in case of ProcessingFailedException (or any other exceptions).

Maybe I'm totally missing something, really appreciate any ideas. Thanks!

[rmcdaniel]

@glukinho

Instead of the using the while loop in a workflow like my previous example, you could make it work via exceptions and retries.

use Exception;
use Workflow\Activity;
use Workflow\Exceptions\NonRetryableException;

final class CheckForCompletionActivity extends Activity
{
    public function backoff(): array
    {
        return [60];
    }

    public function execute(string $taskId): string
    {
        $status = 'processing'; // replace with call to external API

        if ($status === 'processing') {
          throw new Exception('processing');
        }

        if ($status === 'failed') {
          throw new NonRetryableException('failed');
        }
    }
}

then your workflow would look like this

use function Workflow\activity;
use Workflow\Workflow;

final class UploadWorkflow extends Workflow
{
    public function execute(string $filePath)
    {
        $taskId = yield activity(UploadFileActivity::class, $filePath);

        yield activity(CheckForCompletionActivity::class, $taskId);

        yield activity(DownloadResultActivity::class, $taskId);
    }
}

If there is an exception, the activity will retry every 60 seconds forever based on the custom backoff policy and the default retry policy.

The key here is NonRetryableException which can stop the retries when needed.

[glukinho]

I don't get it. Why exactly throw new Exception('processing') retries an activity and throw new NonRetryableException('failed') does not?

[rmcdaniel]

@glukinho A NonRetryableException skips the retry policy and fails the activity immediately. But Exception is just counted as "using up one of the activity's tries". The default retry policy is public $tries = 0; which means to keep retrying forever and never fail the activity.

See:
https://laravel-workflow.com/docs/configuration/options
https://laravel-workflow.com/docs/failures-and-recovery

[glukinho]

Ah, I missed this in documentation. I thought NonRetryableException is just an example. Now I see \Workflow\Exceptions\NonRetryableException is never retried, this is by design of the package for cases such as mine. Very nice!

Thank you very much for quick and meaningful answers! Good luck!

[rmcdaniel]

@glukinho

The way to do that is to have your activity return finished when it's done otherwise pause for 60 seconds inside the workflow and then retry it.

use function Workflow\{activity, seconds};
use Workflow\Workflow;

final class UploadWorkflow extends Workflow
{
    public function execute(string $filePath)
    {
        $taskId = yield activity(UploadFileActivity::class, $filePath);

        while (true) {
            $status = yield activity(CheckForCompletionActivity::class, $taskId);

            if ($status === 'finished') {
                break;
            }

            yield seconds(60);
        }

        yield activity(DownloadResultActivity::class, $taskId);
    }
}

Under the hood, the yield seconds(60); will do $this->release(60); and cause the workflow to pause for 60 seconds and then resume.

[glukinho]

So, you suggest to handle retrying on workflow side, while activity doesn't know about retrying itself at all, it just returns obtained status. I didn't think this way, maybe because I was tending to keep workflows as clean as possible. Now I see this approach is clean and nice. Thank you very much!

[rmcdaniel]

@glukinho It is just one option. The exception version is cleaner (as you have pointed out) and I have also posted an example of how that would work. Actually, my preferred method would not be polling at all but rather have the remote system notify via a webhook when completed and then use a signal to resume the workflow from there. But that might not be an option for you.

[glukinho]

Yes, polling is the only option here, the external API doesn't support anything else.

Honestly, polling from my side seems more reliable (and retriable) to me than having a webhook. If my server is down or external service has failed sending me a notification it would be hard to resume the workflow. When I poll the external service I have full control of the situation.