Job failed to run "Clear Destination" and stuck in "InProgress" state

[lkzcgfvf]

Environment

• Deploy method: Kubernetes using latest Helm chart
• Olake version: 0.4.1-0.5.1

Description

If you add new table to the list of streams then Olake runs "Clear Destination" for destination folder to remove any files in folder that will be used to store Parquet files for this new table. Its correct and expected, however if this API call to destination fails - then job stuck in "InProgress" state forever.

Steps to reproduce

1. Create destination with type Amazon S3 and point it to Google GCS bucket ( its compatible with S3 API)
2. Create job that will use this destination
3. Add any table to the list of streams
4. Run sync, it should finish correctly
5. Add new table to the list of streams
6. You will see that "Clear Destination" job started. It generated error ( its expected) and it stuck forever ( not expected)

Here is error message generated by worker

failed to clear destination: 
failed to drop the streams: 
failed to clear S3 files: 
failed to clear S3 prefix <source>/<table>: 
batch delete failed for filt Path <source>/<table>: 
BatchedDeleteIncomplete: 
some objects have failed to be deleted. caused by: 
failed to perform batch operation on "<source>/<table>/XYZ.parquet" to "tpt_olake": NotImplemented: 
A header or query you provided requested a function that is not implemented. status code: 400, request id: , 
host id: failed to perform batch operation on "<source>/<table>/XYZ.parquet" to "tpt_olake": NotImplemented: 
A header or query you provided requested a function that is not implemented. status code: 400, request id: , host id:

Expected behavior

Jobs fails, container/process is not running, job status updated to "Failed".
It also would be great to fallback to individual deletes when S3 backend returns NotImplemented for DeleteObjects.

Actual behavior

job stucks in "InProgress" state forever.

[vishalm0509]

@lkzcgfvf Since clear-destination is a destructive operation, we cannot allow it to fail in a partially completed state. In the event of an error, it should continue retrying indefinitely until the issue is resolved, ensuring consistency and preventing any half-applied changes. (This is an intended design decision)

In your case, as the clear-destination error is expected, we have an api to recover in-progress (failing) clear-destination operation. It cancels the clear operation and resumes the sync operation.

COOKIE_JAR="/tmp/olake.cookies"; trap 'rm -f "$COOKIE_JAR"' EXIT
curl --location --request POST 'http://localhost:8000/login' \
  --header 'Content-Type: application/json' \
  --data-raw '{"username":"admin","password":"password"}' \
  --cookie-jar "$COOKIE_JAR"

curl -i --location --request POST 'http://localhost:8000/internal/project/123/jobs/<JOB_ID>/clear-destination/recover' \
  --cookie "$COOKIE_JAR"

Note: Please update the hostname, port and the job_id as per your configuration.

If data is already present in the destination for the affected stream, It is recommended to drop the data manually before continuing with this process of recover

[lkzcgfvf]

@vishalm0509 @vaibhav-datazip
I agree that retry logic is very important for this operation, however "continue retrying indefinitely" could be dangerous approach.
Job that stuck in "Running" state forever and it

1. doesnt generate error
2. keeps running and do not allow to run itself next scheduled run

As result it will break pipeline and nobody will notice. In my particular case job was in this state ~18 hours, CDC pipeline was not collecting data from MySQL and we didnt know about it.

I would propose to adjust retry logic

1. add logic that split errors in 2 categories: recoverable and non-recoverable
2. add retry logic into recoverable errors ( network error, http 429, )
3. add configurable limits to retry logic ( ow many times to retry OR for how long)
4. stop work and exit with error for non-recoverable error ( like AccessDenied OR NotImplemented for DeleteObjects)

[vikaxsh]

hey @lkzcgfvf , We will implement a fallback mechanism in few days, in case the S3 backend returns a NotImplemented error for DeleteObjects. Thanks for reporting this issue

[shubham19may]

@lkzcgfvf how much of a blocker this is? whats the priority? if its an old production affecting job, and you have already done a big historical load and cdc was running, and mysql deletes binlog (usually in 7-15 days depending on retention setting) so it will be a big problem as you will have to do historical load again.
So we can internally prioritise this a little faster. We are a little short on bandwidth right now, thats the reason we are asking.

Also can you specify on how exactly was this clear destination was triggered ?

The reason why we are launching the fix for NotImplemented error for DeleteObjects for GCS and not the indefinite retry thing,

Clear dest, has two types,

1. for all streams
2. For only some of the affected streams.

For 1, you can go to job settings and start it again from there. no issues.
but for 2, there is no retry feature for clear destination for specific streams. We need time to think about product flow for this.

[lkzcgfvf]

its not super urgent, we found workaround but it would be great improve error handling behavior

[vikaxsh]

hey @lkzcgfvf, We’ve released Olake v0.6.1 with fixes for the following issue. Please try it out and let us know your feedback.

[vaibhav-datazip]

hi @lkzcgfvf , were you able to test this update. Incase you are still facing some issue please let us know.

[vaibhav-datazip]

Closing this issue as it was solved in v0.6.1