Skip to content
This repository was archived by the owner on Oct 11, 2025. It is now read-only.
This repository was archived by the owner on Oct 11, 2025. It is now read-only.

Task is ahead/behind of ShardState #98

@ckhan

Description

@ckhan

We use mapreduce and pipelines for various administrative jobs on GAE.
Almost every day, I'll see warnings like this in the log:

LOG 2 1468861523504730 Task 1571100778565B9A65B43-5-0 is behind ShardState 1. Dropping task.
..
LOG 2 1468861525573810 Shard 1571100778626ECE727ED-7 is not active. Possible spurious task execution. Dropping this task.
..
LOG 2 1468861525628980 ShardState is {active:False,last_work_item:Key('FileEntity', 5711710446944256, namespace='5666083260334080'),result_status:success,slice_id:3,update_time:2016-07-18 17:04:00.927030,}
..
LOG 2 1468618964079700 Task 15711250257171C594209-3-2 is ahead of ShardState 1. Waiting for it to catch up.

They seem to be innocuous (just retries?), but they are making a lot of unwanted noise in the logs.
Questions:

  • what do these errors mean?
  • if it's innocuous, should it be logged as warning? should we drop logging level in our fork?
  • if not, what can we do to mitigate the underlying issue?

Thanks for any pointers!
-ck

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions