Stale SubjectSet during linking of large number of subjects

[lcjohnso]

We're still running into issues with stale subject sets during subject linking (as part of subject upload).

Connected to #290 but opening a new issue here to document a specific technical issue and solution.

Problem

A SubjectSet.add() call returns a 409 status and raises the following exception:
PanoptesAPIException: Attempted to touch a stale object: SubjectSet

Note: retries only apply to status code >=500; 409 breaks out of retry loop, and json_request raises the exception.

Recent instance: while trying to link 37k subjects London HogWatch (see Freshdesk ticket), stale object exception is raised after 23k are uploaded.

It is not clear what action is happening to the SubjectSet to trigger the stale object error. Originally, I thought this was happening due to subject set completeness checks and updates, but not all cases are explained by this (exception occurred when subject set is unlinked from workflow). Testing with a subject set name change (one of the only other editable fields on SubjectSet resource) while linking is in progress did not result in a stale object. Perhaps related to the

Previous Action

#298 added a SubjectSet.reload() just before the add() which helps, but if the input list is large (think: 10k subjects that is handed 100 at a time by batchable()), then the add action can run for long enough that something can happen to the subject set.

Potential Proposed Solution

Would we consider adding a SubjectSet.reload() as part of each batched add()? Specifically, we could implement this in a few different ways:

1. (my pref) make the SubjectSet.add() function batchable instead of batching the super LinkCollection.add(), thereby forcing the reload for SubjectSet each time.
2. (probably bad idea) add a reload to super LinkCollection class (see here: https://github.com/zooniverse/panoptes-python-client/blob/master/panoptes_client/panoptes.py#L1044), either for all parent classes or adding logic to only refresh for SubjectSet parent.

I also wonder if we should try to catch the 409 for SubjectSet.add() explicitly and retry. I'm bullish about essentially ignoring and overriding stale object errors for subject sets because adding subjects should not override any existing state that we would be concern about (unlike edits of other resources like Workflows or Projects).

[PmasonFF]

I have tried to get around the problem by forcing the subject_set.add into batches of one hundred with a full regeneration of the subject set for every batch ie not just a SubjectSet.reload() ) but subject_set = SubjectSet.where(project_id=project.id, display_name=set_name).next()

Then within a batch if I get ANY panoptes_client.panoptes.PanoptesAPIException I try to first to reload the subject set and try again up to three times. If that fails I try to the whole thing over from where it failed including a fresh connection to panoptes, and a new instance of the subjectset. Generally if a reload() works, it happens on the first retry, if it does not the best solution seems to be the full start from scratch route.

Overkill? Yes certainly, but the loss of time this has cost various teams, and the variety of errors we were seeing at this step last year forced this approach. (ie linking subjects to subject sets - "stale object" is only one of several errors that were occurring).

Keeping track of the subjects created and linking them at the end in batches is a pain compared to doing a subject.save() subject_set.add(subject) for each subject. For a while back in 2024 that approach was near impractical (recall the discussions and comments in the forums) but recently the error rate has been somewhat lower and I have note heard as many complaints. Still it would be nice to know exactly what would be needed to actually fix this.

[PmasonFF]

I have tried to get around the problem by forcing the subject_set.add into batches of one hundred with a full regeneration of the subject set for every batch ie not just a SubjectSet.reload() ) but subject_set = SubjectSet.where(project_id=project.id, display_name=set_name).next()

Then within a batch if I get ANY panoptes_client.panoptes.PanoptesAPIException I try to first to reload the subject set and try again up to three times. If that fails I try to the whole thing over from where it failed including a fresh connection to panoptes, and a new instance of the subjectset. Generally if a reload() works, it happens on the first retry, if it does not the best solution seems to be the full start from scratch route.

Overkill? Yes certainly, but the loss of time this has cost various teams, and the variety of errors we were seeing at this step last year forced this approach. (ie linking subjects to subject sets - "stale object" is only one of several errors that were occurring).

Keeping track of the subjects created and linking them at the end in batches is a pain compared to doing a subject.save() subject_set.add(subject) for each subject. For a while back in 2024 that approach was near impractical (recall the discussions and comments in the forums) but recently the error rate has been somewhat lower and I have not heard as many complaints. Still it would be nice to know exactly what would be needed to actually fix this.

[lcjohnso]

Hi @PmasonFF -- I'm glad to hear that API behavior has generally improved as of late, likely tied to our migration to new database instances back in late February. Nevertheless, we'd like to include some additional SubjectSet.reload() calls in an attempt to avoid stale object errors when we can.

Do you have a reliable way of triggering these stale object exceptions? If you have identified a set of circumstances that tends to lead to errors, that would be helpful in further debugging. If not, and this is truly an unpredictable and intermittent issue, then we'll proceed with our approach of using more reload() calls to head off issues before errors occur. As I state above, I have not been able to trigger the failure on my own, holding back deeper investigation.

As an aside re: proces -- when I recently worked with the London HogWatch team, I preferred a scripted upload process where subject creation & media upload were separated from subject linking to a SubjectSet, by appending subjects to a list as they were created, and then linking all subjects using one call of SubjectSet.add(). One advantage to this: failures caused by linking happened later after all subjects and media were uploaded, meaning the recovery action (linking any remaining unlinked subjects) could happen quickly and we didn't have to wait for additional uploads to occur. And passing a single large list delegated the batching to the Client's built-in batchable functionality.

[PmasonFF]

Unfortunately I have no useful data - anecdotally it appeared worse in the US afternoons, when the subjects being created were not sequential implying more than one team uploading. At the worst, sometime in the spring of 2024 Nebraska Wildlife could not run through subject sets of even just a few hundred. Recently Back Bay has uploaded a set with 43000+ and I believe they used essentially the same script - with a subjectset.add for subject as it is created and uploaded. I do not know what size of batches the team uploaded - we could likely look at the zooniverse numbers and sort that out. I updated Nebraska Wildlife to Python 3.12 and the client to 1.6.1 with no apparent difference - you may recall we were getting SSL errors as well as stale object errors at the time - the SSL errors continued into late spring and then vanished. I believe Back Bay is on python 3.13 with added setuptools and panoptes 1.6.2.

So no, I have no way of triggering the error. Clearly it is worse the longer the subject set is held open - My hardened script even tried to minimize the time from creating the subject to when the metadata and locations were updated though I do not believe I have ever seen a stale object error for the subject, just for the subject set my scripts was attempting to keeping open for the full duration of the upload session.

Perhaps you can write a script to open a subject set and query it periodically (we did get stale object errors while doing things like updating metadata for large subject sets and other operations, not just uploading and adding subjects) the challenge would be to capture the etags and other info in the API to see what is what when it finally crashes...

[astrohayley]

Ran into this issue with uploading subjects for Galaxy Zoo.

Following suggestions here, I finally got it to work by creating 100 subjects and then adding them to the subject set with the following catch:

try:
    subject_set.add(new_subjects)
except: # PanoptesAPIException
    subject_set.reload()
    subject_set.add(new_subjects)

which isn't ideal, but did work.

I was also catching issues with Subject.async_saves() which wasn't allowing me to run subject.save() or subject_set.add(new_subjects) without erroring out, because it said I wasn't logged in (despite having logged in). Not sure if it's related, but thought it was worth mentioning, since I was running into both issues at the same time.

I don't know how to reliably trigger this issue. I know my code does reliably trigger it, so let me know if it's helpful to post the whole thing here. I'll note that I'm running this code from a server in Spain (ESA Datalabs) and it was causing me problems I don't usually run into when I'm working from a local machine.