Attempted to touch a stale object: SubjectSet.

[fulsdavid]

Having an intermittent issue when uploading subjects via the python client. No indication that I am losing network connectivity. I spawn up to 20 processes working on preparing subjects from individual files, then all uploading to the same subject set. I basically create the subject, add the png file, get the subject id, add that to the already prepared dictionary with the manifest, then add the manifest to the subject. Once all subjects are complete, I try to add them to the proper subject set where I sometimes get this error:

panoptes_client.panoptes.PanoptesAPIException: Attempted to touch a stale object: SubjectSet.

Relevant code snippet is:

lNewSubjects = []
        for det in dManifest:
            s=Subject()
            s.links.project=project.id
            s.links.subject_set=subject_set.id
            for filename in dManifest[det]['sImageFilenames']:
                pngFilename = var.sWorkingDirectory + filename.rstrip('.arch.fz') + '_' + det + '.png'
                #Catch for any subject with less than max number of files
                if os.path.isfile(pngFilename):
                    s.add_location(pngFilename)
            if var.bUpload:
                s.save()
                dManifest[det]['iSubjectId'] = int(s.id)
                s.metadata.update(dManifest[det])
                s.save()
                lNewSubjects.append(s)
                #subject_set.add(s)
            else:
                dManifest[det]['iSubjectId'] = None
                
        if var.bUpload:
            try:
                subject_set.add(lNewSubjects)
            except Exception as ex: 
                var.logger.error('SUBJECTUPLOAD Error: ' + str(ex))
                var.logger.error(traceback.format_exc())

Is it not possible to add subjects to a subject set from simultaneous processes?

Thanks for any info!

[camallen]

We use conditional requests to ensure a client doesn't clobber another clients changes.

As such you'll need to reload the subject set to get a refreshed view with the updated changes (most likely new linked subjects count or similar) before trying to add the new batch of subjects.

Running these requests concurrently creates a race condition of state changes, you may have to add an exception handler with some retry logic to correctly handle this issue as it could be unpredictable.

You may want to investigate how we do this in the cli tool https://github.com/zooniverse/panoptes-cli/blob/fb5da0d61fe50d441baef5d62079d1f91e2b5a46/panoptes_cli/commands/subject_set.py#L411-L424 using async_saves() and small batch links to do concurrent uploads that avoids the race condition on the subject set.