For export processes that use the "All data added or updated since last export" filter, need to modify what's considered the "last export"

[jthomale]

In a production environment, scheduled export processes that run regularly to sync data from Sierra to the API need to ensure that they capture all changes made since the last time the sync process ran. If they don't, you end up with records in the API that are out of sync with Sierra.

The "all data added or updated since last export" filter is supposed to ensure this happens correctly. Right now, it tries to find the datetime that the last export job with the same export type ran, and it uses that in a date range query to pull all records that have been updated since that time.

The problem is, if you manually run an export with a different filter (one that targets a specific record range, for instance), then the next scheduled process that runs uses that job as the "last export." This means anything added or updated between the previous scheduled job and the manual job is lost.

Example.

• 2:00 PM: Scheduled ItemsToSolr sync runs. The API is up-to-date with Sierra when it finishes.
• 3:30 PM: I manually trigger an ItemsToSolr job to load just one specific record.
• 4:00 PM: The next scheduled ItemsToSolr sync runs. Instead of grabbing everything that was updated from 2:00 PM to 4:00 PM, it grabs everything that was updated from 3:30 PM to 4:00 PM (because the 3:30 job becomes the last export). Records updated between 2:00 PM to 3:30 PM never get synced.

There are a couple of potential solutions.

1. Maybe the "last export" should be the last export of the same type that used the same filter type.
2. Maybe the "last export" should be the last export of the same type triggered by the same user.

Fix 1 makes more logical sense, but in some cases it's useful to be able to test scheduled exports (in a development environment) without loading the entire database. In this case, you can just run an export manually and then start the scheduled exports. So fix 2 may be the way to go, as long as you set up one user to use for (most) scheduled exports and another to use for manual exports. (And, of course, set the EXPORTER_AUTOMATED_USERNAME setting appropriately.)

[jthomale]

After years of managing production export processes, I've actually found the behavior that this issue is calling a bug to be more helpful than not. I've made it a habit to delete any one-off jobs I run in production that would interfere with the scheduled jobs. As the last paragraph in the issue notes, it is useful to be able to use a combination of targeted manual exports and automated "everything since the last export" jobs to manage what gets loaded when.

The only thing I sometimes find myself wishing I could do more easily is to be able to schedule a one-off process to run later, like to run a specific update for a specific group of materials, and not have that one-off process interfere with what gets selected for the normal scheduled update processes. (I.e., if I'm not around to delete that one-off when I schedule it to run.)

I still think that the recommended fix 2 mentioned in the issue is worth making. But, in the case I just mentioned, since all scheduled tasks run as the same user, it wouldn't help.

So, I'd add this to the suggested fix.

Add the ability to select which user an export task runs as, both when scheduling a task and when triggering a manual task to run. This would address the additional use case and would just make it easier all around to control the behavior without having to log out and log back in as a different user.