-
Notifications
You must be signed in to change notification settings - Fork 16
Open
Description
Going forward, there are likely many times we may want to normalize records. As such, we need a smart script:
Musts
- not bring down production to run script
- it should be a script we can tune to not hammer the db, and quietly run in the background
- easy to define/add transformations
- fault tolerant
- keep track of versioning
- must not repeat work - the script should be stopped/started without hesitation, and start off where it left off
- fearlessly concurrent
- the api will write a
lastUpdatedfield to the user record which should allow all ops to be done without losing data
- the api will write a
Niceties
- dynamically asks database how loaded it is, and accordingly adjusts ops/rps
Notes
Currently, the idea is to add a schemaVersion field to the user record.
I would prefer to write this in Rust:
- Ease of distribution
- Strict typing
That said, this might go against "easy to define/add transformations".
Metadata
Metadata
Assignees
Labels
No labels