Skip to content

[feat]: ultimate normalization script #222

@ShaunSHamilton

Description

@ShaunSHamilton

Going forward, there are likely many times we may want to normalize records. As such, we need a smart script:

Musts

  • not bring down production to run script
    • it should be a script we can tune to not hammer the db, and quietly run in the background
  • easy to define/add transformations
  • fault tolerant
  • keep track of versioning
    • must not repeat work - the script should be stopped/started without hesitation, and start off where it left off
  • fearlessly concurrent
    • the api will write a lastUpdated field to the user record which should allow all ops to be done without losing data

Niceties

  • dynamically asks database how loaded it is, and accordingly adjusts ops/rps

Notes

Currently, the idea is to add a schemaVersion field to the user record.

I would prefer to write this in Rust:

  1. Ease of distribution
  2. Strict typing

That said, this might go against "easy to define/add transformations".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions