Skip to content

Add ingester staging-directory config and path safety validation #4

@ian-ross

Description

@ian-ross

Part of #3.

Implement the configuration and path-safety foundation for ingester database isolation.

Decisions

  • Add a required paths.staging-directory option.
  • Preserve existing semantics:
    • data-directory: shared/public SQLite snapshots for clients.
    • staging-directory: durable local ingester-owned SQLite files; must survive ingester restart.
    • scratch-directory: ephemeral receiver/export scratch; may be deleted between restarts.
  • All three path roots must be distinct and none may be nested inside another, using resolved/real paths.
  • Config should validate path configuration but should not create directories.
  • DBCache should also validate paths because it can be instantiated directly.
  • DBCache should require data-directory to exist and may create staging-directory and dedicated export scratch subdir as needed.

Implementation checklist

  • Add staging_directory to Config._init_paths() from paths.staging-directory.
  • Add required path distinct/non-nested validation in Config.
  • Add duplicate validation in DBCache.__init__().
  • Update config.toml.template comments to document all three path roles.
  • Update feder-ingest config wiring to pass data_directory, staging_directory, and scratch_directory explicitly to DBCache.
  • Add ingester config values:
    • ingester_export_interval, default 1 hour.
    • ingester_finalize_after, default 12 hours.
  • Pass those values explicitly to DBCache.

Tests

  • Config requires paths.staging-directory.
  • Config rejects equal or nested combinations of data-directory, staging-directory, and scratch-directory.
  • DBCache also rejects equal/nested path roots when constructed directly.

Notes

This is the safety boundary for #3: public data, durable staging, and ephemeral scratch must be separate so the ingester cannot accidentally expose live SQLite/WAL files to clients.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions