Skip to content

Add transaction log#719

Closed
whilo wants to merge 1 commit intomainfrom
718-use-transaction-log
Closed

Add transaction log#719
whilo wants to merge 1 commit intomainfrom
718-use-transaction-log

Conversation

@whilo
Copy link
Copy Markdown
Member

@whilo whilo commented Nov 4, 2024

SUMMARY

Fixes #718

Checks

Bugfix
  • Related issues linked using fixes #number
  • Integration tests added
  • Architecture Decision Record added if design changes necessary
  • Formatting checked
Feature
  • Implements an existing feature request. Make sure the feature request has been accepted for implementation before opening a PR.
  • Related issues linked using fixes #number
  • Integration tests added
  • Documentation added
  • Architecture Decision Record added
  • Formatting checked

ADDITIONAL INFORMATION

@whilo whilo changed the title Initial prototype for tx-log. Use transaction log. Nov 4, 2024
@whilo whilo changed the title Use transaction log. Add transaction log Nov 4, 2024
@TimoKramer TimoKramer force-pushed the main branch 6 times, most recently from 15ca402 to 6149e16 Compare August 29, 2025 15:44
@whilo
Copy link
Copy Markdown
Member Author

whilo commented Jan 13, 2026

For now we will stick to no transaction log to optimize for distributed readers and minimal IO during reading. The problem with transaction logs in our case would be that distributed readers would have to reapply them on every read, a disadvantage we also had with the hitchhiker-tree (in a fractal form). Since we have a single writer architecture and write throughput is generally fast enough such that we don't increase latency by writing each new path into the tree indices in parallel on a transaction, the main price we pay is more storage requirement and higher write throughput. The storage requirement can be handled with running gc regularly. And the write throughput is not that much better with a transaction log, too. As soon as you hit a high enough throughput rate and the log overflows on most transactions you have the same costs. So for now let's postpone the transaction log for simplicity and optimal read performance. On high write throughput Datahike also auto-batches transactions and reduces write amplification this way.

@whilo whilo closed this Jan 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use transaction log

1 participant