Skip to content

Refactor indexer data model to not store tx_hash multiple times in different tables#437

Closed
aditya1702 wants to merge 31 commits intobackfilling-metricsfrom
statechanges-data-model
Closed

Refactor indexer data model to not store tx_hash multiple times in different tables#437
aditya1702 wants to merge 31 commits intobackfilling-metricsfrom
statechanges-data-model

Conversation

@aditya1702
Copy link
Copy Markdown
Contributor

@aditya1702 aditya1702 commented Dec 21, 2025

What

[TODO: Short statement about what is changing.]

Why

[TODO: Why this change is being made. Include any context required to understand the why.]

Known limitations

[TODO or N/A]

Issue that this PR addresses

[TODO: Attach the link to the GitHub issue or task. Include the priority of the task here in addition to the link.]

Checklist

PR Structure

  • It is not possible to break this PR down into smaller PRs.
  • This PR does not mix refactoring changes with feature changes.
  • This PR's title starts with name of package that is most changed in the PR, or all if the changes are broad or impact many packages.

Thoroughness

  • This PR adds tests for the new functionality or fixes.
  • All updated queries have been tested (refer to this check if the data set returned by the updated query is expected to be same as the original one).

Release

  • This is not a breaking change.
  • This is ready to be tested in development.
  • The new functionality is gated with a feature flag if this is not ready for production.

aditya1702 and others added 21 commits December 21, 2025 17:07
@aditya1702 @claude
Add TOID utility methods for deriving values from Total Order IDs
01b5e45 
Add constants and methods to derive ledger_number, operation_id, and tx_id
from TOID (Total Order ID) values stored in to_id fields. This enables
removing redundant columns from the database schema.

Methods added:
- Transaction.GetLedgerNumber()
- Operation.GetLedgerNumber(), GetTxID()
- StateChange.GetLedgerNumber(), GetOperationID(), HasOperation(), GetTxID()

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Add enum mappings for SMALLINT database storage
dc25439 
Add bidirectional mappings between string enum types and SMALLINT values
for efficient database storage:
- StateChangeCategory (10 values)
- StateChangeReason (18 values)
- OperationType (27 values)

Each type has ToInt16() method and FromInt16() function for conversion.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Optimize database schema for storage efficiency
f820e70 
Migration changes:
- transactions: PK changed from hash to to_id, removed ledger_number and ingested_at
- transactions_accounts: tx_hash (TEXT) -> tx_id (BIGINT)
- operations: removed tx_hash, ledger_number, ingested_at; operation_type TEXT -> SMALLINT
- operations_accounts: removed created_at
- state_changes: removed tx_hash, operation_id, ledger_number, ingested_at;
  category and reason TEXT -> SMALLINT

All removed fields can be derived from TOID (to_id) using bit operations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Remove redundant fields from Transaction, Operation, and StateChange …
f5b16d5 
…structs

Fields removed (can be derived from TOID):
- Transaction: LedgerNumber, IngestedAt
- Operation: LedgerNumber, TxHash, IngestedAt
- StateChange: LedgerNumber, OperationID, TxHash, IngestedAt

Use GetLedgerNumber(), GetOperationID(), GetTxID() methods to derive values.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update indexer processors for optimized schema
b6cdc63 
- Simplify StateChangeBuilder: set ToID directly in constructor (txID)
  and WithOperationID() now directly overrides ToID
- Remove redundant parameters from NewStateChangeBuilder (ledgerNumber, txHash)
- Update ConvertTransaction/ConvertOperation to not populate removed fields
- Remove stale GetIngestedAt() and duplicate GetLedgerNumber() methods from types.go
- Update all processor callers to use new signature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update operations.go for optimized schema
11be479 
- Remove tx_hash and ledger_number columns from BatchInsert and BatchCopy
- Change operation_type from TEXT to SMALLINT using ToInt16()

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update statechanges.go for optimized schema
db7f370 
- Remove ledger_number, operation_id, tx_hash columns from BatchInsert and BatchCopy
- Change state_change_category and state_change_reason from TEXT to SMALLINT
- Update helper function pgtypeTextFromReasonPtr -> pgtypeInt2FromReasonPtr

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Remove unused ledgerNumber variables from processors
d173970 
Since ledgerNumber is no longer passed to NewStateChangeBuilder,
remove the unused variable declarations from effects.go and sac.go.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update test utils for optimized schema
1042623 
- Remove IngestedAt comparison in assertStateChangeEqual
- Use HasOperation(), GetOperationID(), GetTxID() methods instead of
  deleted fields in assertStateChangeBase and assertFeeEvent

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702
Update transactions data layer with the changes
f4264a4
@aditya1702
graphql changes - 1
ff75fb9
@aditya1702 @claude
Add TxHash/OperationID fields to WithCursor types for JOIN-based queries
154ac94 
These fields are populated via JOINs with the transactions table, not stored
in the database. Required for dataloaders to group results by transaction hash.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update operations.go BatchGetByTxHash/BatchGetByTxHashes to use JOINs
1b8dd0e 
Since tx_hash column was removed from operations table, these queries now
JOIN with transactions table using TOID derivation: (o.id & ~4095) = t.to_id

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update BatchGetByStateChangeIDs to use TOID-based join
5a24526 
Since operation_id column was removed from state_changes table, the join
now uses to_id directly (for operation-related state changes, to_id equals
operation_id when to_id & 4095 != 0).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update BatchGetByAccountAddress to use JOINs for txHash filter
7e22ef4 
- txHash filter now JOINs with transactions table via TOID derivation
- operationID filter now uses to_id directly (to_id = operation_id for
  operation-related state changes)
- Added table alias 'sc' for proper column qualification

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update statechanges.go BatchGetByTxHash/BatchGetByTxHashes to use JOINs
e5bc0b2 
Since tx_hash column was removed from state_changes table, these queries
now JOIN with transactions table via TOID derivation: (sc.to_id & ~4095) = t.to_id

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702 @claude
Update statechanges.go BatchGetByOperationID/BatchGetByOperationIDs t…
6868d84 
…o use to_id

Since operation_id column was removed from state_changes table, these queries
now use to_id directly. For operation-related state changes, to_id equals
the operation ID (when to_id & 4095 != 0).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@aditya1702
Update graphql resolvers and schema
d3a7370
@aditya1702
Fix all tests and make check errors
3e27915
@aditya1702
fix failing tests - 1
3ae8ab9
@aditya1702
Fix failing tests - 2
dab7af0
@aditya1702 aditya1702 changed the title Statechanges data model Refactor indexer data model to not store tx_hash multiple times in different tables Dec 22, 2025
@aditya1702 aditya1702 marked this pull request as draft December 22, 2025 20:53
@aditya1702 aditya1702 marked this pull request as ready for review December 22, 2025 22:41
@aditya1702 aditya1702 closed this Jan 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@aditya1702