Issue/712 blockforst compatible api for epoch module

[ducpm2303]

#712
#768

Epoch Module

Status Checklist (per endpoint)

Endpoint	Difficulty	Data	Service Available	Status
GET /epochs/latest	[DONE]	[PARTIAL]	EpochReadService + EpochStakeStorageReader (optional active_stake)	[x]
GET /epochs/latest/parameters	[DONE]	[YES]	EpochParamService	[x]
GET /epochs/{number}	[DONE]	[PARTIAL]	EpochReadService + EpochStakeStorageReader (optional active_stake)	[x]
GET /epochs/{number}/parameters	[DONE]	[YES]	EpochParamService	[x]
GET /epochs/{number}/next	[DONE]	[PARTIAL]	BFEpochStorageReader + EpochStakeStorageReader (optional active_stake)	[x]
GET /epochs/{number}/previous	[DONE]	[PARTIAL]	BFEpochStorageReader + EpochStakeStorageReader (optional active_stake)	[x]
GET /epochs/{number}/stakes	[DONE]	[PARTIAL]	EpochStakeStorageReader (requires ledger-state/adapot)	[x]
GET /epochs/{number}/stakes/{pool_id}	[DONE]	[PARTIAL]	EpochStakeStorageReader (requires ledger-state/adapot)	[x]
GET /epochs/{number}/blocks	[DONE]	[YES]	BFEpochStorageReader	[x]
GET /epochs/{number}/blocks/{pool_id}	[DONE]	[YES]	BFEpochStorageReader	[x]

Open Issues / Notes

• Fees mismatch for epoch infor endpoint (dto.fees) —

The total_fees epoch calulated wrong somewhere and mismatch with blockforst api.
The correct fees can calculate by the first query bebow

select sum(b.total_fees) from block b where b.epoch = ?  -- correct
select e.total_fees from epoch e where e."number" = ? -- mismatch

• Block distribution query is slow. Miight need composit indexes. Example query:
  • select b.hash from block b where b.epoch = ? order by b."number" asc limit 5
• Block distribution by pool — same performance issue as block distribution.

---

Address Module

• Storage access: JOOQ-only in BFAddressStorageReaderImpl using generated tables from extensions/blockfrost/address/build/generated-src/jooq.
• Aggregation: /addresses/{address} and /extended read from address_balance_current when account aggregation is enabled; otherwise they fall back to unspent sums from address_utxo.
• Totals: /addresses/{address}/total and unspent balance use lovelace_amount for ADA and JSONB expansion for non-lovelace assets.
• Range filters: /addresses/{address}/transactions supports inclusive from/to with block[:txIndex] format.
• Pagination: Blockfrost 1-based paging is enforced in controllers (page >= 1, internal page-1).

Endpoint	Difficulty	Data	Service Available	Status
GET /addresses/{address}	[DONE]	[YES]	BFAddressService (address_balance latest per unit)	[x]
GET /addresses/{address}/extended	[DONE]	[PARTIAL]	BFAddressService (address_balance latest per unit)	[x]
GET /addresses/{address}/utxos	[DONE]	[YES]	AddressService	[x]
GET /addresses/{address}/utxos/{asset}	[DONE]	[YES]	AddressService	[x]
GET /addresses/{address}/transactions	[DONE]	[YES]	BFAddressStorageReader (address_utxo + tx_input + transaction)	Supports from/to + tx_index
GET /addresses/{address}/total	[DONE]	[YES]	BFAddressStorageReader (address_utxo + tx_input)	JSONB aggregate by unit

Open issue Notes:

• /addresses/{address}/extended currently mirrors the base response. decimals and has_nft_onchain_metadata are not available because we do not ingest CIP-68 metadata in the indexer yet (no core source of truth).
• Performance: index address_utxo(owner_addr) is required to avoid full scans. tx_input(tx_hash, output_index) should be present (PK). Add address_utxo(owner_addr_full) if you support long Byron addresses.

---

Assets Module

Implementation Notes (Design + Changes)

• Storage access: JOOQ-only in BFAssetStorageReaderImpl (asset extension) with generated jOOQ tables from core stores (assets, transaction, utxo).
• Pagination: Blockfrost 1-based paging is enforced in controller (page >= 1, internal page-1).
• Mapping: DTO conversion now uses BFAssetMapper (MapStruct), service no longer builds DTO inline.
• Blockfrost compatibility:
  • asset_name is returned in hex (derived from unit.substring(56)).
  • metadata, onchain_metadata, onchain_metadata_standard, onchain_metadata_extra are null (no token-registry/CIP metadata ingestion in store).
• GET /assets strategy (no N+1):
  1. Resolve first-seen asset units page using a boundary/prefix/candidates/enriched/ranked CTE chain:
     • first_mints — DISTINCT ON (unit) to find each asset's earliest mint slot (materialized, ~1.37M rows)
     • page_window — approximate page via OFFSET/LIMIT ordered by (slot, unit)
     • boundary — extract min/max slot from the page window
     • prefix — count rows strictly before the boundary (for offset calculation)
     • candidates — all first_mints rows within the boundary slot range
     • enriched — lateral join to assets + transaction to fetch tx_hash and tx_index (only for candidate rows, ~100-200)
     • ranked — ROW_NUMBER() ordered by (slot, tx_index, unit) matching Blockfrost ordering
     • Final SELECT filters by rank to extract the exact page
  2. Aggregate quantities only for units in the selected page.
• Query ordering: Deterministic order uses slot + tx_index + unit/tx_hash tie-breakers to match Blockfrost asc/desc behavior.
• Dialect: Asset queries rely on PostgreSQL semantics (JSONB operators, lateral JSON expansion).

Status Checklist (per endpoint)

Endpoint	Difficulty	Data	Service Available	Status
GET /assets	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/{asset}	[DONE]	[PARTIAL]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/{asset}/history	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/{asset}/txs	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/{asset}/transactions	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/{asset}/addresses	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]
GET /assets/policy/{policy_id}	[DONE]	[YES]	BFAssetService + BFAssetStorageReader	[x]

Index Script (PostgreSQL)

• Script content:

-- Blockfrost asset query index (PostgreSQL)
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_address_utxo_amounts
    ON address_utxo USING gin (amounts);

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_unit_slot
    ON assets (unit, slot) INCLUDE (tx_hash, mint_type, quantity);

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_unit_policy
    ON assets (unit, policy) INCLUDE (slot, tx_hash, quantity, asset_name, fingerprint);

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_unit_qty
    ON assets (unit) INCLUDE (quantity);

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_policy
    ON assets (policy) INCLUDE (unit, slot, tx_hash, quantity);

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_mint_unit_slot
    ON assets (unit, slot) INCLUDE (tx_hash)
    WHERE mint_type = 'MINT';

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_mint_slot_tx_hash
    ON assets (slot, tx_hash) INCLUDE (unit)
    WHERE mint_type = 'MINT';

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_transaction_tx_hash_tx_index
    ON transaction (tx_hash, tx_index);

Open Issues / Notes

• Performance: GET /assets takes ~2.5s (warm) due to first_mints CTE materializing ~1.37M rows. Recommended fix: summary table asset_first_mint for <10ms response.
• Deep paging (page very large) is still offset-bound and may degrade; current implementation minimizes scan scope but cannot fully avoid OFFSET cost.
• Metadata fields remain partial by design until token metadata ingestion is available in store.
• 2 phantom assets in Preprod cause 1-row pagination shift on deep pages (data issue, not query issue).

[Sotatek-HuyLe3a]

Block distribution query is slow. Miight need composit indexes. Example query:
select b.hash from block b where b.epoch = ? order by b."number" asc limit 5

I checked in mainnet and you're right, it' slow. Postgres used idx_block_number index, but WHERE condition is epoch filter, so postgres scanned a large number of rows.

ex: explain analyze select b.hash from block b where b.epoch = 547 order by b."number" asc limit 5;

QUERY PLAN                                                                                                                                          |
----------------------------------------------------------------------------------------------------------------------------------------------------+
Limit  (cost=0.43..598.73 rows=5 width=73) (actual time=19217.248..19217.255 rows=5.00 loops=1)                                                     |
  Buffers: shared hit=5512978 read=1986309 written=267                                                                                              |
  ->  Index Scan using idx_block_number on block b  (cost=0.43..2346266.25 rows=19608 width=73) (actual time=19217.236..19217.242 rows=5.00 loops=1)|
        Filter: (epoch = 547)                                                                                                                       |
        Rows Removed by Filter: 11678085                                                                                                            |
        Index Searches: 1                                                                                                                           |
        Buffers: shared hit=5512978 read=1986309 written=267                                                                                        |
Planning Time: 0.774 ms                                                                                                                             |
Execution Time: 19217.375 ms                                                                                                                        |

I think composite index is right choice here, should be applied after sync. Another option is rewriting query, but I tried, and the results was still the same.

[Sotatek-HuyLe3a]

Fees mismatch for epoch infor endpoint (dto.fees)

Should we create another issue for this? It seems we might have an issue in epoch-aggr.

[satran004]

Should we create another issue for this? It seems we might have an issue in epoch-aggr.

Also, adapot table has fee column which is getting calculated during epoch transition. So, we may be able to use that for all previous epochs. For current epoch, we may need to make the query in transaction table.

[ducpm2303]

Should we create another issue for this? It seems we might have an issue in epoch-aggr.

Here we are: #781

For current epoch, we may need to make the query in transaction table.

@satran004 you mean we sum all of fee in transaction table for specified epoch ?

select sum(tx.fee) from transaction tx where tx.epoch = ?

it kind of works, but I’m quite sure this query very slowly

[satran004]

@satran004 you mean we sum all of fee in transaction table for specified epoch ?

select sum(tx.fee) from transaction tx where tx.epoch = ?

Yes, it could be slow for current epoch. But in adapot, we use a similar query to calculate the last epoch’s total fee. An in-memory cache with a 20-second expiry (block time) may help reduce the load, but it’s probably not the real solution.

My concern with epoch_aggr is that I am not sure whether rollbacks are handled properly in that module. It’s a very old module and hasn’t been used before.
@Sotatek-HuyLe3a, you may want to check the logic in epoch-aggr.

One possible approach is to focus on data accuracy first rather than performance, while keeping a note of all endpoints that will need additional performance optimization later.

[sonarqubecloud]

Quality Gate passed

Issues
58 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.5% Duplication on New Code

See analysis details on SonarQube Cloud

[satran004]

Thanks @ducpm2303 . LGTM