Settle payment rails in order of last settlement time

[pyropy]

In #353, I proposed a different design that does not require multiple concurrent transactions:

1. For each data set, keep a column in our DB storing the time of the last successful attempt to settle the linked payment rail. For new data sets, set this value to the time when the data set was created.
2. Periodically pick a data set with the oldest time of the last rail settlement. Ignore data sets that were settled less than X days ago, where X is the interval of how often we want to settle. I propose 1 day for calibnet and 1 week for mainnet.
3. Settle the payment rail for this data set.
4. If the TX is successful, then update our DB with the new time of the last successful settlement.
5. If the TX failed - TODO, we need to figure out how to handle this.
6. If the outer TX takes too long to get accepted by the network because of the elevated gas base fee. We need to replace the TX with a new TX using the same nonce. The new TX does not need to be the same.

Since the time I wrote my comment, we have moved to settling CDN payment rails in batches. The steps above can remain mostly the same with a slight tweak - we can settle up to N oldest data sets in a single transaction.

Originally posted by @bajtos in #353

[FAIZ1409]

Hi ... I would like to would like to work on this issue

[bajtos]

@FAIZ1409, this task is a bit complex; it's not a good candidate for the first contribution. Before we assign it to you, can you please post a high-level plan of the implementation?

[Chaitu-Tatipamula]

hey i was looking into this ticket and and found that most of the infra already exists

• cdn_payments_settled_until column is already in the DB
• the indexer already handles the timestamp updates on success when it receives on-chain
• transaction retying is also haldled by TransactionMonitorWorkflow

but to make this possible i am thinking to

• add SETTLEMENT_INTERVAL_DAYS config in wrangler.toml with calibration being set to 1 day and mainnet to 7 days

• modify the getDataSetsForSettlement() query in rail-settlement.js by ordering it oldest first and a filter to skip datasets settled within the interval

• and finally add tests for ordering and interval filtering to ensure everything is working as intended

wdyt? can you assign it to me? @pyropy @bajtos if there is something i am missing please let me know!

[bajtos]

@Chaitu-Tatipamula Nice! I agree that with my #598 in place, this is easier to implement than before.

The missing piece in your description is to look only at the data sets where some new usage was reported since the last time we settled the payment. Otherwise you will eventually end up settling the same set of payment rails with no new usage.

Unfortunately, we discovered a major problem. There is a race condition between reporting usage and setting payments. As a result, when the user exhausts their pre-paid egress quota, it's possible to end up in a state where there is some (over) usage reported since the last settlement, but no funds are left, and therefore this over-usage won't be settled until the user tops up, which may not happen.

We are discussing how to best fix this problem. Until then, this issue cannot be worked on.

[bajtos]

I think the following pull request is the last missing bit: #613