Position reconciliation: detect TP/SL/liquidation fills from Binance and sync local DB

[fray-cloud]

Problem Statement

PR3 + #97에서 사용자는 LLM 시그널로 Binance Futures 포지션을 열고 TP/SL을 algoOrder로 부착할 수 있게 됐고, 수동 종료 버튼도 추가됐다. 그런데 TP 또는 SL이 거래소에서 발동해 포지션이 자동 종료된 경우, 우리 시스템은 이 종료 이벤트를 전혀 인지하지 못한다:

• Binance 측에서는 algoOrder 트리거 → reduceOnly market 주문 fill → 포지션 0 → 정상 종료
• 우리 워커는 Kafka trading.order.requested 만 처리하고 있어서, Binance가 우리에게 능동적으로 알려주는 채널이 없음 (User Data Stream 미사용)
• 그 결과 DB는 status='filled', closedAt=null 상태로 영구히 "열려있는 포지션"처럼 남음
• 사용자가 대시보드/포트폴리오/주문 상세를 보면 이미 끝난 포지션이 활성으로 보이고, 미실현 PnL이 계속 갱신되며, 실현 손익(realizedPnl) 도 0인 채 유지됨
• 유일한 복구 수단은 사용자가 수동으로 "포지션 종료" 버튼을 한 번 더 누르는 것 (UI/UX overhaul: testnet/mainnet split, USD/KRW toggle, /orders/[id], dashboard rebuild #97 reconcile 경로). 이건 운영상 허용 불가 — 자동화가 핵심 가치인데 사후 정리를 사람이 해야 함

다음 시나리오 모두에서 같은 문제가 발생한다:

1. TP 트리거로 익절 청산
2. SL 트리거로 손절 청산
3. 강제 청산 (liquidation)
4. 사용자가 Binance 앱/웹에서 직접 닫음 (우리를 통하지 않음)
5. 다른 reduceOnly 주문이 포지션을 깎아서 0으로 만듦

Solution

워커 서비스에 Position Reconciler 백그라운드 컴포넌트를 추가해 30초 주기로 미종료 실거래 주문을 polling하면서 Binance 실제 상태와 동기화한다. 단계별:

1. DB에서 status='filled' AND closedAt IS NULL AND mode='real' 인 모든 주문 조회 (보통 사용자당 0~5건이라 가벼움)
2. 각 주문에 대해 거래소 API로 현재 포지션 상태 조회 (positionRisk)
3. 포지션이 사라졌으면 → 해당 시점 이후의 income 기록(REALIZED_PNL 트랜잭션)을 1회 조회해서 정확한 실현 손익 합산 → DB를 status='closed', closedAt, realizedPnl 로 업데이트
4. 닫힌 사유(TP fill / SL fill / 수동 / 청산)를 가능한 한 추론해서 알림 메시지에 포함
5. 알림 발송 (Telegram + Web Push) + WebSocket 브로드캐스트로 UI 즉시 갱신

이로써 사용자가 별도 액션을 하지 않아도 자동으로 포지션 상태가 정리되고, 실현 손익이 정확히 채워지며, 대시보드/포트폴리오의 카운트가 즉시 업데이트된다.

User Stories

1. As a 트레이더, I want my position to be marked closed in the local app within ~30 seconds of Binance's TP firing, so that the dashboard's "active positions" count is accurate without me having to act.
2. As a 트레이더, I want my position to be marked closed within ~30 seconds of Binance's SL firing, so that I'm not misled into thinking I still have exposure.
3. As a 트레이더 whose position got liquidated, I want the order to be marked closed with the liquidation outcome flagged, so that I can review the loss without confusion.
4. As a 트레이더 who closed a position directly on Binance's app, I want the local app to detect that and reconcile, so that there's a single source of truth.
5. As a 트레이더, I want the realizedPnl on a TP-closed order to reflect Binance's actual settled PnL (including funding fees and trading fees), so that my realized total is accurate to the cent.
6. As a 트레이더, I want to receive a Telegram (and Web Push) notification when TP fires automatically, so that I know my profit was taken.
7. As a 트레이더, I want to receive a Telegram (and Web Push) notification when SL fires automatically, so that I'm alerted to the loss.
8. As a 트레이더, I want the dashboard's "오늘 실현 손익" card to refresh automatically after a TP/SL fires, so that I see the up-to-date PnL.
9. As a 트레이더, I want the order detail page (/orders/[id]) to update from "open" to "closed" automatically when reconciliation completes, so that the Close button disappears and a closed badge appears.
10. As a 트레이더, I want the activity feed to show the auto-close event (with reason: TP / SL / Liquidation / Manual on exchange), so that I have a complete history.
11. As a 트레이더, I want the reconciler to never double-process the same fill, so that my realized PnL doesn't get doubled.
12. As a 트레이더, I want the reconciler to keep working even if Binance API is temporarily down, so that orders eventually settle when API recovers.
13. As an 운영자, I want the reconciler to log every reconcile attempt with order id and outcome, so that I can debug "why is this position still open" issues from logs.
14. As an 운영자, I want the reconciler to respect Binance's rate limits, so that polling doesn't get our IP/keys throttled or banned.
15. As a 트레이더 with both testnet and mainnet keys, I want the reconciler to work for both networks independently, so that practice positions also reconcile correctly.
16. As a 트레이더, I want the reconciler not to interfere with my manual close action, so that pressing 종료 still works without race conditions.
17. As an 운영자, I want a reconcile attempt failure (e.g., expired API key) not to block reconciliation of unrelated orders, so that one bad key doesn't stall everyone.
18. As a 트레이더, I want orders that have been open for an unreasonably long time (e.g., >7 days) without any movement to still be considered for reconcile, so that abandoned orders don't accumulate.
19. As an 운영자, I want the reconciler interval to be configurable via env var, so that I can tune cadence in staging vs production without redeploy.
20. As a 트레이더, I don't want the reconciler to incorrectly mark an order closed when the position is actually still open and just had its TP/SL price re-attached, so that my open trades stay open.

Implementation Decisions

Architecture

• New service: PositionReconcilerService in worker-service. Runs as a singleton with a setInterval-driven loop (matching ExchangesService exchange-rate polling pattern). No @nestjs/schedule dependency added.
• Default interval: 30 seconds, overridable via RECONCILE_INTERVAL_MS env var.
• Reconciler is order-driven, not user-driven — it queries the DB for all status='filled' AND closedAt IS NULL AND mode='real' orders and processes them. Multi-tenancy comes for free.
• Per-iteration concurrency: process orders in a loop with bounded parallelism (e.g., Promise.all over chunks of ≤5). Avoids hammering Binance with hundreds of parallel signed calls if many orders are open.
• Binance signed-request weight: each iteration is ≤ 2 × N_open_orders weight. With 50 open orders this is 100 weight per 30s — well under the 2400/min futures limit. We log a warning at 1000 weight/min.

Reconcile algorithm (per order)

1. Decrypt user's exchange key once per user (cache for the iteration).
2. Call getPosition(symbol). If position exists with non-zero quantity, the order is still open → skip.
3. If position is gone (or quantity === 0):
   a. Call getIncome(symbol, startTime=order.createdAt, endTime=now) to fetch all realized-pnl/funding-fee/commission records for that symbol since the order opened. Filter to those whose tradeId ties back to our entry's exit (best-effort) or summed within the window.
   b. Compute realizedPnl = Σ income.income over relevant rows. If income endpoint returns nothing (rare, can lag), fall back to estimate: (markPrice - entryPrice) × quantity × (long?+1:-1), then re-poll on next cycle to refine.
   c. Determine close reason heuristic:
   • If a REALIZED_PNL row exists with order id = TP algo order id → reason: take_profit
   • Else if = SL algo order id → reason: stop_loss
   • Else if Binance returns LIQUIDATION income type → reason: liquidation
   • Else → reason: manual_or_unknown
     d. Atomically update DB: status='closed', closedAt, realizedPnl, closeReason.
     e. Emit Kafka events: trading.order.result (for WebSocket fan-out / activity feed) and notification.send (Telegram + Web Push, body includes reason).
4. Idempotency: check closedAt again right before update inside a transaction; if already set, skip emit. Use a Redis lock keyed by reconcile:order:<id> with short TTL (e.g., 60s) so manual close (UI/UX overhaul: testnet/mainnet split, USD/KRW toggle, /orders/[id], dashboard rebuild #97) and reconciler can't both process the same order.

Schema

• New optional column on Order: closeReason: String? with values 'take_profit' | 'stop_loss' | 'liquidation' | 'manual' | 'manual_on_exchange' | 'reconciled_unknown'. Backfill = null for historical rows.
• Reuses existing fields: closedAt, realizedPnl, tpOrderId, slOrderId, entryPrice, filledQuantity.

Exchange adapter additions

• New method on IExchangeRest: getIncome(credentials, opts: { symbol?, startTime, endTime, incomeType? }): Promise<IncomeRecord[]>
• IncomeRecord = { tradeId?, orderId?, symbol, incomeType, income, asset, time }
• Backed by Binance Futures /fapi/v1/income. Returns up to 1000 records per call; reconciler pagination not needed within typical 30s window.

API contract changes

• GET /orders/:id response gains optional closeReason field on the embedded order object.
• GET /dashboard/summary and GET /portfolio/summary already aggregate realizedPnl, no shape change.
• WebSocket order:updated payload gains closeReason so UI can show "TP 익절" / "SL 손절" / "청산" badges.

Frontend changes

• /orders/[id] and /dashboard rely on existing 5-10s refetchInterval, so reconciler-driven changes appear within one refresh cycle — no architecture change needed.
• New badge variant on order detail page header: maps closeReason → user-facing label (익절 / 손절 / 청산 / 수동 / 정리됨).
• Activity feed: enrich order item description with reason when present.

Operational

• Reconciler logs structured events: { orderId, userId, symbol, action: 'skip'|'close'|'error', reason?, latencyMs }. Pino JSON output is ingested by existing log pipeline.
• Failure of one order's reconcile must NOT abort the iteration; wrap each order in its own try/catch.
• A user with an expired/revoked API key produces repeating errors — after 3 consecutive auth failures we emit a notification.send event suggesting key rotation, and skip that user's orders for 1 hour (Redis cooldown key).

Testing Decisions

External-behavior testing only; no asserting on internal call sequences.

• PositionReconcilerService.runOnce() unit (Vitest, mocked Prisma + mocked adapter):
  • Open orders with live position → no DB write, no Kafka emit
  • Open orders with vanished position + matching REALIZED_PNL income from TP order id → DB updated to closed, closeReason='take_profit', realizedPnl summed correctly
  • Same scenario but matching SL order id → closeReason='stop_loss'
  • Income includes LIQUIDATION row → closeReason='liquidation'
  • Position gone but income endpoint returns empty → DB updated with estimated realizedPnl, closeReason='reconciled_unknown'
  • Adapter throws auth error → that order is skipped, others still processed; auth-failure counter increments
  • Concurrent run with manual close (UI/UX overhaul: testnet/mainnet split, USD/KRW toggle, /orders/[id], dashboard rebuild #97 path) → idempotency lock prevents double update (only one Kafka emit observed)
• getIncome adapter unit: signed-request URL contains correct query params; response array is mapped to IncomeRecord[] shape.
• Worker integration smoke (existing test pattern): mock Binance HTTP; start reconciler with 100ms interval; insert a fixture order with no live position; assert DB row transitions to closed within 1s.
• Idempotency end-to-end: simulate two reconciler ticks back-to-back on the same closed order — second tick is a no-op (verified by Kafka producer mock not being called the second time).

Prior art:

• apps/worker-service/src/exchanges/exchanges.service.ts for setInterval lifecycle
• apps/api-server/src/orders/sagas/saga-timeout.watchdog.ts for periodic worker pattern
• apps/api-server/src/portfolio/portfolio.service.test.ts for mocked-adapter tests with the FakeBinanceRest pattern
• apps/worker-service/src/orders/sagas/close-position-saga.test.ts for hoisted-mock idiom

Out of Scope

• Binance User Data Stream (WebSocket push for ORDER_TRADE_UPDATE). This would deliver fills in <1s instead of ≤30s, but adds listenKey lifecycle, reconnect-on-disconnect, key-rotation logic, and a parallel Kafka producer pipeline. Defer to a follow-up PRD; the polling reconciler is the reliable safety net regardless of whether WS is added later.
• Reconciliation for spot orders — out of scope because PR1 dropped spot trading; only futures positions exist now.
• Reconciliation for paper orders — paper mode was retired; not applicable.
• Backfilling historical orders that are stuck in the DB — covered indirectly because the reconciler will pick them up on next tick. No special migration.
• Automatic re-attaching TP/SL to a position that lost them somehow — out of scope; reconciler only observes, never re-arms.
• Exchange-side audit (matching every Binance trade to a local order) — that's a deeper finance/accounting feature; reconciler only handles open→closed transitions.
• Multi-exchange reconciler — Binance only for now (matches the rest of the platform).
• Configurable per-user polling cadence — single global interval is enough.

Further Notes

• The 30-second cadence + Binance income endpoint gives us authoritative realized PnL including the funding fee and commission components that the simple (mark - entry) × qty math misses. This is the right tradeoff for accuracy.
• The new closeReason column is the smallest schema delta needed and is forward-compatible with adding a User Data Stream consumer later (the consumer would just write the same column).
• The reconciler must not race with UI/UX overhaul: testnet/mainnet split, USD/KRW toggle, /orders/[id], dashboard rebuild #97's manual-close path. The shared Redis lock key (reconcile:order:<id>) is the synchronization primitive; both writers acquire it before updating DB. Manual close already acquires saga:close-lock:<requestId> — this PRD adds an order-scoped lock as well.
• The income endpoint can lag by a few seconds after a fill. The reconciler tolerates this by running again 30s later and overwriting an estimated realizedPnl with the authoritative value if the previous tick wrote an estimate. To make this safe we require the second update to ONLY happen when realizedPnl was an estimate (a small realizedPnlSource field, or just "if income now returns data and our row was set without a tradeId reference, refine it once"). Keep the implementation simple: only refine within a 5-minute window after the close.
• Once this lands, UI/UX overhaul: testnet/mainnet split, USD/KRW toggle, /orders/[id], dashboard rebuild #97's manual close button can be left as-is for the user's own safety/UX (lets them act faster than the next 30s tick), but the system is no longer dependent on that button for correctness.