diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..a3d4207 --- /dev/null +++ b/.env.example @@ -0,0 +1,27 @@ +# BrowserStack App Automate credentials. Copy this file to .env and +# fill in the values from `Account → Settings → Access Keys` in the +# BrowserStack dashboard. Both scripts in scripts/ read these via +# `node --env-file=.env`. +BROWSERSTACK_USERNAME= +BROWSERSTACK_ACCESS_KEY= + +# Optional escape hatch: pin builds to an existing BrowserStack +# project rather than letting BS auto-create one from the bundle ID. +# Leave unset for the normal path — the runner sends no `project` +# field and BrowserStack auto-creates a project named after the +# uploaded app's bundle ID. Only set this if your access key lacks +# project-creation permission AND you want to attach builds to a +# pre-existing project. Use the project name as it appears in the +# App Automate dashboard. +# BENCH_BROWSERSTACK_PROJECT= + +# Apple Developer Team ID (10-char identifier). Required by +# `apps/benchmark/scripts/build-ipa.sh` to archive the bench app for +# BrowserStack iOS dispatches. Visible at the top right of any page on +# https://developer.apple.com/account/. The bundle id +# `com.comapeo.core.benchmark` must already be registered as an +# Identifier under the same team. No Capabilities or App Store Connect +# entry are required — BrowserStack auto-resigns IPAs with their own +# provisioning profile on upload. +# APPLE_DEVELOPMENT_TEAM_ID=ABC1234DEF + diff --git a/.gitignore b/.gitignore index cdfe0ae..c28d88d 100644 --- a/.gitignore +++ b/.gitignore @@ -83,3 +83,19 @@ build/ # eslint .eslintcache + +# Local secrets (BrowserStack credentials etc). The .example file is +# checked in and lists the variables; copy it to .env and fill in. +.env +.env.local +.env.*.local + +# Bench result NDJSONs, written by `scripts/bench-summarize.ts` after +# pulling device logs from a BrowserStack build. Treat as build +# artifacts; copy out of this dir if you want to commit a specific +# run's results. +apps/benchmark/results/ + +# `apps/benchmark/scripts/build-ipa.sh` output (xcarchive + IPA staged +# for BrowserStack upload). Regenerable; never commit. +apps/benchmark/ios-build/ diff --git a/android/build.gradle b/android/build.gradle index 5a3a115..3df42bb 100644 --- a/android/build.gradle +++ b/android/build.gradle @@ -42,6 +42,40 @@ def comapeoAbiFilters = { return effective }() +// Override for the assets subdirectory the loader reads `index.mjs` +// from. Defaults to `nodejs-project` (the production bundle path +// emitted by `scripts/build-backend.ts`). A consumer that ships its +// own backend bundle in a sibling assets directory (e.g. +// `nodejs-bench/`) sets `comapeoBackendDir=nodejs-bench` in its +// `android/gradle.properties` and supplies the directory at +// `app/src/main/assets/nodejs-bench/` itself; AGP's normal asset merge +// places it alongside `nodejs-project/` in the APK and the loader +// reads the override path. The bench app at `apps/benchmark/` uses +// this hook via its `with-comapeo-bench` config plugin. +def comapeoBackendDir = (rootProject.findProperty('comapeoBackendDir') ?: project.findProperty('comapeoBackendDir') ?: 'nodejs-project').toString() + +// Opt-out for the keystore-backed rootkey. When set, the loader sends +// a deterministic 16-zero-byte stub on the init frame instead of +// calling `RootKeyStore.loadOrInitialize()`. Intended only for builds +// whose backend doesn't construct a `MapeoManager` and never reads +// the rootkey value (the benchmark app being the canonical case) — +// production consumers MUST leave this unset so real identity +// material is encrypted at rest. The keystore-backed path requires +// the device to have a user ECDH key (set up by the user via screen +// lock); BrowserStack's stock Samsung devices don't, which is why +// the bench app needs this hook. +def comapeoStubRootKey = (rootProject.findProperty('comapeoStubRootKey') ?: project.findProperty('comapeoStubRootKey') ?: 'false').toString() == 'true' + +// Extra argv string passed verbatim to nodejs-mobile after the +// canonical positional args (socket paths, dataDir). The loader +// splits on whitespace at runtime, so this can hold one or more +// CLI flags. Default empty; production consumers leave it unset. +// The bench app uses it to override its telemetry sink (e.g. +// `--telemetry=file:/tmp/spans.ndjson`); the bench backend's +// default sink writes spans to stdout where Android logcat picks +// them up, so an empty value is the right path for the BS sweep. +def comapeoBackendArgs = (rootProject.findProperty('comapeoBackendArgs') ?: project.findProperty('comapeoBackendArgs') ?: '').toString() + def expoModulesCorePlugin = new File(project(":expo-modules-core").projectDir.absolutePath, "ExpoModulesCorePlugin.gradle") apply from: expoModulesCorePlugin applyKotlinExpoModulesCorePlugin() @@ -76,6 +110,13 @@ android { versionCode 1 versionName "0.1.0" testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner" + // Surfaces `comapeoBackendDir` to runtime Kotlin via + // `BuildConfig.COMAPEO_BACKEND_DIR`. The Java-string literal + // form (with embedded quotes) is what `buildConfigField` + // requires. + buildConfigField "String", "COMAPEO_BACKEND_DIR", "\"${comapeoBackendDir}\"" + buildConfigField "boolean", "COMAPEO_STUB_ROOTKEY", "${comapeoStubRootKey}" + buildConfigField "String", "COMAPEO_BACKEND_ARGS", "\"${comapeoBackendArgs}\"" externalNativeBuild { cmake { cppFlags '' @@ -87,6 +128,7 @@ android { abiFilters(*comapeoAbiFilters) } } + lintOptions { abortOnError false } @@ -98,9 +140,6 @@ android { // native module instance × ABI). Both directories ship into the // same APK `lib//` segment. jniLibs.srcDirs 'libnode/bin/', 'src/main/jniLibs/' - assets { - srcDirs 'src/main/assets' - } } } externalNativeBuild { @@ -111,6 +150,9 @@ android { } buildFeatures { prefab true + // Required for `buildConfigField` above to actually generate + // the BuildConfig class; AGP 8.x flipped the default to off. + buildConfig true } packagingOptions { excludes += [ diff --git a/android/src/main/java/com/comapeo/core/NodeJSService.kt b/android/src/main/java/com/comapeo/core/NodeJSService.kt index efb5b25..fc34d93 100644 --- a/android/src/main/java/com/comapeo/core/NodeJSService.kt +++ b/android/src/main/java/com/comapeo/core/NodeJSService.kt @@ -42,7 +42,12 @@ private data class ErrorNativeMessage( const val APK_LAST_UPDATE_TIME_KEY = "apk_last_update_time" const val SHARED_PREFS_NAME_POSTFIX = "_nodejs_preferences" -const val NODEJS_PROJECT_DIRNAME = "nodejs-project" +// Asset subdirectory the loader copies into the app's filesDir and +// runs `index.mjs` from. Sourced from `BuildConfig.COMAPEO_BACKEND_DIR` +// so consumers can override via the `comapeoBackendDir` Gradle +// property (default `nodejs-project`). See android/build.gradle for +// the buildConfigField declaration. +val NODEJS_PROJECT_DIRNAME: String = BuildConfig.COMAPEO_BACKEND_DIR const val NODEJS_PROJECT_INDEX_FILENAME = "index.mjs" /** @@ -409,15 +414,31 @@ class NodeJSService( } } - val exitCode = startNodeWithArguments( - arrayOf( - "node", - jsFile.absolutePath, - comapeoSocketFile.absolutePath, - controlSocketFile.absolutePath, - dataDir, - ) + // Base argv the backend's positional parser expects: + // socket paths + dataDir. After that we append: + // - `--device=` — + // a stable device tag the bench backend uses for + // span attribution. Production backend ignores + // unknown flags so this is a no-op there. + // - whitespace-split tokens from + // `BuildConfig.COMAPEO_BACKEND_ARGS` (set by the + // `comapeoBackendArgs` Gradle property). Empty by + // default; consumers like the bench plugin set + // `--telemetry=` here when they want a + // non-default sink. + val baseArgs = arrayOf( + "node", + jsFile.absolutePath, + comapeoSocketFile.absolutePath, + controlSocketFile.absolutePath, + dataDir, ) + val deviceTag = "${android.os.Build.MANUFACTURER} ${android.os.Build.MODEL} (Android ${android.os.Build.VERSION.RELEASE})" + val extraArgs = mutableListOf("--device=$deviceTag") + if (BuildConfig.COMAPEO_BACKEND_ARGS.isNotBlank()) { + extraArgs += BuildConfig.COMAPEO_BACKEND_ARGS.trim().split("\\s+".toRegex()) + } + val exitCode = startNodeWithArguments(baseArgs + extraArgs.toTypedArray()) log("NodeJS service completed with exit code $exitCode") // Classify the exit. "Requested" means we asked for it @@ -526,7 +547,22 @@ class NodeJSService( */ private fun sendInitFrame() { val rootKeyBytes: ByteArray = try { - RootKeyStore(applicationContext).loadOrInitialize() + if (BuildConfig.COMAPEO_STUB_ROOTKEY) { + // Deterministic 16-zero-byte stub for builds that opt + // out of keystore-backed rootkey persistence (see + // android/build.gradle for the property doc and the + // bench app's `with-comapeo-bench` plugin for where + // it's set). The receiver MUST be a backend that + // doesn't construct a `MapeoManager` — the bench + // backend's relaxed init handler is the canonical + // example. Real identity material is never derived + // from this stub, so production consumers are + // protected by `comapeoStubRootKey` being false by + // default. + ByteArray(16) + } else { + RootKeyStore(applicationContext).loadOrInitialize() + } } catch (e: Exception) { Log.e(TAG, "Failed to load rootkey", e) val info = ErrorInfo("rootkey", e.message ?: e.javaClass.simpleName) diff --git a/apps/benchmark/.gitignore b/apps/benchmark/.gitignore new file mode 100644 index 0000000..f8ca347 --- /dev/null +++ b/apps/benchmark/.gitignore @@ -0,0 +1,40 @@ +# dependencies +node_modules/ + +# Expo +.expo/ +dist/ +web-build/ +expo-env.d.ts + +# Native (regenerated by `expo prebuild`; the with-comapeo-bench config +# plugin re-applies bench wiring on every prebuild, so checking these +# in would only invite drift) +*.orig.* +*.jks +*.p8 +*.p12 +*.key +*.mobileprovision + +# Metro +.metro-health-check* + +# debug +npm-debug.* +yarn-debug.* +yarn-error.* + +# macOS +.DS_Store +*.pem + +# local env files +.env*.local + +# typescript +*.tsbuildinfo + +# expo prebuild +/android +/ios diff --git a/apps/benchmark/.maestro/bench-payload-1KB.yaml b/apps/benchmark/.maestro/bench-payload-1KB.yaml new file mode 100644 index 0000000..3395ed2 --- /dev/null +++ b/apps/benchmark/.maestro/bench-payload-1KB.yaml @@ -0,0 +1,28 @@ +appId: com.comapeo.core.benchmark +--- +# Per-payload-size variant of bench-rpc.yaml: only 1KB selected. + +- launchApp: + clearState: true + +- extendedWaitUntil: + visible: + id: "service-state" + text: "STARTED" + timeout: 60000 + +- tapOn: + id: "size-64" +- tapOn: + id: "size-65536" + +- tapOn: + id: "send-button" + +- extendedWaitUntil: + visible: + id: "benchmark-result" + timeout: 60000 + +- assertVisible: + text: "1KB" diff --git a/apps/benchmark/.maestro/bench-payload-1MB.yaml b/apps/benchmark/.maestro/bench-payload-1MB.yaml new file mode 100644 index 0000000..fd02e4c --- /dev/null +++ b/apps/benchmark/.maestro/bench-payload-1MB.yaml @@ -0,0 +1,35 @@ +appId: com.comapeo.core.benchmark +--- +# Per-payload-size variant of bench-rpc.yaml: only 1MB selected. Note +# 1MB is NOT in the default selection — the flow deselects the three +# defaults and selects 1MB explicitly. Timeout is bumped because each +# round-trip moves a megabyte through the bridge. + +- launchApp: + clearState: true + +- extendedWaitUntil: + visible: + id: "service-state" + text: "STARTED" + timeout: 60000 + +- tapOn: + id: "size-64" +- tapOn: + id: "size-1024" +- tapOn: + id: "size-65536" +- tapOn: + id: "size-1048576" + +- tapOn: + id: "send-button" + +- extendedWaitUntil: + visible: + id: "benchmark-result" + timeout: 180000 + +- assertVisible: + text: "1MB" diff --git a/apps/benchmark/.maestro/bench-payload-64B.yaml b/apps/benchmark/.maestro/bench-payload-64B.yaml new file mode 100644 index 0000000..c268c17 --- /dev/null +++ b/apps/benchmark/.maestro/bench-payload-64B.yaml @@ -0,0 +1,30 @@ +appId: com.comapeo.core.benchmark +--- +# Per-payload-size variant of bench-rpc.yaml: only 64B selected. +# Default selection is {64B, 1KB, 64KB}; deselect the two larger sizes +# so the run records 64B-only RTT distribution. + +- launchApp: + clearState: true + +- extendedWaitUntil: + visible: + id: "service-state" + text: "STARTED" + timeout: 60000 + +- tapOn: + id: "size-1024" +- tapOn: + id: "size-65536" + +- tapOn: + id: "send-button" + +- extendedWaitUntil: + visible: + id: "benchmark-result" + timeout: 60000 + +- assertVisible: + text: "64B" diff --git a/apps/benchmark/.maestro/bench-payload-64KB.yaml b/apps/benchmark/.maestro/bench-payload-64KB.yaml new file mode 100644 index 0000000..dd502f7 --- /dev/null +++ b/apps/benchmark/.maestro/bench-payload-64KB.yaml @@ -0,0 +1,28 @@ +appId: com.comapeo.core.benchmark +--- +# Per-payload-size variant of bench-rpc.yaml: only 64KB selected. + +- launchApp: + clearState: true + +- extendedWaitUntil: + visible: + id: "service-state" + text: "STARTED" + timeout: 60000 + +- tapOn: + id: "size-64" +- tapOn: + id: "size-1024" + +- tapOn: + id: "send-button" + +- extendedWaitUntil: + visible: + id: "benchmark-result" + timeout: 90000 + +- assertVisible: + text: "64KB" diff --git a/apps/benchmark/.maestro/bench-rpc.yaml b/apps/benchmark/.maestro/bench-rpc.yaml new file mode 100644 index 0000000..33f37b1 --- /dev/null +++ b/apps/benchmark/.maestro/bench-rpc.yaml @@ -0,0 +1,34 @@ +appId: com.comapeo.core.benchmark +--- +# UDS / RPC bridge benchmark — bench app launches its own bench backend +# (rolled up from `apps/benchmark/backend/`, dropped into the consumer +# app's native asset tree by the `with-comapeo-bench` Expo plugin and +# loaded via the module's `comapeoBackendDir` override), waits for the +# service to reach STARTED, runs a sweep across the default payload +# sizes, and reads back the on-screen p50/p95/p99 panel. Spans are also +# written to the app's documents directory; "Export results" opens the +# system share sheet. + +- launchApp: + clearState: true + +- extendedWaitUntil: + visible: + id: "service-state" + text: "STARTED" + timeout: 60000 + +- tapOn: + id: "send-button" + +- extendedWaitUntil: + visible: + id: "benchmark-result" + timeout: 120000 + +- assertVisible: + id: "benchmark-result" +- assertVisible: + text: "p50" +- assertVisible: + text: "p99" diff --git a/apps/benchmark/.maestro/config.yaml b/apps/benchmark/.maestro/config.yaml new file mode 100644 index 0000000..6f58b65 --- /dev/null +++ b/apps/benchmark/.maestro/config.yaml @@ -0,0 +1,12 @@ +# Workspace config for the bench Maestro suite. Picked up automatically +# when Maestro runs against this directory; the BrowserStack runner zips +# everything under here into a single parent folder so BS sees the same +# layout. +# +# `flows` constrains discovery to bench-only YAMLs — keeps stray editor +# files (`.swp`, `.DS_Store`) and accidental sibling additions out of the +# uploaded zip. The runner's `bench-summarize.ts` and the BS dispatch +# script don't read this file directly; it's purely for Maestro's CLI +# `maestro test` discovery. +flows: + - "bench-*.yaml" diff --git a/apps/benchmark/App.tsx b/apps/benchmark/App.tsx new file mode 100644 index 0000000..45ccfef --- /dev/null +++ b/apps/benchmark/App.tsx @@ -0,0 +1,573 @@ +import { + unstable_messagePort, + state, + type ComapeoState, +} from "@comapeo/core-react-native"; +import { Directory, File, Paths } from "expo-file-system"; +import * as Sharing from "expo-sharing"; +import React, { useCallback, useEffect, useRef, useState } from "react"; +import { + Pressable, + ScrollView, + StyleSheet, + Text, + View, +} from "react-native"; +import { SafeAreaView } from "react-native-safe-area-context"; + +/** + * Benchmark app entry. Drives the bench RPC bridge through the same + * RN→native→Node UDS path as the production module, but talks to the + * stripped backend in `apps/benchmark/backend/` (selected via the + * module's `comapeoBackendDir` override that the + * `with-comapeo-bench` config plugin sets) — so timings isolate the + * framing / IPC / JSON-RPC bridge from `@comapeo/core` init noise. + * See `apps/benchmark/README.md` for architecture + run instructions. + * + * UI surface: + * - boot status (state observer): waits for "STARTED" before enabling + * the run button. + * - payload-size selector: subset of {64B, 1KB, 64KB, 1MB} per run. + * - "Run benchmark" button (testID="send-button"): runs warmup + + * steady-state sweep, records per-RPC RTT. + * - results panel (testID="benchmark-result"): per-size p50/p95/p99 + * over the steady-state samples. + * - "Export results" button: writes NDJSON to the app's documents + * directory and opens the system share sheet. + * + * Span transport for orchestrated runs is logcat: every recorded span + * is `console.log("BENCH_SPAN " + JSON)`'d, which lands in Android + * logcat (RN bridge tag) / iOS device console. The BS dispatch script + * pulls the device log post-build and grep's BENCH_SPAN lines into + * NDJSON. No transport plumbing on the device. + */ + +const PAYLOAD_SIZES = [64, 1024, 65536, 1048576] as const; +const DEFAULT_SELECTED: ReadonlyArray = [64, 1024, 65536]; +const WARMUP_ITERATIONS = 10; +const STEADY_ITERATIONS = 100; +const REQUEST_TIMEOUT_MS = 30_000; + +type BenchSpan = { + op: "rpc"; + name: string; + startTimestamp: number; + durationMs: number; + attrs: { bytes: number; rttSide: "rn"; device: string }; +}; + +/** + * Stable device label used as `attrs.device` on every span this run + * emits. Matches the format the native loader builds for the bench + * backend's `--device=` arg (" (Android + * )") so the summarizer can group RN-side and backend-side + * spans under one row even when each side computes its own label. + * + * Falls back to `Platform.OS` when constants are unavailable (web / + * stubbed test runners). On iOS, `Platform.constants` exposes + * `systemName` / `systemVersion` rather than Android's Brand/Model; + * we still produce a recognisable label so a future iOS run lands + * in the right RESULTS.md row without further plumbing. + */ +function deriveDeviceTag(): string { + // eslint-disable-next-line @typescript-eslint/no-require-imports + const Platform = require("react-native").Platform as { + OS: string; + constants?: Record; + }; + const c = Platform.constants ?? {}; + if (Platform.OS === "android") { + const brand = (c.Manufacturer ?? c.Brand ?? "android") as string; + const model = (c.Model ?? "device") as string; + const release = (c.Release ?? "?") as string; + return `${brand} ${model} (Android ${release})`; + } + if (Platform.OS === "ios") { + // Platform.constants on iOS exposes `osVersion` (NOT + // `systemVersion`) and `interfaceIdiom` ("phone" | "pad" | …). + // We deliberately don't try to identify the specific model + // (e.g. "iPhone 15 Pro") — that requires a native bridge or a + // package like `expo-device`, and the model match between RN + // and the backend's UIDevice.current.model would still come + // back as "iPhone" on both sides anyway. So we settle for a + // tag that exactly matches what NodeJSService.swift produces + // ("Apple iPhone (iOS 17.5)"), keeping the summarizer's + // group-by-`attrs.device` reliable. + const sysName = (c.systemName ?? "iOS") as string; + const sysVer = (c.osVersion ?? c.systemVersion ?? Platform.Version ?? "?") as string; + const idiom = String(c.interfaceIdiom ?? "phone").toLowerCase(); + const model = idiom === "pad" ? "iPad" : idiom === "tv" ? "Apple TV" : "iPhone"; + return `Apple ${model} (${sysName} ${sysVer})`; + } + return Platform.OS; +} + +const DEVICE_TAG = deriveDeviceTag(); + +type SizeStats = { + sizeBytes: number; + count: number; + p50: number; + p95: number; + p99: number; + min: number; + max: number; +}; + +type RunReport = { + runId: string; + startedAt: string; + device: { os: string; arch?: string }; + stats: SizeStats[]; + spanFile: string; +}; + +type BenchResponse = { result?: unknown; error?: { message: string } }; + +class BenchClient { + private nextId = 0; + private pending = new Map< + string, + { resolve: (r: BenchResponse) => void; timer: ReturnType } + >(); + private listenerInstalled = false; + + ensureListener() { + if (this.listenerInstalled) return; + this.listenerInstalled = true; + unstable_messagePort.addListener("message", (msg) => { + if ( + !msg || + typeof msg !== "object" || + typeof (msg as Record).id !== "string" + ) { + return; + } + const m = msg as { id: string; result?: unknown; error?: { message: string } }; + const entry = this.pending.get(m.id); + if (entry) { + clearTimeout(entry.timer); + this.pending.delete(m.id); + entry.resolve({ result: m.result, error: m.error }); + } + }); + } + + request(method: string, params?: unknown, timeoutMs = REQUEST_TIMEOUT_MS): Promise { + this.ensureListener(); + const id = `bench-${this.nextId++}`; + return new Promise((resolve) => { + // Per-request timeout so a lost frame / disconnected backend + // doesn't hang the run forever and leak the pending entry. + // Caller surfaces `error.message === "timeout"` through the same + // path as a backend-emitted error. + const timer = setTimeout(() => { + if (this.pending.delete(id)) { + resolve({ error: { message: `bench rpc timeout after ${timeoutMs}ms (method=${method})` } }); + } + }, timeoutMs); + // `unref` so the timer doesn't keep the JS runtime alive on its + // own — RN's JS thread doesn't actually exit, but it's a good + // habit and avoids surprises if this code is ported back to Node. + if (typeof (timer as unknown as { unref?: () => void }).unref === "function") { + (timer as unknown as { unref: () => void }).unref(); + } + this.pending.set(id, { resolve, timer }); + unstable_messagePort.postMessage({ id, method, params } as never); + }); + } +} + +function percentile(sortedAsc: number[], p: number): number { + if (sortedAsc.length === 0) return Number.NaN; + // Linear interpolation between closest ranks (a.k.a. the "C=1" / + // NumPy default percentile method). For p=0.5 over 100 samples this + // averages indices 49 and 50; nearest-rank would return index 49 + // alone, biasing low for small samples. Bench p99 is the usual + // outlier — `n*p=99` lands exactly on the 99th sample so weight=0. + const position = (sortedAsc.length - 1) * p; + const lower = Math.floor(position); + const upper = Math.ceil(position); + if (lower === upper) return sortedAsc[lower]!; + const weight = position - lower; + return sortedAsc[lower]! + (sortedAsc[upper]! - sortedAsc[lower]!) * weight; +} + +function summarise(samples: number[], sizeBytes: number): SizeStats { + const sorted = [...samples].sort((a, b) => a - b); + return { + sizeBytes, + count: sorted.length, + min: sorted[0] ?? Number.NaN, + max: sorted[sorted.length - 1] ?? Number.NaN, + p50: percentile(sorted, 0.5), + p95: percentile(sorted, 0.95), + p99: percentile(sorted, 0.99), + }; +} + +function formatBytes(n: number): string { + if (n >= 1024 * 1024) return `${(n / (1024 * 1024)).toFixed(0)}MB`; + if (n >= 1024) return `${(n / 1024).toFixed(0)}KB`; + return `${n}B`; +} + +export default function App() { + const [serviceState, setServiceState] = useState(state.getState()); + const [selected, setSelected] = useState>(DEFAULT_SELECTED); + const [running, setRunning] = useState(false); + const [progress, setProgress] = useState(""); + const [report, setReport] = useState(null); + + const clientRef = useRef(null); + if (!clientRef.current) clientRef.current = new BenchClient(); + + useEffect(() => { + const onChange = (next: ComapeoState) => setServiceState(next); + state.addListener("stateChange", onChange); + return () => { + state.removeListener("stateChange", onChange); + }; + }, []); + + const toggleSize = useCallback((size: number) => { + setSelected((prev) => + prev.includes(size) ? prev.filter((s) => s !== size) : [...prev, size].sort((a, b) => a - b), + ); + }, []); + + const runBench = useCallback(async () => { + if (running) return; + if (serviceState !== "STARTED") return; + if (selected.length === 0) return; + + setRunning(true); + setReport(null); + const client = clientRef.current!; + const runId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const startedAt = new Date().toISOString(); + const allSpans: BenchSpan[] = []; + const stats: SizeStats[] = []; + + try { + for (const sizeBytes of selected) { + setProgress(`warmup ${formatBytes(sizeBytes)}…`); + for (let i = 0; i < WARMUP_ITERATIONS; i++) { + // Discard timing, just prime caches. + await client.request("payload", { sizeBytes }); + } + + setProgress(`measuring ${formatBytes(sizeBytes)}…`); + const samples: number[] = []; + for (let i = 0; i < STEADY_ITERATIONS; i++) { + const start = global.performance.now(); + const startMs = Date.now(); + const { error } = await client.request("payload", { sizeBytes }); + const durationMs = global.performance.now() - start; + if (error) { + console.warn(`bench: rpc.payload error at size ${sizeBytes}:`, error.message); + continue; + } + samples.push(durationMs); + const span: BenchSpan = { + op: "rpc", + name: "rpc.payload", + startTimestamp: startMs, + durationMs, + attrs: { bytes: sizeBytes, rttSide: "rn", device: DEVICE_TAG }, + }; + allSpans.push(span); + } + stats.push(summarise(samples, sizeBytes)); + } + + // Persist the full NDJSON dump for export. + const dir = new Directory(Paths.document, "comapeo-bench"); + if (!dir.exists) dir.create({ intermediates: true }); + const file = new File(dir, `${runId}.ndjson`); + const ndjson = allSpans.map((s) => JSON.stringify({ ...s, runId })).join("\n") + "\n"; + file.create(); + file.write(ndjson); + + // Ship every span over the bench RPC socket so the backend + // re-emits them via stdout — which lands in Android logcat + // (and iOS os_log via our `dup2`+pipe redirect). Necessary + // because RN's own `console.log` is suppressed by RCTLog's + // default level filter in iOS release builds, so a direct + // RN-side console call wouldn't show up in BS device logs. + // Single batched call after measurement so per-span overhead + // doesn't contaminate the RTT samples. + try { + await client.request( + "ingestSpans", + { spans: allSpans.map((s) => ({ ...s, runId })) }, + ); + } catch (e) { + console.warn("bench: ingestSpans failed", e); + } + + setReport({ + runId, + startedAt, + device: { os: getOs() }, + stats, + spanFile: file.uri, + }); + setProgress(`done — ${allSpans.length} spans`); + } catch (e) { + console.error("bench: run failed", e); + setProgress(`error: ${e instanceof Error ? e.message : String(e)}`); + } finally { + setRunning(false); + } + }, [running, serviceState, selected]); + + const exportResults = useCallback(async () => { + if (!report) return; + try { + const available = await Sharing.isAvailableAsync(); + if (!available) { + setProgress(`file: ${report.spanFile}`); + return; + } + await Sharing.shareAsync(report.spanFile, { + mimeType: "application/x-ndjson", + dialogTitle: "Export bench results", + }); + } catch (e) { + console.warn("bench: export failed", e); + setProgress(`export error: ${e instanceof Error ? e.message : String(e)}`); + } + }, [report]); + + const canRun = serviceState === "STARTED" && !running && selected.length > 0; + + return ( + + + + UDS / RPC Bridge Benchmark + + + + + {serviceState} + + + + + + {PAYLOAD_SIZES.map((s) => { + const active = selected.includes(s); + return ( + toggleSize(s)} + style={[styles.sizeChip, active && styles.sizeChipActive]} + testID={`size-${s}`} + > + + {formatBytes(s)} + + + ); + })} + + + + + + {running ? `Running… ${progress}` : "Run benchmark"} + + + + {report && ( + + + started {report.startedAt} + + + size + n + p50 + p95 + p99 + + {report.stats.map((row) => ( + + {formatBytes(row.sizeBytes)} + {row.count} + {row.p50.toFixed(2)} + {row.p95.toFixed(2)} + {row.p99.toFixed(2)} + + ))} + + (durations in ms, RN-thread RTT) + + Export results (NDJSON) + + + {report.spanFile} + + + + )} + + + ); +} + +function Group(props: { name: string; children: React.ReactNode }) { + return ( + + {props.name} + {props.children} + + ); +} + +function Row(props: { label: string; children: React.ReactNode }) { + return ( + + {props.label} + {props.children} + + ); +} + +function getOs(): string { + // Avoid pulling in `react-native/Platform` types here — the check is + // best-effort metadata, not load-bearing logic. + // eslint-disable-next-line @typescript-eslint/no-require-imports + const Platform = require("react-native").Platform as { OS: string }; + return Platform.OS; +} + +const styles = StyleSheet.create({ + safeArea: { + flex: 1, + backgroundColor: "#eee", + }, + container: { + flex: 1, + }, + scrollContent: { + paddingBottom: 40, + }, + title: { + fontSize: 26, + margin: 20, + fontWeight: "600", + }, + group: { + margin: 12, + backgroundColor: "#fff", + borderRadius: 10, + padding: 16, + }, + groupHeader: { + fontSize: 16, + marginBottom: 10, + fontWeight: "600", + }, + row: { + flexDirection: "row", + alignItems: "center", + justifyContent: "space-between", + marginVertical: 4, + }, + rowLabel: { + color: "#666", + }, + sizeRow: { + flexDirection: "row", + flexWrap: "wrap", + }, + sizeChip: { + paddingHorizontal: 12, + paddingVertical: 8, + borderRadius: 16, + borderWidth: 1, + borderColor: "#bbb", + marginRight: 8, + marginBottom: 8, + }, + sizeChipActive: { + backgroundColor: "#0070f3", + borderColor: "#0070f3", + }, + sizeChipText: { + color: "#333", + }, + sizeChipTextActive: { + color: "#fff", + fontWeight: "600", + }, + button: { + marginHorizontal: 12, + marginVertical: 8, + backgroundColor: "#0070f3", + paddingVertical: 14, + borderRadius: 8, + alignItems: "center", + }, + buttonDisabled: { + backgroundColor: "#9bb", + }, + buttonText: { + color: "#fff", + fontWeight: "600", + fontSize: 16, + }, + table: { + marginTop: 8, + }, + tableHeaderRow: { + flexDirection: "row", + borderBottomWidth: 1, + borderColor: "#eee", + paddingBottom: 4, + marginBottom: 4, + }, + tableHeader: { + fontWeight: "600", + color: "#666", + }, + tableRow: { + flexDirection: "row", + paddingVertical: 4, + }, + tableCell: { + flex: 1, + fontVariant: ["tabular-nums"], + }, + subtle: { + color: "#888", + fontSize: 12, + marginTop: 6, + }, + exportButton: { + marginTop: 12, + paddingVertical: 10, + borderRadius: 6, + backgroundColor: "#eef4ff", + alignItems: "center", + }, + exportButtonText: { + color: "#0070f3", + fontWeight: "600", + }, +}); diff --git a/apps/benchmark/README.md b/apps/benchmark/README.md new file mode 100644 index 0000000..3ca808e --- /dev/null +++ b/apps/benchmark/README.md @@ -0,0 +1,236 @@ +# core-react-native-benchmark + +Measures the `@comapeo/core-react-native` UDS / RPC bridge — boot +phases plus per-payload-size RPC round-trip latency — driving the +real RN→native→nodejs-mobile path with `@comapeo/core` stripped out +so framing / IPC / RPC regressions surface without core noise. + +## What it measures + +- **Boot phases.** `boot.listen-control`, `boot.init`, and + `boot.construct` server-side spans, recorded via a configurable + telemetry sink. (Three more native-side phases — + `ipc-connect (control)`, `rootkey-load`, `ipc-connect (comapeo)` — + will be added when the production loader adopts the same + instrumentation.) +- **RPC round-trip latency** at four payload sizes (64 B / 1 KB / + 64 KB / 1 MB), 10 warmup + 100 steady-state iterations per size. + RN-thread RTT is recorded per request alongside server-side handler + duration so end-to-end vs. server-only timing can be diffed. + +## How it's architected + +The bench app drops a stripped backend bundle into the consumer app's +own native asset tree and tells the module's loader to read from +there. The module sees no bench-specific code: + +- **Module-side override hooks.** `@comapeo/core-react-native` exposes + two paired overrides for non-production consumers: + - `comapeoBackendDir` — Gradle property → + `BuildConfig.COMAPEO_BACKEND_DIR` on Android; `ComapeoBackendDir` + Info.plist key on iOS. Defaults to `nodejs-project` (the + production bundle); `NodeJSService.kt` and + `AppLifecycleDelegate.swift` read it to choose the bundle subdir. + - `comapeoStubRootKey` — Gradle property → + `BuildConfig.COMAPEO_STUB_ROOTKEY` on Android; `ComapeoStubRootKey` + Info.plist key on iOS. Defaults to false. When true, the loader + sends a 16-zero-byte stub on the init frame instead of touching + the keystore/keychain. Required on devices without a configured + screen lock (BrowserStack's stock fleet falls in this bucket — + Android's super-encryption layer fails when + `setUnlockedDeviceRequired(true)` meets a missing user ECDH + key). Production consumers MUST leave this false. +- **Bench plugin.** `plugins/with-comapeo-bench/` is an Expo config + plugin that (a) sets both overrides above, (b) copies the + rolled-up bench bundle from `backend/dist/` into the consumer app's + own native asset tree at prebuild time — + `android/app/src/main/assets/nodejs-bench/` on Android, an Xcode + folder reference under `.app/nodejs-bench/` on iOS. +- **Bench backend.** `backend/index.js` reuses the production + state machine (`pre-listening` → `started` → `ready`) and + path-imports the framing helpers (`server-helper.js`, + `simple-rpc.js`, `message-port.js`) from the module's production + `backend/lib/` so the wire framing is bit-identical to production. + `BenchRpcServer` (`backend/lib/bench-rpc.js`) registers `echo`, + `payload(sizeBytes)`, and `ingestSpans` methods. Telemetry sinks + (`backend/lib/telemetry-sink.js`) are configurable via + `--telemetry=` on argv: `log` (default; one stdout line per + span so logcat / device console captures them), `noop`, or + `file:`. +- **RN side.** `App.tsx` uses `unstable_messagePort` from + `@comapeo/core-react-native` — a generic escape hatch one level + below the public `comapeo` `MapeoClient` — to send raw frames over + the same JSI → native UDS path real users hit. The bench-specific + request/response schema (`{id, method, params}` vs production's + `{id, jsonrpc, ...}`) means the production RPC machinery treats + bench frames as unknown and ignores them. + +``` +React Native (App.tsx) + │ postMessage({ id, method, params }) + ▼ +unstable_messagePort ← @comapeo/core-react-native + │ + ▼ +JSI bridge → native module → Unix-domain socket pair + │ + ▼ +nodejs-mobile (backend/index.js) + │ + ▼ +BenchRpcServer.dispatch → echo / payload(sizeBytes) +``` + +## Run it + +Prerequisites: Xcode (for iOS) / Android SDK, Node 24, an +iOS simulator or Android emulator booted. + +```bash +cd apps/benchmark +npm install +npm run ios # or: npm run android +``` + +Each platform script runs `prebuild:bundle` first (installs bench +backend deps + rolls up `dist/index.mjs`) and then invokes +`expo run:`. After the app launches, wait for +`Backend → state` to read **STARTED**, optionally toggle payload +sizes, then tap **Run benchmark**. + +Per-size p50 / p95 / p99 render on screen. The full per-RPC NDJSON +is written to the app's Documents directory; tap **Export results** +to share via the system share sheet (iOS) or reveal the path +(Android). + +If you've previously generated `android/` or `ios/` and want a clean +prebuild: + +```bash +rm -rf android ios && npm run prebuild +``` + +## Maestro flows + +```bash +maestro test apps/benchmark/.maestro/bench-rpc.yaml # all sizes +maestro test apps/benchmark/.maestro/bench-payload-64B.yaml +maestro test apps/benchmark/.maestro/bench-payload-1KB.yaml +maestro test apps/benchmark/.maestro/bench-payload-64KB.yaml +maestro test apps/benchmark/.maestro/bench-payload-1MB.yaml +``` + +Each flow launches the app, asserts `STARTED`, taps `send-button`, +and asserts the `benchmark-result` panel renders. The `config.yaml` +in the same directory constrains Maestro CLI to the `bench-*.yaml` +discoveries. + +To target a specific simulator/emulator when several are booted: + +```bash +maestro --device test apps/benchmark/.maestro/bench-rpc.yaml +``` + +## Span transport: logcat + +Per-RPC spans are emitted as `BENCH_SPAN ` lines on `console.log` +— from RN's bridge (App.tsx) and from nodejs-mobile (the bench +backend's default `LogSink`). Both surface in Android `logcat` (under +their respective tags) and iOS device console. + +For local standalone runs, watch them with: + +```bash +adb logcat | grep BENCH_SPAN +``` + +Or rely on the on-device `JsonFileSink` that writes the same data to +`/Documents/comapeo-bench/.ndjson` — exportable +via the **Export results** button in the UI. + +For BrowserStack runs, `scripts/run-on-browserstack.ts` pulls each +device's logcat after the build finishes, greps `BENCH_SPAN` lines, +and writes one NDJSON file per device into +`apps/benchmark/results/`. No tunnel, no receiver. (See "Run on +BrowserStack" below.) + +## Phases + +- ✅ **Phase 1–2:** shared sink + bench backend + dual-bundle build + wiring (now: generic `comapeoBackendDir` override + bench-app config + plugin). +- ✅ **Phase 3:** bench app UI, RPC bridge wiring, on-device + p50/p95/p99 render, "Export results", config plugin, Maestro flows. +- ✅ **Phase 4:** BrowserStack App Automate — log-based span pull + across a curated 10-device sweep, auto-batched against plan + capacity, Test R&A integration via `customBuildName` + + `buildIdentifier`. +- ⏳ **Phase 5:** `SentryAdapterSink` once the Sentry plan adopts the + shared instrumentation; bench call sites stay the same. + +## Run on BrowserStack + +### One-time setup + +1. BrowserStack App Automate account, RBAC role with `create:build` + permission. Username + access key from `Account → Settings → + Access Keys`. +2. Copy `.env.example` (repo root) to `.env` and fill in + `BROWSERSTACK_USERNAME` / `BROWSERSTACK_ACCESS_KEY`. If your + account can't auto-create projects, also set + `BENCH_BROWSERSTACK_PROJECT` to an existing project name. + +### Per-run workflow + +```bash +# 1a. Android: build a release APK with the JS bundle embedded. +cd apps/benchmark +npm run prebuild:bundle +cd android && ./gradlew :app:assembleRelease +cd ../../.. + +# 1b. iOS: build a Development-export IPA. Requires +# APPLE_DEVELOPMENT_TEAM_ID in .env (10-char team id) and the +# bundle id com.comapeo.core.benchmark registered in your team's +# Identifiers. BrowserStack auto-resigns on upload, so a +# Development export (not Distribution / App Store) is enough. +cd apps/benchmark && npm run ios:archive && cd ../.. + +# 2. Dispatch — defaults to the curated 10-device Android sweep, +# auto-batches against plan capacity (5+5 → fits in one build). +npm run bench:browserstack -- \ + --app-android apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \ + --app-ios apps/benchmark/ios-build/ipa/corereactnativebenchmark.ipa +# Optional flags: +# --devices-android "" override Android device list +# --devices-ios "" override iOS device list +# --build-tag