You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/docs/hark/architecture.mdx
+45-3Lines changed: 45 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,13 +8,55 @@ Hark is the reference OACP voice assistant. It discovers app capabilities at run
8
8
9
9
## Discovery
10
10
11
-
`OacpDiscoveryHandler.kt` scans for exported `.oacp``ContentProvider`s via `PackageManager`. For each provider it reads two paths:
11
+
Discovery logic lives in `HarkPlatformPlugin.kt` inside the `hark_platform` plugin package. It scans for exported `.oacp``ContentProvider`s via `PackageManager`. For each provider it reads two paths:
12
12
13
13
-`/manifest` - the machine-readable `oacp.json` capability file.
14
14
-`/context` - the LLM-readable `OACP.md` semantic context file.
15
15
16
16
`OACP.md` is validated for presence but **not currently consumed by the LLM**. It is reserved for future BYOK (bring-your-own-key) cloud models that have larger context windows.
17
17
18
+
## Platform plugin
19
+
20
+
`hark_platform` is a Flutter plugin that replaces all raw `MethodChannel` code with type-safe Dart/Kotlin bindings generated by [Pigeon](https://pub.dev/packages/pigeon).
21
+
22
+
A single Pigeon schema defines three API surfaces:
23
+
24
+
| API | Direction | Purpose |
25
+
|-----|-----------|---------|
26
+
|`HarkCommonApi`| Dart -> Kotlin | Shared operations used by both engines (discovery, intent dispatch, result listening) |
Pigeon generates matching Dart classes and Kotlin host implementations from this schema. The plugin auto-registers on every `FlutterEngine` that attaches to it, so both the main engine and the overlay engine receive their bindings without manual setup.
31
+
32
+
## Overlay architecture
33
+
34
+
Hark uses Android's `FlutterEngineGroup` to run two Flutter engines that share a single GPU context and Dart VM:
35
+
36
+
-**Main engine** powers the full app (model loading, STT, NLU, TTS, discovery, dispatch).
37
+
-**Overlay engine** powers a thin UI shell that loads zero models and starts instantly.
38
+
39
+
When the user triggers the assist gesture, Android launches `OverlayActivity`, a dedicated translucent `FlutterActivity` bound to the overlay engine. Because the overlay loads no models, it appears in under 100ms.
40
+
41
+
### State sync
42
+
43
+
The overlay cannot run inference on its own. All heavy work stays on the main engine, and state flows between engines through the native layer:
`HarkOverlayFlutterApi` pushes transcript updates, NLU results, and TTS state into the overlay. `HarkMainFlutterApi` pushes user input events from the overlay into the main engine. The native `OverlayActivity` acts as the relay point, forwarding calls between the two engines.
51
+
52
+
### Session lifecycle
53
+
54
+
1. Assist gesture fires. Android creates (or reuses) the overlay engine and launches `OverlayActivity`.
55
+
2. Overlay sends a session-start signal through `HarkOverlayApi`.
56
+
3. Main engine begins listening via STT, runs NLU, dispatches the intent, and streams status updates back through `HarkOverlayFlutterApi`.
57
+
4. The overlay renders chat bubbles with app icons, a keyboard/mic toggle, and auto-activates the mic on open.
58
+
5. On dismiss, the overlay engine is kept warm for the next invocation.
@@ -42,7 +84,7 @@ Every dispatch includes `org.oacp.extra.REQUEST_ID` for result correlation.
42
84
43
85
```
44
86
OacpResultReceiver (Kotlin)
45
-
-> EventChannel (platform channel)
87
+
-> HarkPlatformPlugin (Pigeon binding)
46
88
-> OacpResultService (Dart)
47
89
-> AssistantScreen (chat bubble + TTS)
48
90
```
@@ -72,7 +114,7 @@ Android validates all of these before allowing an app to be selected as the defa
72
114
|-----------|------|---------|
73
115
|`HarkVoiceInteractionService`| Kotlin | Background service Android binds to. Must declare `BIND_VOICE_INTERACTION` permission. |
74
116
|`HarkSessionService`| Kotlin | Creates `HarkSession` instances when assistant is invoked. |
75
-
|`HarkSession`| Kotlin |`onShow()` launches MainActivity with `EXTRA_LAUNCHED_FROM_ASSIST`. |
117
+
|`HarkSession`| Kotlin |`onShow()` launches `OverlayActivity` with `EXTRA_LAUNCHED_FROM_ASSIST`. Falls back to `MainActivity` if the overlay engine is unavailable. |
76
118
|`HarkRecognitionService`| Kotlin | Stub `RecognitionService` - required by Android to qualify as assistant. Actual STT handled by Flutter's `speech_to_text`. |
77
119
|`voice_interaction_service.xml`|`res/xml/`| Metadata referencing sessionService, recognitionService, and `supportsAssist="true"`. |
78
120
|`ACTION_ASSIST` intent filter | Manifest | On MainActivity - required for assistant role qualification. |
Copy file name to clipboardExpand all lines: content/docs/hark/overview.mdx
+11-1Lines changed: 11 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,6 +39,16 @@ The shipped local pipeline has two stages:
39
39
40
40
This keeps the matching step cheap and only sends the selected action schema to the generative model. For the full reasoning, see [NLU architecture](/docs/hark/nlu-architecture).
41
41
42
+
## Assistant overlay
43
+
44
+
Long-pressing the home button launches a lightweight overlay panel that floats on top of whatever app you are using. This is the fastest way to interact with Hark because the overlay loads zero AI models on its own. It is a thin UI shell: instant startup, no lag.
45
+
46
+
Under the hood, Hark uses `FlutterEngineGroup` to spin up two Flutter engines. The main engine runs the heavy lifting (STT, NLU, TTS, model inference). The overlay engine only renders the chat UI. State is relayed between the two engines through a Pigeon-generated bridge that passes through the native `OverlayActivity` on Android.
47
+
48
+
The overlay shows chat bubbles decorated with the target app's icon, and you can toggle between keyboard and microphone input. By default, the mic starts listening automatically when the overlay opens. If you want more space, tap "Open full app" to continue the same conversation in the full Hark chat screen with no context lost.
49
+
50
+
The session persists when you switch between apps (the overlay stays on screen). Pressing back dismisses the overlay and resets the session.
51
+
42
52
## OACP-enabled apps
43
53
44
54
Hark works with any app that implements OACP. These are tested and working today:
@@ -82,4 +92,4 @@ flutter run
82
92
83
93
## Roadmap
84
94
85
-
The short version: the protocol and Kotlin SDK ship today, and Hark is focused next on self-hosted inference, better speech input, and a lighter assistant UI. See the [roadmap](/docs/roadmap) for the tracked priorities.
95
+
The short version: the protocol and Kotlin SDK ship today, and the lightweight assistant overlay is shipped. Hark is focused next on self-hosted inference, better speech input, wake word activation, and chat persistence. See the [roadmap](/docs/roadmap) for the tracked priorities.
0 commit comments