CarPlay-Aware Assist Triggering

[lowlyocean]

Summary

Enables the Assist widget to trigger voice recording functionality within the CarPlay interface using a "Signal/Observe" pattern via NSUserActivity. The implementation ensures that the intent remains functional for mobile users by correctly targeting the main web view scene when a CarPlay session is not present.

High-Level Overview

The Problem
Users want to trigger the "Assist" feature (voice interaction with Home Assistant) directly from a Home Assistant widget on their iPhone lock screen. Currently, when a user interacts with the widget, the app needs to decide where to show the Assist interface: on the phone's main screen (the web view) or on the car's display (CarPlay). Without a way to distinguish between these two, the app might try to open a mobile popup while the user is driving, which is not only useless but potentially distracting.

The Solution
We implemented a "smart trigger" system. When the widget is pressed, the app now checks: "Is the user currently using CarPlay?"

• If YES (CarPlay): It sends a silent signal to the car's interface to launch a dedicated, safe, and simple voice control screen.
• If NO (Mobile): It launches the standard Assist interface directly within the main app on the phone.

Architectural Overview

We moved from a "one-size-fits-all" intent to a Signal/Observe pattern. Instead of the widget telling the app exactly what to do, it now sends a signal with a payload, and the relevant part of the app observes that signal and decides how to react based on the current context.

Workflow:

1. Widget Interaction: User taps the widget.
2. Context Check: AssistAppIntent checks if a CarPlay scene is active.
3. Path A (Mobile): If no CarPlay scene, the intent uses the webViewWindowControllerPromise to call showAssist directly on the mobile WebViewExternalMessageHandler.
4. Path B (CarPlay): If a CarPlay scene is active, the intent uses activateAnyScene to send an NSUserActivity with a userInfo payload.
5. CarPlay Observation: CarPlaySceneDelegate intercepts the NSUserActivity, extracts the payload, and presents the CarPlayAssistTemplate.

Technical Implementation Details

1. Intent Logic (AssistAppIntent)

The AppIntent performs a context check using the SceneManager.

• Mobile Context: Accesses the mobile UI via Current.sceneManager.webViewWindowControllerPromise.then(\.webViewControllerPromise) to invoke the showAssist method.
• CarPlay Context: Uses Current.sceneManager.activateAnyScene(for: .assist, with: userInfo) where userInfo contains pipelineId, serverId, and withVoice.

2. CarPlay Signaling (CarPlaySceneDelegate)

The CarPlaySceneDelegate implements CPTemplateApplicationSceneDelegate. It monitors the userActivity of the connecting scene:

• It filters for the .assist activityType.
• It extracts the userInfo payload.
• It calls presentAssistTemplate(...), which instantiates a new AssistViewModel specifically for the CarPlay session.

3. State-Driven UI (CarPlayAssistTemplate)

To comply with CarPlay's strict UI requirements, we implemented a CarPlayTemplateProvider.

• ViewModel Mapping: The mapViewModelToState(_:) function translates the AssistViewModel states into a simplified CarPlayAssistState enum (.idle, .listening, .thinking, .speaking).
• Type-Safe Symbols: We use SFSafeSymbols to map these states to appropriate system icons (e.g., .waveform for listening, .ellipsis for thinking, .speaker for speaking) within a CPListItem.

4. Data Flow & Synchronization

• Single Source of Truth: Both the mobile UI and the CarPlay UI observe the underlying logic via AssistViewModel.
• Concurrency: The implementation uses @MainActor to ensure all UI updates occur on the main thread, preventing race conditions during rapid state changes.

[home-assistant]

Hi @lowlyocean

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[home-assistant]

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

[home-assistant]

Hi @lowlyocean

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

Stale

[bgoncal]

bundle exec fastlane autocorrect for lint issues

[bgoncal]

Can you attach a screen recording demonstrating the feature?

[lowlyocean]

Unfortunately I don't own a Mac to actually run XCode & CarPlay simulator.

This PR was prepared with OpenCode + Unsloth's gemma-4-26B-A4B-it-UD-Q3_K_M.

[bgoncal]

When opening PRs to the Open Home Foundation repos it is a requirement that all code has been tested (manually as well) and the contributor understands it's implementation when created by a LLM.

I cannot approve this PR otherwise, but if you don't have access to a mac right now I will reserve sometime to try it myself since it is a feature that I see value, I'm just a bit worried that this goes against Apple's rules, it explicity says that AI/Voice assistant CarPlay apps need to have single purpose and can't present any other UI that is not focused on that task, since we already have a CarPlay app providing different features, this can become an issue.

[lowlyocean]

Yes, I appreciate that this PR may need to be closed in its current form. I wanted this feature (as an end user) sufficiently that I found it worthwhile to at least try to offer some starting point to someone that does have Apple dev access & better familiarity with the codebase.

Thank you for testing it locally when you have some time

I did not know Apple had a rule along those lines. I might suggest that if we consider Assist widget on its own, than it would comply with such guardrails. In other words, no direct navigation from the Assist scene back to the main HA scene (or vice versa). WDYT?

[bgoncal]

To be honest, this is very new, so it will depend on Apple's review team, nevertheless, we should try

[home-assistant]

Hi @lowlyocean

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[bgoncal]

I had to tweak a lot to make it compile, main...carplay_assist

At this point I would prefer closing the PR and starting from scratch to be honest