[FirebaseAI] Add Live Audio screen

[daymxn]

This adds the LiveAudioScreen to the AI sample app. Since the Live API was released with 12.4.0, this adds a sample app showcasing how you can do audio to audio communication with a Gemini model.

The new screen also supports the following:

• Output audio transcripts
• Function calling (via changing the background color)
• Proper interruption propagation in transcripts
• Minimal support for switching audio devices during a live session
• AEC with speakers
• Automatic disabling of AEC when using bluetooth devices (eg; airpods)
• Documentation for everything added, to assist folks in parsing the source

Note:
In a follow-up PR. we'll likely add a sub-screen for the Live API, and provide other code samples (eg; text-text, audio-audio w/o function calling, video, etc.,). For those screens, we'll take advantage of the infra defined here.

[gemini-code-assist]

Summary of Changes

Hello @daymxn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates a new 'Live Audio' feature into the Firebase AI example application. It provides a practical demonstration of how to leverage the Live API for interactive audio experiences with a Gemini model, encompassing real-time transcription, dynamic UI updates via function calls, and sophisticated audio management for various device configurations.

Highlights

• New Live Audio Screen: Introduced a new LiveAudioScreen to the AI sample application, demonstrating real-time audio-to-audio communication with a Gemini model using the Live API (version 12.4.0+).
• Enhanced Audio Interaction: The new screen supports output audio transcripts, function calling (e.g., changing background color), proper interruption propagation in transcripts, and minimal support for switching audio devices during a live session.
• Advanced Audio Processing: Implemented Acoustic Echo Cancellation (AEC) with speakers and automatic disabling of AEC when using Bluetooth devices (like AirPods) to optimize audio quality.
• Comprehensive Documentation: All newly added features and components include documentation to assist developers in understanding and utilizing the source code effectively.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a LiveAudioScreen to the AI sample app, demonstrating audio-to-audio communication with a Gemini model. The implementation is comprehensive, covering audio capture, playback, transcription, and function calling. The code is well-structured, particularly the audio handling logic in AudioController and the typewriter effect for transcripts in TranscriptViewModel. My review focuses on enhancing the robustness and thread safety of the new features. I've provided suggestions to replace fatalError calls with more graceful error handling to prevent application crashes and to convert the AudioController class to an actor to prevent potential race conditions. These changes will improve the stability of the sample app and make it a better reference for production-level code.

[peterfriese]

Great work on this PR! The live audio feature is a fantastic addition and works really well on a physical device. On the simulator, the model tends to interrupt itself, maybe this is something you can look into?

I think it still needs a bit of refinement before it's ready to merge. The main thing is the error handling; could you please address the fatalError calls throughout the audio code?

I've also left a couple of remarks on the SwiftUI side of things with some ideas for improvement.

Again, really great work on this!

[andrewheard]

LGTM, thanks!

[gemini-code-assist]

Code Review

This pull request introduces a new 'Live Audio' feature, which is a great addition to showcase the capabilities of the Gemini Live API. The implementation is comprehensive, covering audio recording and playback, real-time communication with the model, function calling, and UI updates. The code is well-structured, particularly the use of an actor for the AudioController and the clean separation of concerns into different views and view models.

My review includes a few suggestions to improve robustness, maintainability, and UI quality. These include avoiding a potential crash from force-unwrapping, providing higher-resolution image assets, reducing code duplication, and using a consistent logging strategy. Overall, this is a solid contribution.

[peterfriese]

App crashes due to missing NSMicrophoneUsageDescription, both on Simulator and physical device.

The model still seemed to interrupt itself quite a bit unless the volume is turned way down.

The styling of the ConnectButton looks a bit out of place on iOS. We haven't used gradient buttons anywhere else in the app, and this might make it harder to fully adopt liquid glass. Apart from that, the size of the button and the fonts don't match the rest of the UI and should be updated to match. The connected state of the button doesn't seem to have an outline, which looks confusing.

[daymxn]

App crashes due to missing NSMicrophoneUsageDescription, both on Simulator and physical device.

Fixed

The model still seemed to interrupt itself quite a bit unless the volume is turned way down.

This is due to the apple AEC warming up. It should fix itself after a couple seconds. There's technically ways to help alleviate this, but that's adding a moderate amount of testing and code to our already moderately heavy code for AudioController, AudioPlayer, and Microphone.

[peterfriese]

Thanks for making those changes!

LGTM.