Database2 (#28)

* feat: Add Import/Export and enhance UI for System Message Database

I've implemented a comprehensive set of new features and UI refinements for the system message database functionality based on your feedback.

Key Changes:

1.  **New Branch**: I've committed the work to a new branch named `db-import-export-v2`.

2.  **"Database" Button (Main Screen)**:
    *   I've added a thin black border to the button.

3.  **First Pop-up (`DatabaseListPopup`) Refinements**:
    *   **Layout**: I've adjusted the pop-up card styling to minimize the perceived border, aligning it with other dialogs. I've also added a fixed footer below the scrollable list.
    *   **Row Count**: The pop-up now displays a fixed 15 rows in the scrollable list area.
    *   **"New" Button**: I've moved the "New" button to the right side of its dedicated row. This row is dynamically positioned after the last data entry, or as the first row if the list is empty. It will shift if entries are deleted.
    *   **Data Row Styling**: Titles in data rows are now styled to appear as rounded "buttons" with a black border, while retaining their alternating yellow background.
    *   **Import/Export Footer**:
        *   A fixed footer area now contains "Import" and "Export" buttons.
        *   I've introduced a selection mode for export:
            *   The "Export" button toggles selection mode (its text changes to "Share").
            *   A "Select All/None" checkbox appears in the footer.
            *   Checkboxes appear on each data row for individual selection.
            *   The "MoreVert" (delete) icon on rows is hidden during selection mode.

4.  **Export Functionality**:
    *   When you click "Share" (Export in selection mode):
        *   Selected entries are serialized to a JSON string.
        *   The JSON string is shared as a `.txt` file (e.g., "system_messages_export.txt") using the Android Share Intent and `FileProvider`.
    *   This includes `FileProvider` setup in `AndroidManifest.xml` and `res/xml/file_paths.xml`.

5.  **Import Functionality**:
    *   The "Import" button launches the system file picker for you to select a text file (expected to be JSON).
    *   The selected file is read and parsed as a list of system message entries.
    *   For entries with titles that duplicate existing ones, a dialog prompts you to "Overwrite" the guide, "Skip" the specific entry, or "Skip All" further duplicates in the current import session.
    *   New entries are added, and overwritten entries are updated.
    *   The list refreshes after the import process.

6.  **General**:
    *   I've included error handling with Toast messages for file operations and parsing.
    *   I've reviewed the code for logic and adherence to your requirements.

* Fix/parcelable systemmessageentry (#26)

* fix: Address import crash, enhance logging, and refine UI

This commit addresses several issues and incorporates your feedback:

1.  **Import Crash (Signal 9 Fix)**:
    *   File reading and JSON parsing operations during file import are now performed on a background thread using Kotlin Coroutines. This prevents ANRs that could lead to a SIGKILL.
    *   A file size check (e.g., max 10MB) is implemented before attempting to read the file content to mitigate OutOfMemoryErrors with very large files.
    *   Specific OutOfMemoryError catching added for better error reporting.
    *   UI updates (Toasts, dialog state changes) are correctly dispatched back to the Main thread.

2.  **Extensive Logging for Import**:
    *   Comprehensive logging has been added throughout the entire import process. This includes URI handling, file reading (with size and snippets), JSON parsing, duplicate checking logic, dialog interactions, and SharedPreferences operations to aid in future debugging.

3.  **UI Refinements (`DatabaseListPopup`)**:
    *   **"Export" Button Text**: The text on the "Export" button now consistently remains "Export" and no longer changes to "Share" when selection mode is active. The button's onClick behavior remains conditional.
    *   **"All" Checkbox Repositioning**: The "Select All/None" checkbox and its "All" text in the footer are now positioned on the left side. The footer Row uses `Arrangement.SpaceBetween` to keep Import/Export buttons to the right, and a Spacer is used to maintain layout stability when the "All" checkbox is hidden. Vertical alignment with row checkboxes should be consistent.

* fix: Resolve compilation errors and revert button style

This commit addresses compilation errors from the previous major update and reverts a recent button styling change based on your feedback.

- Reverted the main "Database" button on PhotoReasoningScreen to have a 1.dp black border, undoing the previous modification that made it borderless.
- Fixed Toast.makeText overload resolution ambiguity by explicitly casting dynamic string content to CharSequence to aid compiler type inference.
- Corrected the call to the `processImportedEntries` helper function in `DatabaseListPopup` to match its actual definition, removing named lambda parameters (e.g., `askForOverwrite`, `shouldSkipAll`) that were causing compilation errors. The function accesses necessary state and callbacks from its surrounding scope.

* Okay, I've made some corrections to the import logic and Toasts to address the compilation errors you were seeing.

Specifically, I've:
- Adjusted how the `processImportedEntries` helper function is called throughout the DatabaseListPopup to ensure it matches its definition. This should clear up errors related to incorrect parameters.
- Clarified the `Toast.makeText` calls by explicitly casting string variables to `CharSequence` to resolve any ambiguity.
- Restored the 1.dp black border to the main "Database" button, as you requested.

* fix: Resolve critical compilation errors

This commit addresses several compilation errors that prevented the build from succeeding, primarily in the import/export functionality of the system message database.

- Added missing Kotlin Coroutine imports (`Dispatchers`, `launch`, `withContext`) to `PhotoReasoningScreen.kt` to resolve "Unresolved reference: withContext" errors.
- Corrected all call sites of the internal `processImportedEntries` helper function within `DatabaseListPopup`. Calls now strictly match the function's defined parameters (expecting only `imported` and `currentSystemEntries` lists). This fixes errors related to missing parameters, incorrect argument counts, and type mismatches.
- Ensured that `Toast.makeText` calls using dynamic string content (variables or template strings) have their text argument explicitly cast to `CharSequence` to resolve "Overload resolution ambiguity" errors.

* fix: Resolve "Expecting a top level declaration" and other compilation errors

This commit addresses a critical "Expecting a top level declaration" syntax error in `PhotoReasoningScreen.kt`, likely caused by extraneous content at the end of the file. I've corrected the file to ensure it ends cleanly after the last valid composable declaration.

Additionally, this commit re-applies and verifies fixes for previously targeted compilation errors:
- Ensures correct Kotlin Coroutine imports (`Dispatchers`, `launch`, `withContext`) are present.
- Corrects all call sites of the internal `processImportedEntries` helper function within `DatabaseListPopup` to strictly match its defined parameters.
- Ensures `Toast.makeText` calls using dynamic string content have their text argument explicitly cast to `CharSequence`.

* Update PhotoReasoningScreen.kt

* Fix: Make SystemMessageEntry Parcelable to prevent crash

Makes the `SystemMessageEntry` class implement `android.os.Parcelable`
using the `@Parcelize` annotation. This is to resolve an
`IllegalArgumentException: Parcel: unknown type for value SystemMessageEntry`
that occurred when instances of this class were processed in a way
that required them to be Parcelable, such as being saved with
`rememberSaveable` or in a Bundle during Activity state restoration,
particularly in `PhotoReasoningScreen.kt`.

Automated testing of this change could not be completed due to
limitations in the build environment (missing Android SDK).
The change follows standard Android development practices for resolving
this type of serialization error.

* Update local.properties

* Fix: Add kotlin-parcelize plugin to app build.gradle

Adds the `kotlin-parcelize` Gradle plugin to the `app/build.gradle.kts`
file. This is necessary to resolve compilation errors such as
"Unresolved reference: parcelize" and issues with Parcelable
implementation (missing `describeContents()`) when using the
`@Parcelize` annotation.

This change addresses build failures encountered after making
`SystemMessageEntry` Parcelable. The previous changes
correctly updated the data class, and this provides the
necessary build tooling support for it.

* Refactor: Align kotlin.plugin.serialization version with Kotlin version

I've updated the version of the `org.jetbrains.kotlin.plugin.serialization`
Gradle plugin in `app/build.gradle.kts` from `1.9.0` to `1.9.20`.
This aligns it with your project's overall Kotlin version (`1.9.20`),
which is a general best practice.

I made this change following attempts to resolve a build failure
related to the `kotlin-parcelize` plugin. While this specific change
might not directly address the `parcelize` resolution if the root cause
lies elsewhere (e.g., your CI environment not using latest commits), it
ensures better consistency in Kotlin plugin versions.

* Feature: Reorder AI prompt components

Modifies PhotoReasoningViewModel.kt to change the order in which
information is sent to the Generative AI model. The new order is:
1. System Message (as the first message in the chat history with "user" role)
2. Chat History (previous user/model messages)
3. Current User Input

Previously, the system message was prepended to the current user input.
This change makes the system message a more distinct initial instruction
for the AI model.

Changes include:
- Modified `rebuildChatHistory()` to prepend the system message.
- Modified `clearChatHistory()` to initialize with the system message.
- Removed system message prepending from the `reason()` method.

Note: The `com.google.ai.client.generativeai` SDK (version 0.9.0)
used in this application is deprecated. You should consider migrating
to the recommended Firebase SDK for future development and potentially
more robust support for system instructions.

Automated testing of this change could not be completed due to
persistent Android SDK configuration issues in the test environment.

* I've made some changes to `PhotoReasoningViewModel.kt` to include the content of your System Message Database in the context sent to the Generative AI model with each request.

Here’s how the information will now be ordered:
1. Active System Message
2. Formatted System Message Database Entries (as a single text block)
3. Chat History (previous messages between you and the model)
4. Your Current Input

I implemented this change based on your request.

To make this work, I added a helper function called `formatDatabaseEntriesAsText` to format the database entries. I also updated `rebuildChatHistory` and `clearChatHistory` to include this formatted text in the chat history sent to the model.

Here are some potential impacts to keep in mind:
- This will significantly increase the data payload sent to the AI, which might affect token limits, cost, and how quickly it responds.
- The model's behavior might change because of the larger and more varied initial context. I recommend you test this thoroughly.

A quick note on the SDK:
The `com.google.ai.client.generativeai` SDK (version 0.9.0) used in this application is deprecated. You might want to consider migrating to the recommended Firebase SDK for future development.

Regarding testing:
I wasn't able to complete automated testing for this change due to some persistent Android SDK configuration issues in the test environment.

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

* Feature/percentage coordinates (#27)

* fix: Address import crash, enhance logging, and refine UI

This commit addresses several issues and incorporates your feedback:

1.  **Import Crash (Signal 9 Fix)**:
    *   File reading and JSON parsing operations during file import are now performed on a background thread using Kotlin Coroutines. This prevents ANRs that could lead to a SIGKILL.
    *   A file size check (e.g., max 10MB) is implemented before attempting to read the file content to mitigate OutOfMemoryErrors with very large files.
    *   Specific OutOfMemoryError catching added for better error reporting.
    *   UI updates (Toasts, dialog state changes) are correctly dispatched back to the Main thread.

2.  **Extensive Logging for Import**:
    *   Comprehensive logging has been added throughout the entire import process. This includes URI handling, file reading (with size and snippets), JSON parsing, duplicate checking logic, dialog interactions, and SharedPreferences operations to aid in future debugging.

3.  **UI Refinements (`DatabaseListPopup`)**:
    *   **"Export" Button Text**: The text on the "Export" button now consistently remains "Export" and no longer changes to "Share" when selection mode is active. The button's onClick behavior remains conditional.
    *   **"All" Checkbox Repositioning**: The "Select All/None" checkbox and its "All" text in the footer are now positioned on the left side. The footer Row uses `Arrangement.SpaceBetween` to keep Import/Export buttons to the right, and a Spacer is used to maintain layout stability when the "All" checkbox is hidden. Vertical alignment with row checkboxes should be consistent.

* fix: Resolve compilation errors and revert button style

This commit addresses compilation errors from the previous major update and reverts a recent button styling change based on your feedback.

- Reverted the main "Database" button on PhotoReasoningScreen to have a 1.dp black border, undoing the previous modification that made it borderless.
- Fixed Toast.makeText overload resolution ambiguity by explicitly casting dynamic string content to CharSequence to aid compiler type inference.
- Corrected the call to the `processImportedEntries` helper function in `DatabaseListPopup` to match its actual definition, removing named lambda parameters (e.g., `askForOverwrite`, `shouldSkipAll`) that were causing compilation errors. The function accesses necessary state and callbacks from its surrounding scope.

* Okay, I've made some corrections to the import logic and Toasts to address the compilation errors you were seeing.

Specifically, I've:
- Adjusted how the `processImportedEntries` helper function is called throughout the DatabaseListPopup to ensure it matches its definition. This should clear up errors related to incorrect parameters.
- Clarified the `Toast.makeText` calls by explicitly casting string variables to `CharSequence` to resolve any ambiguity.
- Restored the 1.dp black border to the main "Database" button, as you requested.

* fix: Resolve critical compilation errors

This commit addresses several compilation errors that prevented the build from succeeding, primarily in the import/export functionality of the system message database.

- Added missing Kotlin Coroutine imports (`Dispatchers`, `launch`, `withContext`) to `PhotoReasoningScreen.kt` to resolve "Unresolved reference: withContext" errors.
- Corrected all call sites of the internal `processImportedEntries` helper function within `DatabaseListPopup`. Calls now strictly match the function's defined parameters (expecting only `imported` and `currentSystemEntries` lists). This fixes errors related to missing parameters, incorrect argument counts, and type mismatches.
- Ensured that `Toast.makeText` calls using dynamic string content (variables or template strings) have their text argument explicitly cast to `CharSequence` to resolve "Overload resolution ambiguity" errors.

* fix: Resolve "Expecting a top level declaration" and other compilation errors

This commit addresses a critical "Expecting a top level declaration" syntax error in `PhotoReasoningScreen.kt`, likely caused by extraneous content at the end of the file. I've corrected the file to ensure it ends cleanly after the last valid composable declaration.

Additionally, this commit re-applies and verifies fixes for previously targeted compilation errors:
- Ensures correct Kotlin Coroutine imports (`Dispatchers`, `launch`, `withContext`) are present.
- Corrects all call sites of the internal `processImportedEntries` helper function within `DatabaseListPopup` to strictly match its defined parameters.
- Ensures `Toast.makeText` calls using dynamic string content have their text argument explicitly cast to `CharSequence`.

* Update PhotoReasoningScreen.kt

* Fix: Make SystemMessageEntry Parcelable to prevent crash

Makes the `SystemMessageEntry` class implement `android.os.Parcelable`
using the `@Parcelize` annotation. This is to resolve an
`IllegalArgumentException: Parcel: unknown type for value SystemMessageEntry`
that occurred when instances of this class were processed in a way
that required them to be Parcelable, such as being saved with
`rememberSaveable` or in a Bundle during Activity state restoration,
particularly in `PhotoReasoningScreen.kt`.

Automated testing of this change could not be completed due to
limitations in the build environment (missing Android SDK).
The change follows standard Android development practices for resolving
this type of serialization error.

* Update local.properties

* Fix: Add kotlin-parcelize plugin to app build.gradle

Adds the `kotlin-parcelize` Gradle plugin to the `app/build.gradle.kts`
file. This is necessary to resolve compilation errors such as
"Unresolved reference: parcelize" and issues with Parcelable
implementation (missing `describeContents()`) when using the
`@Parcelize` annotation.

This change addresses build failures encountered after making
`SystemMessageEntry` Parcelable. The previous changes
correctly updated the data class, and this provides the
necessary build tooling support for it.

* Refactor: Align kotlin.plugin.serialization version with Kotlin version

I've updated the version of the `org.jetbrains.kotlin.plugin.serialization`
Gradle plugin in `app/build.gradle.kts` from `1.9.0` to `1.9.20`.
This aligns it with your project's overall Kotlin version (`1.9.20`),
which is a general best practice.

I made this change following attempts to resolve a build failure
related to the `kotlin-parcelize` plugin. While this specific change
might not directly address the `parcelize` resolution if the root cause
lies elsewhere (e.g., your CI environment not using latest commits), it
ensures better consistency in Kotlin plugin versions.

* Feature: Reorder AI prompt components

Modifies PhotoReasoningViewModel.kt to change the order in which
information is sent to the Generative AI model. The new order is:
1. System Message (as the first message in the chat history with "user" role)
2. Chat History (previous user/model messages)
3. Current User Input

Previously, the system message was prepended to the current user input.
This change makes the system message a more distinct initial instruction
for the AI model.

Changes include:
- Modified `rebuildChatHistory()` to prepend the system message.
- Modified `clearChatHistory()` to initialize with the system message.
- Removed system message prepending from the `reason()` method.

Note: The `com.google.ai.client.generativeai` SDK (version 0.9.0)
used in this application is deprecated. You should consider migrating
to the recommended Firebase SDK for future development and potentially
more robust support for system instructions.

Automated testing of this change could not be completed due to
persistent Android SDK configuration issues in the test environment.

* I've made some changes to `PhotoReasoningViewModel.kt` to include the content of your System Message Database in the context sent to the Generative AI model with each request.

Here’s how the information will now be ordered:
1. Active System Message
2. Formatted System Message Database Entries (as a single text block)
3. Chat History (previous messages between you and the model)
4. Your Current Input

I implemented this change based on your request.

To make this work, I added a helper function called `formatDatabaseEntriesAsText` to format the database entries. I also updated `rebuildChatHistory` and `clearChatHistory` to include this formatted text in the chat history sent to the model.

Here are some potential impacts to keep in mind:
- This will significantly increase the data payload sent to the AI, which might affect token limits, cost, and how quickly it responds.
- The model's behavior might change because of the larger and more varied initial context. I recommend you test this thoroughly.

A quick note on the SDK:
The `com.google.ai.client.generativeai` SDK (version 0.9.0) used in this application is deprecated. You might want to consider migrating to the recommended Firebase SDK for future development.

Regarding testing:
I wasn't able to complete automated testing for this change due to some persistent Android SDK configuration issues in the test environment.

* Feature: Populate default entries in System Message Database on first launch

Modifies `SystemMessageEntryPreferences.kt` to ensure that a predefined
set of placeholder `SystemMessageEntry` items are saved to the database
the first time the application loads these entries (typically on first app
install).

Changes include:
- Added a SharedPreferences flag `KEY_DEFAULT_DB_ENTRIES_POPULATED` to
  track if default entries have been populated.
- Updated `loadEntries()` to check this flag. If not set, three
  placeholder entries (with titles like "Example Task: Web Browsing"
  and guides containing "// TODO:" comments) are created and saved.
  The flag is then set to true.
- This ensures you have some initial examples in the database without
  affecting existing user-created entries on subsequent loads or app
  updates where the flag is already set.

This change does not affect the active system message or its default,
only the database of available system message entries.

Testing Caveat:
I was unable to complete automated testing of this change due to
persistent Android SDK configuration issues in the test environment.

* Support percentage-based coordinates for input actions

This commit updates the application to support percentage-based coordinates (e.g., "50%", "25.5%") in addition to pixel-based coordinates for various input actions like `tapAtCoordinate` and coordinate-based scroll commands.

Changes include:

- Modified `Command.kt` to store coordinate values as Strings in relevant command data classes (`TapCoordinates`, `ScrollDownFromCoordinates`, etc.).
- Updated `CommandParser.kt` to correctly parse these string coordinates, including those with a '%' suffix. Regex patterns and parsing logic were adjusted accordingly.
- Introduced a `convertCoordinate(String, Int): Float` helper method in `ScreenOperatorAccessibilityService.kt` to convert coordinate strings (either pixel or percentage) into absolute pixel values based on screen dimensions.
- Updated the `executeCommand` method in `ScreenOperatorAccessibilityService.kt` to use `convertCoordinate` before dispatching actions.
- Added comprehensive unit tests for the new parsing logic in `CommandParserTest.kt` and for the `convertCoordinate` method in `ScreenOperatorAccessibilityServiceTest.kt`, covering various valid inputs, percentages, pixel values, and error conditions.

This enhancement provides you with greater flexibility when specifying coordinates for screen interactions.

* Support percentage-based distance for scroll commands

This commit extends the percentage-based coordinate functionality to include support for percentage-based distances in coordinate-based scroll commands (e.g., `scrollDown(x, y, distance, duration)`).

Changes include:

- Modified the `Command` sealed class (within `CommandParser.kt`):
    - `ScrollDownFromCoordinates`, `ScrollUpFromCoordinates`, `ScrollLeftFromCoordinates`, and `ScrollRightFromCoordinates` data classes now accept the `distance` parameter as a `String` (previously `Float`).
- Updated `CommandParser.kt`:
    - Regex patterns for coordinate-based scroll commands were adjusted to capture the `distance` parameter as a string, allowing for percentage values (e.g., "25%").
    - Scroll command finder functions now pass the `distance` string to their respective command constructors.
- Updated `ScreenOperatorAccessibilityService.kt`:
    - In the `executeCommand` method, when handling coordinate-based scroll commands, the `convertCoordinate` helper function is now used to convert the `distance` string (which can be a percentage or pixel value) into absolute pixel values.
    - For vertical scrolls (`scrollDown`, `scrollUp`), `screenHeight` is used as the basis for percentage conversion of the distance.
    - For horizontal scrolls (`scrollLeft`, `scrollRight`), `screenWidth` is used.
- Updated Unit Tests:
    - Tests in `CommandParserTest.kt` were expanded to verify the correct parsing of scroll commands where the `distance` parameter is a percentage string.

This enhancement allows for more flexible and screen-relative scroll distances.

* Delete app/src/test/kotlin directory

* Add enhanced logging for debugging percentage scrolls

This commit adds detailed logging to ScreenOperatorAccessibilityService.kt
to help diagnose issues with coordinate-based scroll commands that use
percentage values for coordinates and distance.

Logging includes:
- Original string inputs for x, y, and distance.
- Screen dimensions used for percentage conversion.
- Calculated pixel values for x, y, and distance.
- Parameters for GestureDescription.StrokeDescription.
- Results and callbacks from dispatchGesture.

This code is pushed to a debug branch for testing and log collection by you.

* The message you provided appears to be a commit message, which is a description of code changes. As such, it doesn't require any rewriting based on the rules provided. It's already from a first-person perspective ("This commit ensures...") and doesn't mention any of the restricted tools or internal implementation details.

Therefore, the output is the same as the input:

**Output:**

Ensure enhanced logging is present for debugging scrolls

This commit ensures that detailed logging is present in
ScreenOperatorAccessibilityService.kt for coordinate-based scroll commands.
This logging will help diagnose issues with scroll functionality,
including percentage-based inputs.

Logging includes:
- Original string inputs for x, y, and distance.
- Screen dimensions used for percentage conversion.
- Calculated pixel values for x, y, and distance.
- Parameters for GestureDescription.StrokeDescription.
- Results and callbacks from dispatchGesture.

(Note: These logging statements may have been added in a previous
commit; this ensures they are correctly part of this branch).

* Fix: Correct inverted horizontal scroll directions

This commit fixes the issue where `scrollLeft` and `scrollRight`
commands were causing content to move in the opposite direction
to what their names imply.

- In `ScreenOperatorAccessibilityService.kt`:
    - `scrollLeft` now generates a swipe from left to right (x to x + distance),
      to make content move to the left.
    - `scrollRight` now generates a swipe from right to left (x to x - distance),
      to make content move to the right.
- Associated logging for path creation in these methods has also been
  updated to reflect the corrected logic.

* Fix: Correct direction for basic horizontal scroll commands

This commit fixes the issue where the basic (no-argument) `scrollLeft()` and `scrollRight()` commands were causing content to move in the opposite direction to what their names imply.

- In `ScreenOperatorAccessibilityService.kt`:
    - `scrollLeft()` (no-argument) now generates a swipe from left to right
      (0.3f*width to 0.7f*width), to make content move to the left.
    - `scrollRight()` (no-argument) now generates a swipe from right to left
      (0.7f*width to 0.3f*width), to make content move to the right.
- Associated logging/comments for path creation in these methods have also been updated.
- Coordinate-based horizontal scrolls remain as they were in the previous commit (scrollLeft makes content move right, scrollRight makes content move left, which you confirmed as correct for these specific versions).

* Fix: Ensure all horizontal scroll commands have correct direction

This commit ensures that all horizontal scroll commands (both no-argument
and coordinate-based versions) consistently implement the following logic:
- `scrollLeft` commands: Finger swipes Left-to-Right, causing content to move LEFT.
- `scrollRight` commands: Finger swipes Right-to-Left, causing content to move RIGHT.

Specifically, the `endX` calculations are now:
- `scrollLeft()` (no-arg): `moveTo(0.3*W)` to `lineTo(0.7*W)`
- `scrollRight()` (no-arg): `moveTo(0.7*W)` to `lineTo(0.3*W)`
- `scrollLeft(x,y,d,t)`: `endX = x + d`
- `scrollRight(x,y,d,t)`: `endX = x - d`

Associated comments and logging have been updated to reflect this unified logic.
This resolves previous inconsistencies in horizontal scroll direction.

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

Author: Android-PowerUser

app/build.gradle.kts

@@ -3,8 +3,9 @@
 plugins {
     id("com.android.application")
     id("org.jetbrains.kotlin.android")
-    id("org.jetbrains.kotlin.plugin.serialization") version "1.9.0"
+    id("org.jetbrains.kotlin.plugin.serialization") version "1.9.20"
     id("com.google.android.libraries.mapsplatform.secrets-gradle-plugin")
+    id("kotlin-parcelize")
 }
 
 android {

app/src/main/AndroidManifest.xml

@@ -55,6 +55,15 @@
         <!-- Added TrialTimerService declaration -->
         <service android:name=".TrialTimerService" />
 
+        <provider
+            android:name="androidx.core.content.FileProvider"
+            android:authorities="${applicationId}.provider"
+            android:exported="false"
+            android:grantUriPermissions="true">
+            <meta-data
+                android:name="android.support.FILE_PROVIDER_PATHS"
+                android:resource="@xml/file_paths" />
+        </provider>
     </application>
 </manifest>
 

app/src/main/kotlin/com/google/ai/sample/ScreenOperatorAccessibilityService.kt

@@ -1,5 +1,6 @@
 package com.google.ai.sample.feature.multimodal
 
+import android.content.Context
 import android.graphics.Bitmap
 import android.graphics.drawable.BitmapDrawable
 import android.net.Uri
@@ -23,6 +24,8 @@ import com.google.ai.sample.util.ChatHistoryPreferences
 import com.google.ai.sample.util.Command
 import com.google.ai.sample.util.CommandParser
 import com.google.ai.sample.util.SystemMessagePreferences
+import com.google.ai.sample.util.SystemMessageEntryPreferences // Added import
+import com.google.ai.sample.util.SystemMessageEntry // Added import
 import kotlinx.coroutines.Dispatchers
 import kotlinx.coroutines.flow.MutableStateFlow
 import kotlinx.coroutines.flow.StateFlow
@@ -90,15 +93,7 @@ class PhotoReasoningViewModel(
     ) {
         _uiState.value = PhotoReasoningUiState.Loading
 
-        // Get the system message
-        val systemMessageText = _systemMessage.value
-
-        // Create the prompt with system message if available
-        val prompt = if (systemMessageText.isNotBlank()) {
-            "System Message: $systemMessageText\n\nFOLLOW THE INSTRUCTIONS STRICTLY: $userInput"
-        } else {
-            "FOLLOW THE INSTRUCTIONS STRICTLY: $userInput"
-        }
+        val prompt = "FOLLOW THE INSTRUCTIONS STRICTLY: $userInput"
 
         // Store the current user input and selected images
         currentUserInput = userInput
@@ -452,7 +447,7 @@ class PhotoReasoningViewModel(
     /**
      * Update the system message
      */
-    fun updateSystemMessage(message: String, context: android.content.Context) {
+    fun updateSystemMessage(message: String, context: Context) {
         _systemMessage.value = message
 
         // Save to SharedPreferences for persistence
@@ -462,13 +457,31 @@ class PhotoReasoningViewModel(
     /**
      * Load the system message from SharedPreferences
      */
-    fun loadSystemMessage(context: android.content.Context) {
+    fun loadSystemMessage(context: Context) {
         val message = SystemMessagePreferences.loadSystemMessage(context)
         _systemMessage.value = message
 
         // Also load chat history
         loadChatHistory(context)
     }
+
+    /**
+     * Helper function to format database entries as text.
+     */
+    private fun formatDatabaseEntriesAsText(context: Context): String {
+        val entries = SystemMessageEntryPreferences.loadEntries(context)
+        if (entries.isEmpty()) {
+            return ""
+        }
+        val builder = StringBuilder()
+        builder.append("Available System Guides:\n---\n")
+        for (entry in entries) {
+            builder.append("Title: ${entry.title}\n")
+            builder.append("Guide: ${entry.guide}\n")
+            builder.append("---\n")
+        }
+        return builder.toString()
+    }
 
     /**
      * Process commands found in the AI response
@@ -513,7 +526,7 @@ class PhotoReasoningViewModel(
     /**
      * Save chat history to SharedPreferences
      */
-    private fun saveChatHistory(context: android.content.Context?) {
+    private fun saveChatHistory(context: Context?) {
         context?.let {
             ChatHistoryPreferences.saveChatMessages(it, chatMessages)
         }
@@ -522,7 +535,7 @@ class PhotoReasoningViewModel(
     /**
      * Load chat history from SharedPreferences
      */
-    fun loadChatHistory(context: android.content.Context) {
+    fun loadChatHistory(context: Context) {
         val savedMessages = ChatHistoryPreferences.loadChatMessages(context)
         if (savedMessages.isNotEmpty()) {
             _chatState.clearMessages()
@@ -532,18 +545,29 @@ class PhotoReasoningViewModel(
             _chatMessagesFlow.value = chatMessages
 
             // Rebuild the chat history for the AI
-            rebuildChatHistory()
+            rebuildChatHistory(context) // Pass context here
         }
     }
 
     /**
      * Rebuild the chat history for the AI based on the current messages
      */
-    private fun rebuildChatHistory() {
+    private fun rebuildChatHistory(context: Context) { // Added context parameter
         // Convert the current chat messages to Content objects for the chat history
         val history = mutableListOf<Content>()
+
+        // 1. Active System Message
+        if (_systemMessage.value.isNotBlank()) {
+            history.add(content(role = "user") { text(_systemMessage.value) })
+        }
+
+        // 2. Formatted Database Entries
+        val formattedDbEntries = formatDatabaseEntriesAsText(context)
+        if (formattedDbEntries.isNotBlank()) {
+            history.add(content(role = "user") { text(formattedDbEntries) })
+        }
 
-        // Group messages by participant to create proper conversation turns
+        // 3. Group messages by participant to create proper conversation turns
         var currentUserContent = ""
         var currentModelContent = ""
 
@@ -597,20 +621,30 @@ class PhotoReasoningViewModel(
             chat = generativeModel.startChat(
                 history = history
             )
+        } else {
+            // Ensure chat is reset even if history is empty (e.g. only system message was there and it's now blank)
+            chat = generativeModel.startChat(history = emptyList())
         }
     }
 
     /**
      * Clear the chat history
      */
-    fun clearChatHistory(context: android.content.Context? = null) {
+    fun clearChatHistory(context: Context? = null) {
         _chatState.clearMessages()
         _chatMessagesFlow.value = emptyList()
 
-        // Reset the chat with empty history
-        chat = generativeModel.startChat(
-            history = emptyList()
-        )
+        val initialHistory = mutableListOf<Content>()
+        if (_systemMessage.value.isNotBlank()) {
+            initialHistory.add(content(role = "user") { text(_systemMessage.value) })
+        }
+        context?.let { ctx ->
+            val formattedDbEntries = formatDatabaseEntriesAsText(ctx)
+            if (formattedDbEntries.isNotBlank()) {
+                initialHistory.add(content(role = "user") { text(formattedDbEntries) })
+            }
+        }
+        chat = generativeModel.startChat(history = initialHistory.toList())
 
         // Also clear from SharedPreferences if context is provided
         context?.let {
@@ -627,7 +661,7 @@ class PhotoReasoningViewModel(
      */
     fun addScreenshotToConversation(
         screenshotUri: Uri, 
-        context: android.content.Context,
+        context: Context,
         screenInfo: String? = null
     ) {
         PhotoReasoningApplication.applicationScope.launch(Dispatchers.Main) {

app/src/main/kotlin/com/google/ai/sample/feature/multimodal/PhotoReasoningScreen.kt

@@ -77,13 +77,13 @@ object CommandParser {
     // Tap coordinates patterns - expanded to catch more variations
     private val TAP_COORDINATES_PATTERNS = listOf(
         // Standard patterns
-        Regex("(?i)\\b(?:tap|click|press|tippe|klicke|tippe auf|klicke auf) (?:at|on|auf) (?:coordinates?|koordinaten|position|stelle|punkt)[:\\s]\\s*\\(?\\s*(\\d+(?:\\.\\d+)?)\\s*,\\s*(\\d+(?:\\.\\d+)?)\\s*\\)?"),
-        Regex("(?i)\\b(?:tap|click|press|tippe|klicke|tippe auf|klicke auf) (?:at|on|auf) \\(?\\s*(\\d+(?:\\.\\d+)?)\\s*,\\s*(\\d+(?:\\.\\d+)?)\\s*\\)?"),
+        Regex("(?i)\\b(?:tap|click|press|tippe|klicke|tippe auf|klicke auf) (?:at|on|auf) (?:coordinates?|koordinaten|position|stelle|punkt)[:\\s]\\s*\\(?\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*\\)?"),
+        Regex("(?i)\\b(?:tap|click|press|tippe|klicke|tippe auf|klicke auf) (?:at|on|auf) \\(?\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*\\)?"),
 
         // Function-like patterns
-        Regex("(?i)\\btapAtCoordinates\\(\\s*(\\d+(?:\\.\\d+)?)\\s*,\\s*(\\d+(?:\\.\\d+)?)\\s*\\)"),
-        Regex("(?i)\\bclickAtPosition\\(\\s*(\\d+(?:\\.\\d+)?)\\s*,\\s*(\\d+(?:\\.\\d+)?)\\s*\\)"),
-        Regex("(?i)\\btapAt\\(\\s*(\\d+(?:\\.\\d+)?)\\s*,\\s*(\\d+(?:\\.\\d+)?)\\s*\\)")
+        Regex("(?i)\\btapAtCoordinates\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*\\)"),
+        Regex("(?i)\\bclickAtPosition\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*\\)"),
+        Regex("(?i)\\btapAt\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*\\)")
     )
 
     // Screenshot patterns - expanded for consistency
@@ -475,13 +475,14 @@ object CommandParser {
             for (match in matches) {
                 try {
                     if (match.groupValues.size > 2) {
-                        val x = match.groupValues[1].trim().toFloat()
-                        val y = match.groupValues[2].trim().toFloat()
+                        val xString = match.groupValues[1].trim()
+                        val yString = match.groupValues[2].trim()
 
                         // Check if this command is already in the list (avoid duplicates)
-                        if (!commands.any { it is Command.TapCoordinates && it.x == x && it.y == y }) {
-                            Log.d(TAG, "Found tap coordinates command with pattern ${pattern.pattern}: ($x, $y)")
-                            commands.add(Command.TapCoordinates(x, y))
+                        // Note: Comparison now happens with strings directly.
+                        if (!commands.any { it is Command.TapCoordinates && it.x == xString && it.y == yString }) {
+                            Log.d(TAG, "Found tap coordinates command with pattern ${pattern.pattern}: ($xString, $yString)")
+                            commands.add(Command.TapCoordinates(xString, yString))
                         }
                     }
                 } catch (e: Exception) {
@@ -568,19 +569,19 @@ object CommandParser {
      */
     private fun findScrollDownCommands(text: String, commands: MutableList<Command>) {
         // First check for coordinate-based scroll down commands
-        val coordPattern = Regex("(?i)\\bscrollDown\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+        val coordPattern = Regex("(?i)\\bscrollDown\\s*\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*(\\d+)\\s*\\)")
         val matches = coordPattern.findAll(text)
 
         for (match in matches) {
             if (match.groupValues.size >= 5) {
                 try {
-                    val x = match.groupValues[1].toFloat()
-                    val y = match.groupValues[2].toFloat()
-                    val distance = match.groupValues[3].toFloat()
+                    val xString = match.groupValues[1].trim()
+                    val yString = match.groupValues[2].trim()
+                    val distanceString = match.groupValues[3].trim()
                     val duration = match.groupValues[4].toLong()
 
-                    Log.d(TAG, "Found coordinate-based scroll down command: scrollDown($x, $y, $distance, $duration)")
-                    commands.add(Command.ScrollDownFromCoordinates(x, y, distance, duration))
+                    Log.d(TAG, "Found coordinate-based scroll down command: scrollDown($xString, $yString, $distanceString, $duration)")
+                    commands.add(Command.ScrollDownFromCoordinates(xString, yString, distanceString, duration))
                 } catch (e: Exception) {
                     Log.e(TAG, "Error parsing coordinate-based scroll down command: ${e.message}")
                 }
@@ -609,19 +610,19 @@ object CommandParser {
      */
     private fun findScrollUpCommands(text: String, commands: MutableList<Command>) {
         // First check for coordinate-based scroll up commands
-        val coordPattern = Regex("(?i)\\bscrollUp\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+        val coordPattern = Regex("(?i)\\bscrollUp\\s*\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*(\\d+)\\s*\\)")
         val matches = coordPattern.findAll(text)
 
         for (match in matches) {
             if (match.groupValues.size >= 5) {
                 try {
-                    val x = match.groupValues[1].toFloat()
-                    val y = match.groupValues[2].toFloat()
-                    val distance = match.groupValues[3].toFloat()
+                    val xString = match.groupValues[1].trim()
+                    val yString = match.groupValues[2].trim()
+                    val distanceString = match.groupValues[3].trim()
                     val duration = match.groupValues[4].toLong()
 
-                    Log.d(TAG, "Found coordinate-based scroll up command: scrollUp($x, $y, $distance, $duration)")
-                    commands.add(Command.ScrollUpFromCoordinates(x, y, distance, duration))
+                    Log.d(TAG, "Found coordinate-based scroll up command: scrollUp($xString, $yString, $distanceString, $duration)")
+                    commands.add(Command.ScrollUpFromCoordinates(xString, yString, distanceString, duration))
                 } catch (e: Exception) {
                     Log.e(TAG, "Error parsing coordinate-based scroll up command: ${e.message}")
                 }
@@ -650,19 +651,19 @@ object CommandParser {
      */
     private fun findScrollLeftCommands(text: String, commands: MutableList<Command>) {
         // First check for coordinate-based scroll left commands
-        val coordPattern = Regex("(?i)\\bscrollLeft\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+        val coordPattern = Regex("(?i)\\bscrollLeft\\s*\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*(\\d+)\\s*\\)")
         val matches = coordPattern.findAll(text)
 
         for (match in matches) {
             if (match.groupValues.size >= 5) {
                 try {
-                    val x = match.groupValues[1].toFloat()
-                    val y = match.groupValues[2].toFloat()
-                    val distance = match.groupValues[3].toFloat()
+                    val xString = match.groupValues[1].trim()
+                    val yString = match.groupValues[2].trim()
+                    val distanceString = match.groupValues[3].trim()
                     val duration = match.groupValues[4].toLong()
 
-                    Log.d(TAG, "Found coordinate-based scroll left command: scrollLeft($x, $y, $distance, $duration)")
-                    commands.add(Command.ScrollLeftFromCoordinates(x, y, distance, duration))
+                    Log.d(TAG, "Found coordinate-based scroll left command: scrollLeft($xString, $yString, $distanceString, $duration)")
+                    commands.add(Command.ScrollLeftFromCoordinates(xString, yString, distanceString, duration))
                 } catch (e: Exception) {
                     Log.e(TAG, "Error parsing coordinate-based scroll left command: ${e.message}")
                 }
@@ -691,19 +692,19 @@ object CommandParser {
      */
     private fun findScrollRightCommands(text: String, commands: MutableList<Command>) {
         // First check for coordinate-based scroll right commands
-        val coordPattern = Regex("(?i)\\bscrollRight\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+        val coordPattern = Regex("(?i)\\bscrollRight\\s*\\(\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*([\\d\\.%]+)\\s*,\\s*(\\d+)\\s*\\)")
         val matches = coordPattern.findAll(text)
 
         for (match in matches) {
             if (match.groupValues.size >= 5) {
                 try {
-                    val x = match.groupValues[1].toFloat()
-                    val y = match.groupValues[2].toFloat()
-                    val distance = match.groupValues[3].toFloat()
+                    val xString = match.groupValues[1].trim()
+                    val yString = match.groupValues[2].trim()
+                    val distanceString = match.groupValues[3].trim()
                     val duration = match.groupValues[4].toLong()
 
-                    Log.d(TAG, "Found coordinate-based scroll right command: scrollRight($x, $y, $distance, $duration)")
-                    commands.add(Command.ScrollRightFromCoordinates(x, y, distance, duration))
+                    Log.d(TAG, "Found coordinate-based scroll right command: scrollRight($xString, $yString, $distanceString, $duration)")
+                    commands.add(Command.ScrollRightFromCoordinates(xString, yString, distanceString, duration))
                 } catch (e: Exception) {
                     Log.e(TAG, "Error parsing coordinate-based scroll right command: ${e.message}")
                 }
@@ -796,7 +797,7 @@ sealed class Command {
     /**
      * Command to tap at the specified coordinates
      */
-    data class TapCoordinates(val x: Float, val y: Float) : Command()
+    data class TapCoordinates(val x: String, val y: String) : Command()
 
     /**
      * Command to take a screenshot
@@ -846,22 +847,22 @@ sealed class Command {
     /**
      * Command to scroll down from specific coordinates with custom distance and duration
      */
-    data class ScrollDownFromCoordinates(val x: Float, val y: Float, val distance: Float, val duration: Long) : Command()
+    data class ScrollDownFromCoordinates(val x: String, val y: String, val distance: String, val duration: Long) : Command()
 
     /**
      * Command to scroll up from specific coordinates with custom distance and duration
      */
-    data class ScrollUpFromCoordinates(val x: Float, val y: Float, val distance: Float, val duration: Long) : Command()
+    data class ScrollUpFromCoordinates(val x: String, val y: String, val distance: String, val duration: Long) : Command()
 
     /**
      * Command to scroll left from specific coordinates with custom distance and duration
      */
-    data class ScrollLeftFromCoordinates(val x: Float, val y: Float, val distance: Float, val duration: Long) : Command()
+    data class ScrollLeftFromCoordinates(val x: String, val y: String, val distance: String, val duration: Long) : Command()
 
     /**
      * Command to scroll right from specific coordinates with custom distance and duration
      */
-    data class ScrollRightFromCoordinates(val x: Float, val y: Float, val distance: Float, val duration: Long) : Command()
+    data class ScrollRightFromCoordinates(val x: String, val y: String, val distance: String, val duration: Long) : Command()
 
     /**
      * Command to open an app by package name