Skip to content

Gemma 4 model integration, language TTS, auto config wrangler URL#28

Open
MathisZerbib wants to merge 20 commits intofarzaa:mainfrom
MathisZerbib:enhancement/lmstudio-gemma4
Open

Gemma 4 model integration, language TTS, auto config wrangler URL#28
MathisZerbib wants to merge 20 commits intofarzaa:mainfrom
MathisZerbib:enhancement/lmstudio-gemma4

Conversation

@MathisZerbib
Copy link
Copy Markdown

@MathisZerbib MathisZerbib commented Apr 8, 2026

📝 Summary

  • Gemma 4 & Local Model Integration: Integrated support for local models through LM Studio via a repurposed OpenAIAPI.swift client pointing to 127.0.0.1:1234. If a selected model name doesn't contain "claude", the app routes Vision and Chat requests locally entirely free of charge. Fetches available local models dynamically.
  • Localization & Language TTS: Added a language selection dropdown. The selected language code is injected directly into system prompts across all pipelines (CRITICAL: Respond EXCLUSIVELY in the following language code: ...), ensuring the TTS and generated text match the user's localized preference.
  • Auto-Config Wrangler URL: Introduced WorkerEnvironment, which performs a fast 0.5s pre-flight check to localhost:8787 on startup. If a local Cloudflare worker is running, it auto-connects to it; otherwise, it seamlessly falls back to the production worker URL.
  • Settings UI Overhaul: All of these configurations (Language, LM Studio API Key, Model Selection) are now tucked neatly behind a new inline Settings view gear icon on the Companion Panel. All preferences persist via UserDefaults.

🗂️ Files Changed

  • CompanionManager.swift: Added @Published settings for selectedLanguageCode, lmStudioAPIKey, and availableModels. Added fetchAvailableModels() to pull models from LM Studio on init. Updated analyzeImageStreaming() pipelines to route appropriately to either Claude or the OpenAI SDK depending on the model, injecting the target language code into the prompt.
  • CompanionPanelView.swift: Introduced an animated gear icon resolving to a new settingsView. Features a secure API key field with a visibility toggle, the new language picker, and the model picker rows smoothly tucked away.
  • WorkerEnvironment.swift: New actor dedicated to sniffing out the http://localhost:8787 worker connection and failing over to the production URL.
  • ElevenLabsTTSClient.swift & OpenAIAPI.swift: Refactored to accept mutable base URLs. Implemented analyzeImageStreaming within the OpenAI client using SSE to conform to the same progressive data streaming expectations as the Claude API.

✅ Test Plan

  • UI Verification: Verify the new gear icon appears in the menu bar panel after onboarding and accurately expands/collapses the new settings rows.
  • Language Test: Select a language other than English (e.g., French or Spanish), run a push-to-talk query, and ensure the model replies and reads back the response in the correct language.
  • Local Model Test: Start LM Studio with Gemma 4 running locally on port 1234. Enter a local API key (if applicable), pick the Gemma model from the dropdown, and fire a vision query. Ensure responses stream back properly using the local backend.
  • Wrangler Fall-over: Run npx wrangler dev in the worker folder, launch the app, and verify it prints 🌐 Auto-detected local worker. Kill the worker, relaunch the app, and verify it falls back to parsing the prod route.
  • Persistence Test: Restart the app to ensure selectedLanguageCode, lmStudioAPIKey, and selectedGemmaModel state values survive termination gracefully via UserDefaults.

- Upgraded wrangler from version 3.0.0 to 4.81.0 in package.json.
- Changed ANTHROPIC_API_KEY to LM_STUDIO_API_KEY in index.ts to reflect new API usage.
- Updated ELEVENLABS_VOICE_ID and added ASSEMBLYAI_API_KEY and LM_STUDIO_API_KEY in wrangler.toml.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Api keys revoqued

@MathisZerbib MathisZerbib changed the title Update dependencies and API keys in worker Enhance LMStudio support Gemma 4 model integration Apr 8, 2026
@MathisZerbib MathisZerbib changed the title Enhance LMStudio support Gemma 4 model integration Gemma 4 model integration + language TTS Apr 8, 2026
@MathisZerbib MathisZerbib changed the title Gemma 4 model integration + language TTS Gemma 4 model integration, language TTS, auto config wrangler URL Apr 8, 2026
@MathisZerbib
Copy link
Copy Markdown
Author

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant