GuidanceAI

🌟 Overview

GuidanceAI is an innovative mobile application designed to empower visually impaired individuals by helping them interact with their environment more effectively. Using advanced AI technology, the app provides real-time assistance and environmental interpretation through voice feedback and accessible interface design.

📱 Demo

✨ Key Features

• Real-time Environment Analysis: Instantly analyze surroundings using advanced computer vision
• Voice-Guided Navigation: Clear audio descriptions and feedback
• Accessibility-First Design: Built from the ground up with accessibility in mind
• Image Recognition: Identify objects, text, and scenes in real-time
• Audio Transcription: Convert spoken words to text for enhanced communication
• Customizable Interface: Adjustable text sizes and contrast settings
• Tutorial Mode: Interactive guidance for new users

🚀 Getting Started

Prerequisites

• Node.js (v14 or higher)
• Expo CLI
• iOS Simulator (for iOS development)
• Android Studio and Android SDK (for Android development)
• Expo Go app on your physical device (for testing)

Installation

1. Clone the repository:

git clone https://github.com/aiea-lab/Guidance_app.git
cd Guidance_app

2. Install dependencies:

npm install

Running the App

Start the development server:

npx expo start --clear

Then you can:

• Press i to open in iOS simulator
• Press a to open in Android emulator
• Scan the QR code with Expo Go app on your physical device

For a clean start (clearing cache):

npx expo start --clear

🎯 Usage

GuidanceAI is designed to be intuitive and accessible:

1. Launch the app
2. Complete the initial tutorial for optimal experience
3. Use the camera to analyze surroundings
4. Receive real-time audio feedback and descriptions
5. Customize settings as needed through the accessible settings menu

🛠 Technical Stack

• React Native · Expo · TypeScript
• React Navigation · TanStack React Query
• OpenAI API (vision) · expo-camera · expo-speech (TTS)
• Accessibility (VoiceOver / TalkBack, screen reader support)
• iOS & Android (native builds)

📚 Citation

This is an ongoing project as a follow-up to our Guidance paper (Right this way: Can VLMs Guide Us to See More to Answer Questions?, NeurIPS 2024). If you use this work, please cite:

@article{liu2024right,
  title={Right this way: Can VLMs Guide Us to See More to Answer Questions?},
  author={Liu, Li and Yang, Diji and Zhong, Sijia and Tholeti, Kalyana Suma Sree and Ding, Lei and Zhang, Yi and Gilpin, Leilani},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={132946--132976},
  year={2024}
}

---