Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
legacy-peer-deps=true
37 changes: 37 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
**Too busy to clean up? Stop the CAP! Save the planet with Glip, the one and only CAPY-CAPY-BARA (and get yourself some CapyCredit while you're at it 😉). Clean Up the Cosmos, One Snap at a Time!**

## Inspiration

While I was attending FullyHacks, I noticed a paper bag blowing about in the wind. Nobody really went ahead to throw it away. I figured that I'd do it myself and that’s when I thought "Hey, people would probably be more inclined to throw away trash if was a bit more fun. Having always been interested in machine learning and video games, I thought I'd go ahead and combine something I enjoyed with a real-world problem I just witnessed.

## What it does

CapyCleanGo! is a mobile app game where users can take pictures of trash where ever they may be and receive "CapyCredit", an in-game currency. However, anyone can take pictures of trash littered about the floor and do nothing about it. Therefore, the users are rewarded on 3 criteria which much be met in the picture itself:

1. The image must contain a identifiable trash object.
2. The image must contain the user's hand.
3. The image must contain an identifiable trash bin.

This way, users are encouraged to actually throw away trash rather than just take a picture to cheat their way through the game.

## How we built it

We used react-native-expo which was an extreme challenge given our lack of mobile development experience as well as react-native in general. Moreover, we utilized a Node.js backend for API calls and ngrok to facilitate connections between the front and back-end.

## Challenges we ran into

Initially, setting up react-native-expo was difficult because of our lack of knowledge setting up mobile applications. Once we overcame that hurdle, through hours of trial and error, we ran into issues connecting our react-native front-end to the back-end. The reason for this was also because of our lack of knowledge with full-stack development. After some time researching, we found out about ngrok, which greatly helped us with our connection issues.

## Accomplishments that we're proud of

We're extremely proud of the fact that we got the camera to work in our application and not only that, but having the app be able to send the snapshot take, have the image run through a trash object detection model, and output an image of the identified object, encapsulated in a box, with labels corresponding to the predicted class.

## What we learned

We learned how to set-up and use react-native expo and its routing features as well as the fundamentals of connections the front and back end on a local host machine.

## What's next for CapyCleanGo!

We aim to bring about multiplayer feature in the future! Similar to the concept of PokemonGo, users will be able to go about a virtual map picking up trash (rather than animals) and gain CapyCredits for in-game cosmetics. Moreover, there will be an in-game shop in the points tab where users can unlock other CapyBaras besides Glip (we still love Glip though!).

Additionally, outside of the app itself, we can actually use the same camera technology for trash bins themselves! While making this project, we noticed a lot of recyclables and non-recycables placed in the wrong bins. If we were to use this same camera technology and have them attached to bins, we can have it such that the trash bins themselves will sort out the trash, reducing unnecessary waste!
14 changes: 12 additions & 2 deletions api/predict.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ export async function predictTrash(imageResource?: any) {
type: 'image/jpeg',
} as any);

formData.append('conf_threshold', '0.3');
formData.append('conf_threshold', '0.36');

const apiUrl =
Constants.expoConfig?.extra?.apiUrl ||
Expand All @@ -41,7 +41,17 @@ export async function predictTrash(imageResource?: any) {
if (jsonResponse && Array.isArray(jsonResponse.data) && jsonResponse.data.length >= 4) {
const imageObj = jsonResponse.data[2];
predictedImgUri = imageObj?.url || '';
prediction = jsonResponse.data[3];

const rawPrediction = jsonResponse.data[3];
if (typeof rawPrediction === 'string') {
if (rawPrediction.startsWith('+1')) {
prediction = "Glip Appreciates your help, you've been given 1 CapyCredit!";
} else {
prediction = "Glip...stares at you.";
}
} else {
prediction = "Glip is confused by the response.";
}
} else {
prediction = JSON.stringify(jsonResponse, null, 2);
}
Expand Down
16 changes: 10 additions & 6 deletions app/(tabs)/_layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import { Pressable } from 'react-native';
import Colors from '@/constants/Colors';
import { useColorScheme } from '@/components/useColorScheme';
import { useClientOnlyValue } from '@/components/useClientOnlyValue';
import FontAwesome6 from '@expo/vector-icons/FontAwesome6';

// You can explore the built-in icon families and icons on the web at https://icons.expo.fyi/
function TabBarIcon(props: {
name: React.ComponentProps<typeof FontAwesome>['name'];
color: string;
Expand All @@ -22,29 +22,33 @@ export default function TabLayout() {
<Tabs
screenOptions={{
tabBarActiveTintColor: Colors[colorScheme ?? 'light'].tint,
// Disable the static render of the header on web
// to prevent a hydration error in React Navigation v6.
headerShown: false,
}}>

{/* index.tsx is the home page */}
<Tabs.Screen
name="index"
options={{
title: 'start-page',
title: 'Home',
tabBarIcon: ({ color }) => <TabBarIcon name="home" color={color} />,
}}
/>

{/* start.tsx is the camera/prediction page */}
<Tabs.Screen
name="start"
options={{
title: 'Camera',
tabBarIcon: ({ color }) => <TabBarIcon name="code" color={color} />,
tabBarIcon: ({ color }) => <TabBarIcon name="camera" color={color} />,
}}
/>

{/* points.tsx is the camera/prediction page */}
<Tabs.Screen
name="points"
options={{
title: 'Points',
tabBarIcon: ({ color }) => <TabBarIcon name="code" color={color} />,
tabBarIcon: ({ color }) => <FontAwesome6 name="ranking-star" size={28} color={color} />,
}}
/>
</Tabs>
Expand Down
71 changes: 62 additions & 9 deletions app/(tabs)/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,71 @@ import EditScreenInfo from '@/components/EditScreenInfo';
import { Text, View } from '@/components/Themed';
import { navigate } from 'expo-router/build/global-state/routing';
import {useRouter} from 'expo-router';
import { Audio } from 'expo-av';


export default function Start() {
const router = useRouter();
const sound = useRef<Audio.Sound | null>(null);

const playSound = async () => {
try {
if (sound.current) {
await sound.current.unloadAsync();
}

const { sound: newSound } = await Audio.Sound.createAsync(
require('../../assets/audio/capysong.mp3') // ✅ Correct path
);
sound.current = newSound;
await sound.current.playAsync();
} catch (error) {
console.error("Failed to play sound", error);
}
};

const handleStart = () => {
router.push("/(tabs)/start");
const handleStart = async () => {
try {
if (sound.current) {
await sound.current.unloadAsync();
}

const { sound: newSound } = await Audio.Sound.createAsync(
require('../../assets/audio/capysong.mp3')
);

sound.current = newSound;

await sound.current.setIsLoopingAsync(true); // 🔁 Loop forever
await sound.current.playAsync();

router.push("/(tabs)/start"); // Navigate immediately
} catch (error) {
console.error("Failed to play sound", error);
router.push("/(tabs)/start"); // Still navigate even if sound fails
}
};

useEffect(() => {
return () => {
// Clean up sound when component unmounts
if (sound.current) {
sound.current.unloadAsync();
}
};
}, []);

return (
<ImageBackground
source={require("../../assets/images/space-background.jpg")}
style={styles.background}
>
<View style={styles.container}>
<Text style={styles.title}>CapyClean!</Text>
<Image source={require("../../assets/images/CapyClean!.png")} style={styles.capyCleanImage}/>
<Image source={require("../../assets/images/capy.png")} style={styles.capyImage}/>
<Text style={{fontSize: 30}}>Help glip clean the Earth!</Text>
<Text style={{fontSize: 30}}>Help Glip clean the Earth!</Text>
<TouchableOpacity style={styles.startButton} onPress={handleStart}>
<Text style={{fontSize: 40, color: "black"}}>Start</Text>
<Text style={{fontSize: 40, color: "black"}}>Go!</Text>
</TouchableOpacity>
</View>
</ImageBackground>
Expand All @@ -37,7 +84,7 @@ const styles = StyleSheet.create({
},
background: {
flex: 1,
resizeMode: 'cover', // or 'contain', depending on how you want the image to be displayed
resizeMode: 'cover',
justifyContent: 'center',
},
container: {
Expand All @@ -57,9 +104,15 @@ const styles = StyleSheet.create({
borderRadius: 10,
},
capyImage: {
width: 300,
height: 300,
resizeMode: 'contain', // Adjust this depending on how you want the image to be displayed
width: 350,
height: 350,
resizeMode: 'contain',
},
capyCleanImage: {
marginTop: 70,
width: 400,
height: 150,
resizeMode: 'stretch',
}
})

Expand Down
Loading