Skip to content

Removed some functionalities and fixed iPad sound issue #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sytraore merged 4 commits into from
Oct 9, 2025
Merged

Removed some functionalities and fixed iPad sound issue #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
07cebf1
Comment out feedback logic in tray click handlers
sytraore Oct 1, 2025
6e5a9d4
Update README.md
sytraore Oct 9, 2025
902e82c
Fixed iPads sound issue.
sytraore Oct 9, 2025
113ceb6
Merge branch 'main' of https://github.com/sytraore/NX-Counting-App
sytraore Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#
# Counting-App on NX

<a alt="Nx logo" href="https://nx.dev" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/nrwl/nx/master/images/nx-logo.png" width="45"></a>
✨ This is Virginia Tech's Echolab [Counting app research project](https://github.com/echo-lab/Counting-App) but within an [NX](https://nx.dev/getting-started/tutorials/react-monorepo-tutorial?utm_source=nx_project&amp;utm_medium=readme&amp;utm_campaign=nx_projects) workspace.

✨ Your new, shiny [Nx workspace](https://nx.dev) is almost ready ✨.
<a alt="Nx logo" href="https://nx.dev" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/nrwl/nx/master/images/nx-logo.png" width="45"></a>
shiny [Nx workspace](https://nx.dev) is almost ready ✨.

[Learn more about this workspace setup and its capabilities](https://nx.dev/getting-started/tutorials/react-monorepo-tutorial?utm_source=nx_project&amp;utm_medium=readme&amp;utm_campaign=nx_projects) or run `npx nx graph` to visually explore what was created. Now, let's get you up to speed!

Expand Down
3 changes: 3 additions & 0 deletions apps/backend/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,8 @@
}
}
}
},
"dependencies": {
"wav": "^1.0.2"
}
}
8 changes: 7 additions & 1 deletion apps/backend/src/server.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ import fetch from 'node-fetch';
import cors from 'cors';
import path from 'path';
import { fileURLToPath } from 'url';
import { Writer } from 'wav';
import { PassThrough } from 'stream';
//import { config } from 'dotenv'; // might move it before importing the db.js file

// change from apps/backend/.env.local to .env.local to enable environment variables to be loaded on server
Expand Down Expand Up @@ -245,7 +247,11 @@ app.post("/register", async (req, res) => {
});

const data = await response.json();
res.json(data);
//res.json(data);
const audioBuffer = Buffer.from(data.audioContent, 'base64');
res.setHeader('Content-Type', 'audio/mpeg');
res.end(audioBuffer);

} catch (error) {
console.error('Server Error in Google Text-to-Speech:', error);
res.status(500).json({ message: error.toString() });
Expand Down
4 changes: 2 additions & 2 deletions apps/frontend/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="/favicon.ico" />
<link rel="icon" href="/logoecholabs.png" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="Counting App"
/>
<link rel="apple-touch-icon" href="/logo192.png" />
<link rel="apple-touch-icon" href="/logoecholabs.png" />
<link rel="manifest" href="/manifest.json" />
<title>Counting App</title>
</head>
Expand Down
Binary file added apps/frontend/logoecholab.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions apps/frontend/src/App.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,23 @@ function App() {
document.removeEventListener("contextmenu", handleContextMenu);
};
}, []);


useEffect(() => {
const unlockAudio = () => {
const silent = new Audio('data:audio/mp3;base64,SUQzBAAAAAABEVRYWFgAAAAtAAADY29tbWVudABCaWdTb3VuZEJhbmsuY29tIC8gTGFTb25vdGhlcXVlLm9yZwBURU5DAAAAHQAAA1N3aXRjaCBQbHVzIMHumPMHBgAAA');
silent.play().catch(() => {});
document.body.removeEventListener('click', unlockAudio);
document.body.removeEventListener('touchend', unlockAudio);
};
document.body.addEventListener('click', unlockAudio);
document.body.addEventListener('touchend', unlockAudio);
return () => {
document.body.removeEventListener('click', unlockAudio);
document.body.removeEventListener('touchend', unlockAudio);
};
}, []);

return (
<div className="App">
<SoundProvider>
Expand Down
132 changes: 84 additions & 48 deletions apps/frontend/src/helpers/textToSpeech.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ export async function textToSpeech(utterance, onSpeechEnd) {
return;
}

if (currentAudio) {
currentAudio.pause(); // Stop any currently playing audio
currentAudio = null;
}

try {
const requestData = {
text: utterance,
Expand All @@ -27,64 +32,95 @@ export async function textToSpeech(utterance, onSpeechEnd) {
});

console.log('Response status:', response.status);

// expected response is a blob
const audioBlob = await response.blob();
const audioUrl = URL.createObjectURL(audioBlob);
const audio = new Audio(audioUrl);
currentAudio = audio;

audio.addEventListener('ended', () => {
currentAudio = null;
// Clean up the temporary URL to release memory
URL.revokeObjectURL(audioUrl);
if (typeof onSpeechEnd === 'function') {
onSpeechEnd();
}
});

if (!response.ok) {
const errorText = await response.text();
console.error('Server error response:', {
status: response.status,
statusText: response.statusText,
errorText
});
return;
}
audio.addEventListener('error', (e) => {
console.error('Audio element error:', e);
URL.revokeObjectURL(audioUrl); // Also clean up on error
});

try {
await audio.play();
console.log('Playback started successfully');
} catch (playError){
console.error('Playback failed:', {
name: playError.name,
message: playError.message
});
URL.revokeObjectURL(audioUrl);
}

// if (!response.ok) {
// const errorText = await response.text();
// console.error('Server error response:', {
// status: response.status,
// statusText: response.statusText,
// errorText
// });
// return;
// }

const data = await response.json();
if (currentAudio) {
currentAudio.pause();
currentAudio = null;
}
// const data = await response.json();
// if (currentAudio) {
// currentAudio.pause();
// currentAudio = null;
// }

console.log('Parsed response data:', data);
// console.log('Parsed response data:', data);

// Validate audio content
if (!data || !data.audioContent) {
console.error('No audio content in response', data);
return;
}
// // Validate audio content
// if (!data || !data.audioContent) {
// console.error('No audio content in response', data);
// return;
// }

// Stop any currently playing audio
if (currentAudio) {
currentAudio.pause();
currentAudio = null;
}
// // Stop any currently playing audio
// if (currentAudio) {
// currentAudio.pause();
// currentAudio = null;
// }

const audioSrc = `data:audio/mp3;base64,${data.audioContent}`;
// const audioSrc = `data:audio/mp3;base64,${data.audioContent}`;

const audio = new Audio(audioSrc);
currentAudio = audio;
// const audio = new Audio(audioSrc);
// currentAudio = audio;

// Set up event listeners
audio.addEventListener('error', (e) => {
console.error('Audio element error:', e);
currentAudio = null;
});
// // Set up event listeners
// audio.addEventListener('error', (e) => {
// console.error('Audio element error:', e);
// currentAudio = null;
// });

audio.addEventListener('ended', () => {
currentAudio = null;
if (typeof onSpeechEnd === 'function') {
onSpeechEnd();
}
});
// audio.addEventListener('ended', () => {
// currentAudio = null;
// if (typeof onSpeechEnd === 'function') {
// onSpeechEnd();
// }
// });

try {
await audio.play();
console.log('Playback started successfully');
} catch (playError) {
console.error('Playback failed:', {
name: playError.name,
message: playError.message
});
}
// try {
// await audio.play();
// console.log('Playback started successfully');
// } catch (playError) {
// console.error('Playback failed:', {
// name: playError.name,
// message: playError.message
// });
// }

} catch (error) {
console.error('Comprehensive error in Text-to-Speech:', {
Expand Down
30 changes: 15 additions & 15 deletions apps/frontend/src/pages/TouchTrainingPage.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,21 +248,21 @@ const TouchTrainingPage = () => {
const handleTrayClick = (trayType) => {
setSelectedTray(trayType);
// if the correct tray has been clicked
if (trayType === "greenTray" && sectionTrainData.pages[currentPage].greenTray[0].biscuits.length === sectionTrainData.pages[currentPage].cookies.length) {
textToSpeech("Green is correct, Good job!");
}
else if (trayType === "purpleTray" && sectionTrainData.pages[currentPage].purpleTray[0].biscuits.length === sectionTrainData.pages[currentPage].cookies.length){
textToSpeech("Purple is correct, Well done!");
}
// if the wrong tray has been clicked
else if (trayType === "greenTray" && sectionTrainData.pages[currentPage].greenTray[0].biscuits.length !== sectionTrainData.pages[currentPage].cookies.length) {
const explanation = `No, ${trayType} has ${sectionTrainData.pages[currentPage].greenTray[0].biscuits.length} cookies. Try again!`;
textToSpeech(explanation);
}
else{
const explanation = `Wrong answer, ${trayType} has ${sectionTrainData.pages[currentPage].purpleTray[0].biscuits.length} cookies. Try again!`;
textToSpeech(explanation);
}
// if (trayType === "greenTray" && sectionTrainData.pages[currentPage].greenTray[0].biscuits.length === sectionTrainData.pages[currentPage].cookies.length) {
// textToSpeech("Green is correct, Good job!");
// }
// else if (trayType === "purpleTray" && sectionTrainData.pages[currentPage].purpleTray[0].biscuits.length === sectionTrainData.pages[currentPage].cookies.length){
// textToSpeech("Purple is correct, Well done!");
// }
// // if the wrong tray has been clicked
// else if (trayType === "greenTray" && sectionTrainData.pages[currentPage].greenTray[0].biscuits.length !== sectionTrainData.pages[currentPage].cookies.length) {
// const explanation = `No, ${trayType} has ${sectionTrainData.pages[currentPage].greenTray[0].biscuits.length} cookies. Try again!`;
// textToSpeech(explanation);
// }
// else{
// const explanation = `Wrong answer, ${trayType} has ${sectionTrainData.pages[currentPage].purpleTray[0].biscuits.length} cookies. Try again!`;
// textToSpeech(explanation);
// }
storeAnswer(currentPage, trayType);
};

Expand Down
30 changes: 15 additions & 15 deletions apps/frontend/src/pages/basePage.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,21 +85,21 @@ function basePage() {
const handleTrayClick = (trayType) => {
setSelectedTray(trayType);
// if the correct tray has been clicked
if (trayType === "greenTray" && baseData.pages[currentPage].greenTray[0].biscuits.length === baseData.pages[currentPage].cookies.length) {
textToSpeech("Green is correct, Good job!");
}
else if (trayType === "purpleTray" && baseData.pages[currentPage].purpleTray[0].biscuits.length === baseData.pages[currentPage].cookies.length){
textToSpeech("Purple is correct, Well done!");
}
// if the wrong tray has been clicked
else if (trayType === "greenTray" && baseData.pages[currentPage].greenTray[0].biscuits.length !== baseData.pages[currentPage].cookies.length) {
const explanation = `No, ${trayType} has ${baseData.pages[currentPage].greenTray[0].biscuits.length} cookies. Try again!`;
textToSpeech(explanation);
}
else{
const explanation = `Wrong answer, ${trayType} has ${baseData.pages[currentPage].purpleTray[0].biscuits.length} cookies. Try again!`;
textToSpeech(explanation);
}
// if (trayType === "greenTray" && baseData.pages[currentPage].greenTray[0].biscuits.length === baseData.pages[currentPage].cookies.length) {
// textToSpeech("Green is correct, Good job!");
// }
// else if (trayType === "purpleTray" && baseData.pages[currentPage].purpleTray[0].biscuits.length === baseData.pages[currentPage].cookies.length){
// textToSpeech("Purple is correct, Well done!");
// }
// // if the wrong tray has been clicked
// else if (trayType === "greenTray" && baseData.pages[currentPage].greenTray[0].biscuits.length !== baseData.pages[currentPage].cookies.length) {
// const explanation = `No, ${trayType} has ${baseData.pages[currentPage].greenTray[0].biscuits.length} cookies. Try again!`;
// textToSpeech(explanation);
// }
// else{
// const explanation = `Wrong answer, ${trayType} has ${baseData.pages[currentPage].purpleTray[0].biscuits.length} cookies. Try again!`;
// textToSpeech(explanation);
// }
storeAnswer(currentPage, trayType);

};
Expand Down
Loading
Loading