XpressiveAI

XpressiveAI is a real-time deep learning project that bridges a crucial gap in communication. It translates sign language to spoken words while also capturing the signer's emotions from their facial expressions. By giving both words and feelings a voice, this system creates a richer, more human connection for the deaf and hard-of-hearing community.

---

Table of Contents

1. Project Directory Structure
2. Model Architecture
3. Results
4. Dataset Links
5. Installation and Setup
6. Running the Project
7. Custom Dataset for Sign Language
8. Prototyping

---

Project Directory Structure

XpressiveAI/
|
├── captures/                           # Sample images captured during pipeline execution
|   ├── face/                           # Captured face frames
|   ├── frame/                          # Full captured frames
|   └── hand/                           # Captured hand gestures
|
├── custom_dataset/                     # Tools for creating custom sign language dataset
|   ├── sign_dataset_web.py             # FastAPI web app for capturing sign images
|   └── ...
|
├── data/                               # Dataset storage directory
|   ├── final_dataset/                  # Preprocessed emotion images (7 classes)
|   |   ├── anger/
|   |   ├── disgust/
|   |   ├── fear/
|   |   ├── happy/
|   |   ├── neutral/
|   |   ├── sad/
|   |   └── surprise/
|   |
|   ├── final_split_dataset/            # Train/test/val split
|   |   ├── test/
|   |   ├── train/
|   |   └── val/
|   |
|   └── final_mapping.csv               # Image to label mapping
|
├── MediaPipe/                          # Sign language recognition module
|   ├── artifacts/                      # Trained sign language model and labels
|   |   ├── asl_mlp.keras               # Sign language classifier model
|   |   └── labels.json                 # Label mapping for sign classes
|   ├── requirements.txt                # MediaPipe module dependencies
|   └── src/                            # MediaPipe source code
|       ├── config.py                   # Configuration settings
|       ├── features.py                 # Hand feature extraction
|       ├── train.py                    # Sign language model training
|       ├── evaluate.py                 # Model evaluation
|       └── live_demo.py                # Real-time sign language demo
|
├── new_weights/                        # Trained emotion recognition model weights
|   ├── best_resnet50_agfn.keras        # Best performing model checkpoint
|   └── resnet50_agfn_final.keras       # Final trained model
|
├── old_weights/                        # Previous model weight versions
|
├── Research/                           # Research artifacts and analysis
|   ├── model_architecture_info/        # Model architecture diagrams and details
|   ├── metrics/                        # Performance metrics and evaluation results
|   └── charts_plot.py                  # Visualization scripts
|
├── src/                                # Main source code directory
|   ├── data_scripts/                   # Data preprocessing scripts
|   |   ├── emotion_dataset_preprocess.py   # Emotion dataset preprocessing
|   |   ├── emotion_dataset_splitting.py    # Train/test/val splitting
|   |   ├── download_datasets.py            # Download datasets from Kaggle
|   |   ├── preprocess_asl_alphabet.py      # ASL alphabet preprocessing
|   |   ├── preprocess_asl_digits.py        # ASL digits preprocessing
|   |   ├── preprocess_sign_mnist.py        # Sign MNIST preprocessing
|   |   ├── augment_sign_data.py            # Sign language data augmentation
|   |   └── combine_sign_csv.py             # Combine sign language CSVs
|   |
|   ├── model_scripts/                  # Model training scripts
|   |   ├── resnet50_afgn_train.py      # ResNet50-AGFN emotion model training
|   |   ├── emotion_model_train.py      # Alternative emotion model training
|   |   └── resnet_50.py                # Baseline ResNet50 model
|   |
|   ├── inference/                      # Inference and evaluation scripts
|   |   ├── realtime_emotion_inference.py   # Real-time emotion recognition
|   |   ├── resnet50_agfn_infer.py          # Batch inference for ResNet50-AGFN
|   |   ├── baseline_evaluation.py          # Baseline model evaluation
|   |   └── test_model.py                   # Model testing utilities
|   |
|   └── pipeline/                       # Video capture pipeline
|       └── liveCameraCapture.py        # Live camera capture for face/hand
|
├── prototype/                          # Prototyping scripts
|
├── .gitignore                          # Git ignore file
├── README.md                           # This file
├── requirements.txt                    # Main project dependencies
└── requirements_emotion.txt            # Emotion model specific dependencies

---

Model Architecture

XpressiveAI System Architecture

The complete XpressiveAI system combines two deep learning models for real-time sign language and emotion recognition:

Emotion Recognition Model - ResNet50-AGFN

The emotion recognition module uses a ResNet50 backbone enhanced with Attention-Guided Feature Network (AGFN) for improved facial expression classification:

Sign Language Recognition - MediaPipe

The sign language module uses Google MediaPipe for hand landmark detection combined with an MLP classifier:

---

Results

Emotion Recognition Performance

ResNet50-AGFN Model (Our Model)

Emotion	Precision	Recall	F1-Score	Support
Anger	0.854	0.803	0.828	1125
Disgust	0.935	0.990	0.962	1125
Fear	0.823	0.692	0.752	1125
Happy	0.967	0.964	0.965	1125
Sad	0.727	0.804	0.764	1125
Surprise	0.903	0.981	0.940	1125
Neutral	0.812	0.788	0.800	1125
Overall Accuracy			86.03%	7875

Baseline ResNet50 Model (Comparison)

Emotion	Precision	Recall	F1-Score	Support
Anger	0.783	0.751	0.767	1125
Disgust	0.927	0.948	0.937	1125
Fear	0.807	0.599	0.688	1125
Happy	0.867	0.912	0.889	1125
Sad	0.716	0.709	0.713	1125
Surprise	0.884	0.863	0.874	1125
Neutral	0.666	0.838	0.743	1125
Overall Accuracy			80.29%	7875

Performance Improvement

The ResNet50-AGFN model achieves 5.74% improvement in overall accuracy compared to the baseline ResNet50 model.

---

Dataset Links

Emotion Recognition Datasets

Dataset	Images	Original Size	Processed Size
FER2013	35,887	48x48	224x224
CK+	981	48x48	224x224
RAF-DB	15,339	100x100	224x224

Sign Language Datasets

Dataset	Images	Original Size	Processed Size
Sign Language MNIST	34,627	28x28	48x48
ASL Alphabet	87,000	200x200 RGB	48x48
ASL Digits (0-9)	3,000	100x100 RGB	48x48

Emotion Datasets Summary and Universal Class Mapping

Label	Emotion	FER Count	CK+ Count	RAF Count	Total Count
0	anger	4,953	135	867	5,955
1	disgust	547	177	877	1,601
2	fear	5,121	75	355	5,551
3	happy	8,989	207	5,957	15,153
4	sad	6,077	84	2,460	8,621
5	surprise	4,002	249	1,619	5,870
6	neutral	6,198	-	3,204	9,402
7	contempt	-	54	-	54
Total		35,887	981	15,339	52,207

Note: Contempt class (54 images) is excluded from training due to class imbalance.

RAF-DB Label Mapping

RAF-DB Label	Emotion	Project Label
1	surprise	5
2	fear	2
3	disgust	1
4	happy	3
5	sad	4
6	anger	0
7	neutral	6

---

Installation and Setup

Prerequisites

• Python 3.10 or higher
• Webcam for real-time inference
• CUDA-compatible GPU (optional, for faster training)

Step 1: Create Virtual Environment

python -m venv .venv

Activate the environment:

Windows:

.\.venv\Scripts\activate

Linux/macOS:

source .venv/bin/activate

Step 2: Install Dependencies

For the main project:

pip install -r requirements.txt

For emotion model training specifically:

pip install -r requirements_emotion.txt

For MediaPipe sign language module:

cd MediaPipe
pip install -r requirements.txt
cd ..

Step 3: Configure Kaggle API (for dataset download)

1. Go to Kaggle Settings
2. Create an API key and download kaggle.json
3. Place the file at:
   • Windows: C:/Users/<username>/.kaggle/kaggle.json
   • Linux/macOS: ~/.kaggle/kaggle.json

---

Running the Project

Option 1: Real-Time Emotion Recognition

Run the real-time emotion inference using your webcam:

python src/inference/realtime_emotion_inference.py

Optional arguments:

• First argument: Camera index (default: 0)
• Second argument: Minimum face size (default: 80)

Example:

python src/inference/realtime_emotion_inference.py 0 100

Controls:

• Press q or ESC to quit

Option 2: Real-Time Sign Language Recognition

Run the sign language live demo:

cd MediaPipe
python -m src.live_demo

Controls:

• Press ESC to quit
• Press c to clear typed text
• Press w to save typed text to file

Option 3: Combined Pipeline (Face and Hand Capture)

To capture face and hand frames from the camera:

python src/pipeline/liveCameraCapture.py

Controls:

• Press s to capture frames
• Captured images are saved to the captures/ folder

---

Training the Models

Emotion Recognition Model

Step 1: Download and Preprocess Emotion Datasets

python src/data_scripts/emotion_dataset_preprocess.py

This will:

1. Download FER2013, CK+, and RAF-DB datasets from Kaggle
2. Preprocess and resize images to 224x224
3. Apply universal label mapping
4. Store processed images in data/final_dataset/

Step 2: Split Dataset

python src/data_scripts/emotion_dataset_splitting.py

This creates train/test/val splits:

Class	Test	Train	Validation
Per class	1,125	5,250	1,125
Total	7,875	36,750	7,875

Step 3: Train the Model

python src/model_scripts/resnet50_afgn_train.py

Trained weights will be saved in the new_weights/ directory.

Sign Language Model

Step 1: Download Datasets

python src/data_scripts/download_datasets.py

Step 2: Preprocess Datasets

Run in order:

python src/data_scripts/preprocess_sign_mnist.py
python src/data_scripts/preprocess_asl_alphabet.py
python src/data_scripts/preprocess_asl_digits.py

Step 3: Combine CSV Files

python src/data_scripts/combine_sign_csv.py

Step 4: Augment Data

python src/data_scripts/augment_sign_data.py

Step 5: Train the Model

cd MediaPipe
python -m src.train

---

Custom Dataset for Sign Language

You can create your own sign language dataset using the built-in web capture tool.

Step 1: Start the FastAPI Server

cd custom_dataset
uvicorn sign_dataset_web:app --reload

Step 2: Capture Images

Open your browser and navigate to:

http://127.0.0.1:8000

Controls:

• Press c to capture images
• Press n to move to the next letter

Step 3: Stop the Server

Once you have captured images for all letters (A to Z), stop the server with Ctrl + C.

---

Prototyping

For quick prototyping and experiments:

cd prototype
python -m venv .sub_venv

Activate the virtual environment:

Windows:

.\.sub_venv\Scripts\activate

Linux/macOS:

source .sub_venv/bin/activate

Install prototype dependencies:

pip install -r prototype_req.txt

Run prototype scripts as needed.

---

Model Evaluation

Evaluate Emotion Model

To evaluate the trained ResNet50-AGFN model:

python src/inference/resnet50_agfn_infer.py

To compare with baseline:

python src/inference/baseline_evaluation.py

Evaluate Sign Language Model

cd MediaPipe
python -m src.evaluate

---

License

This project is developed for academic and research purposes.

---

Acknowledgments

• FER2013, CK+, and RAF-DB dataset creators
• Sign Language MNIST, ASL Alphabet, and ASL Digits dataset creators
• Google MediaPipe team
• TensorFlow and Keras communities