Word2Vec Training and Gender Bias Analysis

This project implements and trains Word2Vec models (CBOW and Skip-gram) from scratch and performs gender bias analysis on word embeddings using GloVe embeddings.

Overview

The project is divided into two parts:

Part 1: Word2Vec Model Training

• CBOW (Continuous Bag of Words): Predicts center words from surrounding context
• Skip-gram: Predicts surrounding words from center words
• Both models use shared embedding weights between input and output layers
• Models are trained from scratch on a text corpus

Part 2: Gender Bias Analysis

• Gender Subspace Identification: Uses PCA to identify gender bias direction in embedding space

• Direct Bias Measurement: Quantifies gender bias in profession embeddings

• Hard Debiasing: Removes gender bias from embeddings using subspace projection

• WEAT (Word Embedding Association Test): Measures implicit bias in word associations

Model Architecture

SharedNNLM Base Class

Both CBOW and Skip-gram share the same underlying architecture:

• Embedding Layer: Maps word indices to dense vectors (300 dimensions)
• Projection Layer: Maps embeddings to vocabulary logits (shared weights with embedding)
• No Hidden Layers: Simplified NNLM architecture without hidden layers

CBOW Model

• Input: Lists of surrounding word indices (window size = 3)
• Processing: Sums embeddings of surrounding words
• Output: Predicts center word
• Loss: Cross-entropy over vocabulary

Skip-gram Model

• Input: Center word indices

• Processing: Direct embedding lookup

• Output: Predicts surrounding words

• Loss: Cross-entropy over vocabulary

Training Results

CBOW Training Performance

• Initial Loss: 60.22
• Final Loss: 4.81

Skip-gram Training Performance

• Initial Loss: 10.16

• Final Loss: 5.53

Model Evaluation Results

Word Similarity Test (Query: "man")

CBOW Results: ['woman', 'a', 'another', 'in', 'is']

Skip-gram Results: ['a', 'woman', 'is', 'in', 'and']

Word Analogy Test (man - woman + girl = ?)

Expected Answer: 'boy'

CBOW Results: ['boy', 'little', 'a', 'young', 'is']

Skip-gram Results: ['boy', 'little', 'young', 'is', 'a']

Analysis

• Both models successfully solve the analogy task

• Skip-gram shows slightly better ranking of semantically related words

• Function words still appear but don't dominate the results

• Models capture gender relationships (man vs woman) and age relationships (girl vs boy)

Gender Bias Analysis Results

Gender Subspace

• Dimension: (1, 300) - Single principal component extracted
• Method: PCA on normalized gender attribute word pairs
• Purpose: Identifies the primary direction of gender bias in embedding space

Scalar Projection Analysis

Word "man" onto Gender Direction: -1.5679 -> association with masculine direction

Profession Gender Bias

Most Feminine Professions :

1. dietitian
2. housekeeper
3. receptionist
4. childcare worker
5. paralegal
6. registered nurse
7. vocational nurse
8. hairdresser
9. dental hygienist
10. salesperson

Most Masculine Professions :

1. operating engineer
2. secretary
3. conductor
4. crossing guard
5. director of religious activities
6. mobile equipment mechanic
7. lodging manager
8. security system installer
9. judge
10. mining machine operator

Direct Bias Metric: 0.5261

• Scale: 0 (no bias) to 1 (maximum bias)

• Interpretation: High gender bias detected in profession embeddings

Debiasing Results

Hard Debiasing Application

• Method: Subtracts projections onto gender subspace from all embeddings

Post-Debiasing Measurements

Direct Bias (Debiased): 0.00

• Reduction: From 0.5261 -> 0.00
• Success: Hard debiasing effectively neutralizes gender bias in embeddings

Permutation Test (Math/Arts Stereotype)

Test Configuration:

• Target Sets: X = [math, algebra, geometry, ...], Y = [poetry, art, dance, ...]
• Purpose: Tests for math-male/arts-female stereotype after debiasing

P-value: 0.03

• Interpretation: Statistically significant association remains (p < 0.05)

WEAT Analysis

Test Configuration:

• Target Sets: X = ["doctor", "mechanic", "engineer"], Y = ["nurse", "artist", "teacher"]
• Attribute Sets: A = ["male", "man", "boy", ...], B = ["female", "woman", "girl", ...]

Results:

• WEAT Association (doctor): 0.0175
  • Positive value indicates 'doctor' is more associated with masculine attributes
• WEAT Differential Association: 0.3557
  • Strong positive value indicates systematic bias
  • X words (doctor, mechanic, engineer) are more associated with masculine attributes
  • Y words (nurse, artist, teacher) are more associated with feminine attributes

References

1. Word2Vec: Mikolov et al., "Efficient Estimation of Word Representations in Vector Space", 2013. https://arxiv.org/abs/1301.3781
2. GloVe: Pennington et al., "GloVe: Global Vectors for Word Representation", 2014. https://nlp.stanford.edu/projects/glove/
3. WinoBias: Zhao et al., "Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods", 2018. https://aclanthology.org/N18-2003/
4. BERT Gender Bias: Bartl et al., "Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias", 2020. https://arxiv.org/abs/2010.14534