Feature Proposal: Slang & Roman Urdu Normalization for spaCy

[MudassarGill]

Feature Proposal: Smart Text Normalizer for Roman Urdu & English Slang

Problem Statement

Current NLP pipelines struggle with informal text from social media, WhatsApp chats, and casual conversations. While spaCy handles standard English well, there's no built-in support for:

1. English slang normalization (u → you, idk → I don't know, gonna → going to)
2. Roman Urdu normalization (kya → what, hai → is, bhai → brother)

Roman Urdu is especially important for South Asian languages - millions of people write Urdu using English script on social media, but no major NLP library supports it.

Example

Input:
u kya kar raha hai bhai? idk btw

text

Current spaCy Output (without feature):
u → u
kya → kya
kar → kar
raha → raha
hai → hai
bhai → bhai
idk → idk
btw → btw

text

Expected Output (with feature):
u → you
kya → what
kar → kar
raha → going
hai → is
bhai → brother
idk → I don't know
btw → by the way

text

Proposed Implementation

1. Roman Urdu Dictionary (New File: spacy/lang/ur/norm_exceptions.py)

NORM_EXCEPTIONS = {
    # Pronouns
    "main": "I",
    "tum": "you",
    "aap": "you",
    "hum": "we",
    "mera": "my",
    "tera": "your",
    
    # Common verbs
    "hai": "is",
    "hain": "are",
    "tha": "was",
    "the": "were",
    "ho": "are",
    "karna": "do",
    "karo": "do",
    "ja": "go",
    "jao": "go",
    "aa": "come",
    "aao": "come",
    "raha": "going",
    "rahi": "going",
    
    # Question words
    "kya": "what",
    "kaise": "how",
    "kahan": "where",
    "kyun": "why",
    "kab": "when",
    "kon": "who",
    
    # Negations
    "nahi": "not",
    "mat": "don't",
    
    # Family/Relations
    "bhai": "brother",
    "behen": "sister",
    "maa": "mother",
    "baap": "father",
    "beta": "son",
    "beti": "daughter",
    
    # Time words
    "kal": "tomorrow/yesterday",
    "aaj": "today",
    "ab": "now",
    
    # Common words
    "acha": "good",
    "theek": "fine",
    "sahi": "correct",
    "galat": "wrong",
    "haan": "yes",
    "shukriya": "thanks",
    "meherbani": "please",
    "scene": "situation",
    "baat": "talk",
    "naam": "name",
}
2. English Slang Dictionary (Add to spacy/lang/en/norm_exceptions.py)
python
# Add these to existing NORM_EXCEPTIONS
{
    # Texting abbreviations
    "u": "you",
    "ur": "your",
    "idk": "I don't know",
    "idc": "I don't care",
    "imo": "in my opinion",
    "imho": "in my humble opinion",
    "tbh": "to be honest",
    "btw": "by the way",
    "fyi": "for your information",
    "asap": "as soon as possible",
    "aka": "also known as",
    "dm": "direct message",
    "pm": "private message",
    
    # Social media expressions
    "lol": "laughing out loud",
    "lmao": "laughing my ass off",
    "rofl": "rolling on the floor laughing",
    "omg": "oh my god",
    "wtf": "what the fuck",
    "wth": "what the hell",
    "smh": "shaking my head",
    
    # Shortened words
    "gonna": "going to",
    "wanna": "want to",
    "gotta": "got to",
    "kinda": "kind of",
    "sorta": "sort of",
    "outta": "out of",
    "lemme": "let me",
    "gimme": "give me",
    
    # Thanks/Please
    "thx": "thanks",
    "ty": "thank you",
    "tysm": "thank you so much",
    "np": "no problem",
    "yw": "you're welcome",
    "pls": "please",
    "plz": "please",
    
    # Others
    "bc": "because",
    "cuz": "because",
    "afaik": "as far as I know",
    "irl": "in real life",
    "jk": "just kidding",
    "brb": "be right back",
    "afk": "away from keyboard",
    "gtg": "got to go",
    "ttyl": "talk to you later",
}
3. Urdu Language Support (New File: spacy/lang/ur/__init__.py)
python
from spacy.language import Language
from spacy.attrs import LANG, NORM
from spacy.lang.tokenizer_exceptions import BASE_EXCEPTIONS
from spacy.util import add_lookups
from .norm_exceptions import NORM_EXCEPTIONS

class UrduDefaults(Language.Defaults):
    lex_attr_getters = dict(Language.Defaults.lex_attr_getters)
    lex_attr_getters[LANG] = lambda text: "ur"
    lex_attr_getters[NORM] = add_lookups(
        Language.Defaults.lex_attr_getters[NORM],
        NORM_EXCEPTIONS
    )
    tokenizer_exceptions = BASE_EXCEPTIONS.copy()

class Urdu(Language):
    lang = "ur"
    Defaults = UrduDefaults

__all__ = ["Urdu"]




Why This Should Be in Core spaCy
No heavy dependencies - Just dictionary files

Uses existing norm_exceptions mechanism - Follows spaCy's patterns

Improves all downstream tasks - Better POS tagging, NER after normalization

Addresses real-world use case - Social media, customer support, chat analysis

Unique value proposition - First major NLP library with Roman Urdu support

Implementation Plan
If approved, I'll submit a PR with:

 Urdu language module with Roman Urdu norm exceptions

 Extended English slang exceptions

 Comprehensive tests for both features

 Documentation updates

Questions for Maintainers
Is this approach acceptable for core inclusion?

Any concerns about adding slang to the core library?

Suggestions for dictionary expansion or validation?

Looking forward to feedback!