Installation
pip install git+https://github.com/crazyfrogspb/RedditScore.git
Steps to reproduce
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer(hashtags=False)
text = "Let's #makeamericagreatagain#americafirst"
print(tokenizer.tokenize(text))
Expected output
["let's", "#makeamericagreatagain", "#americafirst"]
Actual output
['let', "'s", '#makeamericagreatagain', '#americafirst']
Environment
Kaggle's Notebook
Installation
Steps to reproduce
Expected output
Actual output
Environment
Kaggle's Notebook