Turkish Spell Checker

This tool is a spelling checker for Modern Turkish. It detects spelling errors and corrects them appropriately, through its list of misspellings and matching to the Turkish dictionary.

Simple Web Interface

Link 1 Link 2

Video Lectures

For Developers

You can also see Python, Java, C++, C, Swift, Js, or C# repository.

Requirements

• Python 3.7 or higher
• Git

Python

To check if you have a compatible version of Python installed, use the following command:

python -V

You can find the latest version of Python here.

Git

Install the latest version of Git.

Pip Install

pip3 install NlpToolkit-SpellChecker-Cy

Download Code

In order to work on code, create a fork from GitHub page. Use Git for cloning the code to your local or below line for Ubuntu:

git clone <your-fork-git-link>

A directory called SpellChecker will be created. Or you can use below link for exploring the code:

git clone https://github.com/starlangsoftware/TurkishSpellChecker-Cy.git

Open project with Pycharm IDE

Steps for opening the cloned project:

• Start IDE
• Select File | Open from main menu
• Choose DataStructure-CY file
• Select open as project option
• Couple of seconds, project will be downloaded.

Detailed Description

• Creating SpellChecker
• Spell Correction

Creating SpellChecker

SpellChecker finds spelling errors and corrects them in Turkish. There are two types of spell checker available:

• SimpleSpellChecker

  • To instantiate this, a FsmMorphologicalAnalyzer is needed.

      fsm = FsmMorphologicalAnalyzer()
      spellChecker = SimpleSpellChecker(fsm)   
    

• NGramSpellChecker,

  • To create an instance of this, both a FsmMorphologicalAnalyzer and a NGram is required.

  • FsmMorphologicalAnalyzer can be instantiated as follows:

      fsm = FsmMorphologicalAnalyzer()
    

  • NGram can be either trained from scratch or loaded from an existing model.

    • Training from scratch:

        corpus = Corpus("corpus.txt");
        ngram = NGram(corpus.getAllWordsAsArrayList(), 1)
        ngram.calculateNGramProbabilities(LaplaceSmoothing())
      

    There are many smoothing methods available. For other smoothing methods, check here.

    • Loading from an existing model:

        ngram = NGram("ngram.txt")
      

  For further details, please check here.

  • Afterwards, NGramSpellChecker can be created as below:

      spellChecker = NGramSpellChecker(fsm, ngram)
    

Spell Correction

Spell correction can be done as follows:

sentence = Sentence("Dıktor olaç yazdı")
corrected = spellChecker.spellCheck(sentence)
print(corrected)

Output:

Doktor ilaç yazdı

For Contibutors

Setup.py file

1. Do not forget to set package list. All subfolders should be added to the package list.

    packages=['Classification', 'Classification.Model', 'Classification.Model.DecisionTree',
              'Classification.Model.Ensemble', 'Classification.Model.NeuralNetwork',
              'Classification.Model.NonParametric', 'Classification.Model.Parametric',
              'Classification.Filter', 'Classification.DataSet', 'Classification.Instance', 'Classification.Attribute',
              'Classification.Parameter', 'Classification.Experiment',
              'Classification.Performance', 'Classification.InstanceList', 'Classification.DistanceMetric',
              'Classification.StatisticalTest', 'Classification.FeatureSelection'],

2. Package name should be lowercase and only may include _ character.

    name='nlptoolkit_math',

3. Package data should be defined and must ibclude pyx, pxd, c and py files.

    package_data={'NGram': ['*.pxd', '*.pyx', '*.c', '*.py']},

4. Setup should include ext_modules with compiler directives.

    ext_modules=cythonize(["NGram/*.pyx"],
                          compiler_directives={'language_level': "3"}),

Cython files

1. Define the class variables and class methods in the pxd file.

cdef class DiscreteDistribution(dict):

    cdef float __sum

    cpdef addItem(self, str item)
    cpdef removeItem(self, str item)
    cpdef addDistribution(self, DiscreteDistribution distribution)

2. For default values in class method declarations, use *.

    cpdef list constructIdiomLiterals(self, FsmMorphologicalAnalyzer fsm, MorphologicalParse morphologicalParse1,
                               MetamorphicParse metaParse1, MorphologicalParse morphologicalParse2,
                               MetamorphicParse metaParse2, MorphologicalParse morphologicalParse3 = *,
                               MetamorphicParse metaParse3 = *)

3. Define the class name as cdef, class methods as cpdef, and __init__ as def.

cdef class DiscreteDistribution(dict):

    def __init__(self, **kwargs):
        """
        A constructor of DiscreteDistribution class which calls its super class.
        """
        super().__init__(**kwargs)
        self.__sum = 0.0

    cpdef addItem(self, str item):

4. Do not forget to comment each function.

    cpdef addItem(self, str item):
        """
        The addItem method takes a String item as an input and if this map contains a mapping for the item it puts the
        item with given value + 1, else it puts item with value of 1.

        PARAMETERS
        ----------
        item : string
            String input.
        """

5. Function names should follow caml case.

    cpdef addItem(self, str item):

6. Local variables should follow snake case.

	det = 1.0
	copy_of_matrix = copy.deepcopy(self)

7. Variable types should be defined for function parameters, class variables.

    cpdef double getValue(self, int rowNo, int colNo):

8. Local variables should be defined with types.

    cpdef sortDefinitions(self):
        cdef int i, j
        cdef str tmp

9. For abstract methods, use ABC package and declare them with @abstractmethod.

    @abstractmethod
    def train(self, train_set: list[Tensor]):
        pass

10. For private methods, use __ as prefix in their names.

    cpdef list __linearRegressionOnCountsOfCounts(self, list countsOfCounts)

11. For private class variables, use __ as prefix in their names.

cdef class NGram:
    cdef int __N
    cdef double __lambda1, __lambda2
    cdef bint __interpolated
    cdef set __vocabulary
    cdef list __probability_of_unseen

12. Write __repr__ class methods as toString methods
13. Write getter and setter class methods.

    cpdef int getN(self)
    cpdef setN(self, int N)

14. If there are multiple constructors for a class, define them as constructor1, constructor2, ..., then from the original constructor call these methods.

cdef class NGram:

    cpdef constructor1(self, int N, list corpus):
    cpdef constructor2(self, str fileName):
    def __init__(self,
                 NorFileName,
                 corpus=None):
        if isinstance(NorFileName, int):
            self.constructor1(NorFileName, corpus)
        else:
            self.constructor2(NorFileName)

15. Extend test classes from unittest and use separate unit test methods.

class NGramTest(unittest.TestCase):

    def test_GetCountSimple(self):

16. For undefined types use object as type in the type declarations.

cdef class WordNet:

    cdef object __syn_set_list
    cdef object __literal_list

17. For boolean types use bint as type in the type declarations.

	cdef bint is_done

18. Enumerated types should be used when necessary as enum classes, and should be declared in py files.

class AttributeType(Enum):
    """
    Continuous Attribute
    """
    CONTINUOUS = auto()
    """

19. Resource files should be taken from pkg_recources package.

	fileName = pkg_resources.resource_filename(__name__, 'data/turkish_wordnet.xml')