Pronunciation perception is very subjective. Thus annotators with different language backgrounds will naturally disagree on which IPA symbols to label a given speech sound with. In our and Menzel et al.'s experience (from email discussions), this is a large portion of annotations and is not something that can necessarily be directly resolved through discussion between the annotators, visual inspection in Praat, nor even has a "right" answer. We need to evaluate annotator consistency (like in Menzel et al.'s paper) for all our datasets.
Pronunciation perception is very subjective. Thus annotators with different language backgrounds will naturally disagree on which IPA symbols to label a given speech sound with. In our and Menzel et al.'s experience (from email discussions), this is a large portion of annotations and is not something that can necessarily be directly resolved through discussion between the annotators, visual inspection in Praat, nor even has a "right" answer. We need to evaluate annotator consistency (like in Menzel et al.'s paper) for all our datasets.