Dara Loader bug: t + ʃ -> tʃ and d + ʒ -> dʒ

[bi1101]

With the current data loader pipeline, affricates are merged together, affecting model performance

Example: "that she" ['ð', 'æ', 't', 'ʃ', 'i'] becomes ['ð', 'æ', 'tʃ', 'i']

[SanderGi]

Thank you for double checking our data loading code! Do you have an example of where this bug occurs (dataset + sample id)? Since we concatenate all the phonemes into a string and ['ð', 'æ', 't', 'ʃ', 'i'] and ['ð', 'æ', 'tʃ', 'i'] both become ðætʃi, so I'm not quite following how model performance is affected.

[bi1101]

Hi, it is the concatenation that is causing the issue. in the example ['ð', 'æ', 't', 'ʃ', 'i'] the t + ʃ are 2 phoneme, that makes up the word "that she". But during tokenization with the current model vocab, it will be ['ð', 'æ', 'tʃ', 'i'] (tha chi) instead of "that she".

Phonetically, 2 sounds t + ʃ are NOT the same as a single sound tʃ (in children). Think of how pronounce "hot shot" vs "ho chot"

[SanderGi]

Right, I see now that you're talking about the model tokenization, not directly the data loaders. I'm aware that t + ʃ and tʃ are not the same. Since both the individual affricates and the combined voiced/voiceless postalveolar affricates are in the vocab, the model should be able to predict it however your point is that because the dataloaders concatenate everything into one big string, that training signal is lost and the model is not able to learn that distinction? Let me know if I've understood your question correctly

[bi1101]

Yes, that is what I mean

[SanderGi]

Gotcha, thank you for clarifying. I suspect the impact is small since the transformer portion of the model has likely learned to just map both the tʃ sound and t + ʃ to tʃ, however we should definitely fix this and measure the impact. I believe all the data loaders except DoReCo and Speech Accent Archive return the correctly grouped phonemes via timestamped phonemes, so for those, it should be a matter of updating the training code in wav2vec2_train.py to use those phoneme groupings for the labels. Speech Accent Archive has bigger issues that need to be resolved so it is not currently used for training/eval. DoReCo is used but doesn't seem to annotate for the distinction as far as I can see, so it might require some manual effort to fix.