Add U+FE00 to the model and Phake training data

[sffc]

The following string is Unicode, but it detects as Zawgyi. It contains a lot of U+FE00. If we add that code point to the model, it might make this text correctly detect as Unicode, even without a lot of training data.

ꩬ︀ံꩭုဝ︀်ꩬ︀ိပ︀်တ︀ိꩫ︀်ၸ︀ႝꩫ︀ိုဝ︀်ꩫ︀ိꩫ︀်မ︀ေ︀ပꩫ︀ႃ ။ ၸ︀ၞ်ꩭူၺꩫ︀်တ︀ႝꩡ︀ွ်မ︀ႃꩭေ︀ႃကꩭၞ်ꩫ︀ႝမ︀ွက︀်လ︀ွ်ꩡ︀ွ်
တ︀ႃ ။ ꩬ︀ိပ︀်တ︀ိꩫ︀်ꩬ︀ံꩭုဝ︀်ၸ︀ႝꩫ︀ိꩫ︀ၵ︀ံမ︀ေ︀ပꩫ︀ႃ ။ ၸ︀ၞ်ꩭူမ︀ႃꩭေ︀ႃၺꩫ︀်ၸ︀ြႃကꩭၞ်ꩫ︀ႝမ︀ွက︀်လ︀ွ် ꩡ︀ွ်တ︀ႃ ။ ꩬ︀ုတ︀်ယ︀ွ် ။
ဝ︀ွႃꩭင︀်ထ︀ႝꩫ︀ႃ ။

CC @sven-oly

[markusicu]

U+FE00 is a variation selector.

• Is it normal for Myanmar text to use variation selectors?
• Can you just ignore all variation selectors?
• Unicode property: Variation_Selector
• https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G19053

[sffc]

We could make the model skip over variation selectors.

Could we just ignore the 14 here? https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=\p{subhead=Variation%20selectors}

Or should we ignore all 260 here? https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5B%3AVariation_Selector%3A%5D&g=&i=

[sffc]

On the other hand, if variation selectors are a hallmark of Unicode text, then it would be harmful to the model to ignore them.

[markusicu]

Could we just ignore the 14 here? https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=\p{subhead=Variation%20selectors}

This is a bogus query: You are asking for characters under a certain heading in the code charts. (Parsed from NamesList.txt)
Headings are useful for readers, but not for property queries.

Or should we ignore all 260 here? https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5B%3AVariation_Selector%3A%5D&g=&i=

Yes, that's the real property.

if variation selectors are a hallmark of Unicode text, then it would be harmful to the model to ignore them.

They request a different glyph without changing character identity, that is, they don't change the meaning of text. Seems totally appropriate to ignore them here.

On the other hand, the Myanmar variation sequences might indicate that the text is not in the Myanmar language. See the proposal that added these: https://www.unicode.org/L2/L2015/15320-khamti-vs.pdf

I am not sure what this means for detecting whether the string is in the Zawgyi font-hack encoding.

See the list of variation sequences in plain text form here (search for "Myanmar"): https://www.unicode.org/Public/UCD/latest/ucd/StandardizedVariants.txt

[sffc]

See the list of variation sequences in plain text form here (search for "Myanmar"): https://www.unicode.org/Public/UCD/latest/ucd/StandardizedVariants.txt

A-ha, so the set of variation selectors for a given code point is fixed. I see that only \uFE00 is used in Myanmar code point sequences. So it strikes me as important that we distinguish say 1000 FE00 from just 1000. It seems that FE00 is unlikely to appear in Zawgyi text, but it might appear in Myanmar Unicode text.

[markusicu]

It seems that FE00 is unlikely to appear in Zawgyi text, but it might appear in Myanmar Unicode text.

good point; yes zawgyi text is unlikely to use proper unicode features