88 issue in Chinese transliteration

[gzhang-lc]

• Source text: 88
• Expected transliteration: 88
• Actual transliteration from SS: ba ba

Per meeting discussion with LC China Section, we discovered that when translating anything that has double 8, SS would turn it into "ba ba", this issue doesn't occur to any other numbers, e.g. 44, 55, 11 would be kept in Arabic numeral. If there odd numbers of 8, SS would still group two of the 8s together but leave the odd one as 8, e.g. 888 would get "ba ba 8", 88888 would get "ba ba ba ba 8". Initially came across this issue when using SS with an author that had a birth year in 1887 which was turned into "1 ba ba 7".

[scossu]

This is mapped literally in https://github.com/lcnetdev/scriptshifter/blob/main/scriptshifter/tables/data/_chinese_base.yml#L4165 :

    "\u0038\u0038": "ba ba "

@tventimi what are your thoughts on this?

[tventimi]

Hi Stefano,

Good catch, this entry can certainly be removed. This is some texting slang from the MDBG dictionary that is not relevant for library applications.

Tom

[scossu]

OK, I will go ahead and remove this entry.

Also, I scanned the whole Chinese table for keys that contain single-byte (\u00XX) characters:

21:    "\u0063\u0068\u0069\u006E\u0065\u0073\u0065": "pinyin "
943:    "\u88DD\u0062": "zhuang bi "
944:    "\u88C5\u0062": "zhuang bi "
1860:    "\u0074\u0061": "ta "
2057:    "\u50BB\u0062": "sha bi "
2065:    "\u0033\u0071": "san q "
2066:    "\u0033\u0070": "san p "
2068:    "\u0033\u0063": "san c "
2189:    "\u0071\u5F48": "q tan "
2190:    "\u0071\u5F39": "q tan "
2201:    "\u0070\u006F": "po "
2302:    "\u725B\u0062": "niu bi "
2707:    "\u006B\u4ED4": "k zai "
2964:    "\u0069\u4EBA": "i - ren "
3379:    "\u0065\u4ED4": "e zai "
3391:    "\u0065\u4EBA": "e - ren "
4099:    "\u0062\u683C": "bi ge "
4165:    "\u0038\u0038": "ba ba "
24904:    "\u0025": "pa "
47444:    "\u0062": "bi "
48637:    "\u00B7": ", "

Can you tell if there are any other lines to remove? (lline # is the first digit of each line)

Thanks.

[tventimi]

In principle, you could keep the ones that contain at least one Chinese character (e.g. "\u006B\u4ED4": "k zai "), because these are actual Chinese terms and are not as likely to cause confusion with other roman text. In practice, I'm not sure how important any of them are. I didn't look them all up, but they seem to be names of drugs and other slang or even offensive terms.

To be conservative, you could start by excluding entries that only have characters in the range 00-FF on the left side. This will exclude the roman-to-roman entries that are most likely to produce undesirable results.

[scossu]

Fixed in #276.