Support or better defaults for encoding of special characters (e.g., nordic letters "ÆØÅ")

[soerendamsbo]

Description

Hi! I am hoping my issue is one that is easily remedied – e.g., by defaulting to utf characters or changing an argument in the code that writes responses.

When the respondent selects something on a drop-down list (type 'select') with a special character like the Danish letter "Ø", the stored value removes the character if it comes first/last in the word and replaces it with "_" if it is placed in the middle. "Ærø" becomes "r", and "Allerød" becomes "aller_d".

This is a manageable bug, but somewhat unpredictable and easy to forget later.

[jhelvy]

Good question, and I just posted an issue about this in #262 . I think we'll have to modify a few things to allow this to be avoided. In the meantime, if you provide a label and a value for each option, then the values will be stored in the data instead and then you could join back on the labels with special characters afterwards once you've collected your survey data. For example, if you added values like this:

sd_question(
  type   = 'mc',
  id     = 'artist',
  label  = "Which artist do you prefer most from this list?",
  option = c(
    "Taylor Swift" = "taylor_swift",
    "Lady Gaga"    = "ladygaga",
    "Ed Sheeran"   = "ed_sheeran"
  )
)

Then you'd have things like taylor_swift and ed_sheeran in your data instead of the labels. All of the labels and values used though will be available in the _survey/question.yml file, which you can read in later while data processing to join on the correct labels. That at least will make the data cleaning process a bit easier as a temporary work around.

[soerendamsbo]

Your solution improves control significantly. Great to hear that you are aware of the encoding bug. Thanks, John!

[pingfan-hu]

We have addressed this in our v1.1.0 version. Now we don't do auto snake case conversion for short-handed option definition.
For example, for this question:

sd_question(
  ...
  option = c("ÆØÅ", "Option 2")
)

The options will have their original values stored in db.
We will release this update soon!

[soerendamsbo]

Thanks a lot, that's a nice improvement. I've tested it and it works. Moreover, it seems that option = c("(Æ)" = "æ") now correctly stores the value as "æ" (not "a") – at least when storing to a local csv. So, perhaps you solved the encoding issue too? Thanks anyways and congratulations on a great and swiftly improving survey platform.

[jhelvy]

Yeah these issues were linked, so this should all be working now.