Align format requirements for csv headers to be in line with RFC 4180

[danielfeismann]

We defined to have our csv input data in line with RFC 4180. However, over the time some entries, at least in the tests, drifted appart. They should be in snake_case format.

For example in CsvDataSourceTest.

active_power_gradient is correct.
cosphi_rated should be cos_phi_rated.
cosPhiFixed should be cos_phi_fixed.

see:

PowerSystemDataModel/src/test/groovy/edu/ie3/datamodel/io/source/csv/CsvDataSourceTest.groovy

Lines 151 to 164 in 23f970d

	def "A CsvDataSource should build a valid fields to attributes map with valid data as expected"() {
	given:
	def validHeadline = [
	"uuid",
	"active_power_gradient",
	"capex",
	"cosphi_rated",
	"eta_conv",
	"id",
	"opex",
	"s_rated",
	"olmcharacteristic",
	"cosPhiFixed"
	] as String[]

todo

• format whatever case to snake_case for csv file inputs within the tests
• think about how this can be forced and if it should be forced

[sebastian-peter]

Let's shed some light on the historical background (which does not mean I would oppose changing it)...

Before our time 😉, there were efforts to support multiple naming conventions. The idea was to not enforce a strict naming convention, but allow multiple, with the most support behind snake case and camel case.

You can find some remnants in PSU:

• edu.ie3.util.naming.Naming
• edu.ie3.util.naming.NamingConvention

Also issues like this in PSDM were created in that context:

• Rethink column name determination in SQL source #514
• Couchbase column names in snake case #516
• Fix PsdmTimeBasedWeatherValueFactory #267 (especially discussion by Chris)

Long story short, since both snake case and camel case needed to be supported. In order to work with a suitable "format in the middle", iron case was invented, which just means lower case of camel case or snake case with underscores removed. That's why all the tests you stumbled upon are written in that way.

I had to do some digging to find the places where all of this is done, basically it is: In edu.ie3.datamodel.io.source.csv.CsvDataSource#buildFieldsToAttributes we have

1. a TreeMap with CASE_INSENSITIVE_ORDER, thus camel case gets flattened
2. StringUtils.snakeCaseToCamelCase being applied to the headline items, thus underscores get removed

I'm actually not sure if edu.ie3.datamodel.io.factory.Factory#validate actually validates the same thing...

In any case, of course this can be changed. I had thoughts about defining a fixed format that users have to use, but never had the time to get it done. Once there is a fixed format, all the bells and whistles around naming convention and case conversion could be removed.

[danielfeismann]

Thanks for adding the background to this topic. I agree that also the support of other cases is great. Since edu.ie3.datamodel.io.source.csv.CsvDataSource#parseCsvRow which is called by edu.ie3.datamodel.io.source.csv.CsvDataSource#buildFieldsToAttributes, refers to RFC 4180 which defines header to be in snake case, I would like to be in line with that, at least for the CSV files. This I would see as part of the first to do.
Regarding the definition of some fixed format and I clearly see also some advantages, however I'm also not sure if it is worth taking that big step. So maybe let's skip that second one for now.

[sebastian-peter]

Ok. Just be aware that changing requirements for CSV headers means that most of our existing simona grids need to be adapted.

[danielfeismann]

Maybe I was a bit unprecise at that part. I first only want to fix this in the tests etc. So I don't see that this will require the adaption of existing simona grids. Or did I miss something?

[sebastian-peter]

Ok, I misunderstood. Yes, adapting the tests does not have any ramifications.