2020-07-17 Binning(?)

[jguild]

Apologies for what might be a basic misunderstanding on my part, but I didn't quite follow the "Binning" approach that was discussed during the 2020-07-17 call. I got a little lost on what the point was.
Were the "better" results just exemplifying that the means of multiple random samples of a population will always follow a normal distribution, even when the underlying population may not be?

[mcgeeyoung]

Jeff, this isn't a bad question at all! And this can be kind of hard to grok at first blush. The fundamental challenge that's being solved for is trying to make the comparison group look like the treated group. The issue arises because the treated group doesn't look like the general population, so we need to select a subset of the general population in order to find a set of non-participants whose consumption patterns are closer in fact to the treated customers. One way to do this is by stratified sampling, which is in essence "binning". What you do is take your whole treated population and divide it into, say the top 50% of annual consumption and the bottom 50% of annual consumption. This creates two bins. The cutoff point between the two bins will likely not be the same cutoff point as what would happen if you were to replicate this in the general population. Let's say, as an example, that this 50% cutoff point in the comparison group would divide the general population into one group that comprised 60% of the general population and another group that comprised 40% of the general population. So the distribution of annual consumption would be different in the general population than the treated population. As a solution to this, we sample from the general population so that we get 50% on each side. We would choose fewer, proportionately, of the general population that fell into the 60% group and more, proportionately, that fell into the 40% group.

We can make this more complicated by creating more complicated bins (more than one characteristic) and by creating more bins (say 10, so each one represents 10% of the treated population). As we do this, we get a closer and closer match of the comparison group to the particular usage patterns of the treated group.

[jguild]

Got it... Stratified sampling based on identifiable characteristics within the general population.
Curious if you guys have given any more thought about the suggestion during last Friday's call to use the load shape characteristics as the selection strata.