Regarding method 2.4: Reduce CVRMSE limit down to ASHRAE guideline?

[steevschmidt]

From the draft findings document, page 4:

2.4. Remove customers with daily baseline CVRMSE values in excess of 1.0.

I understand that CVRMSE is an indication of the quality of the model fit, and lower is better. During early CalTRACK discussions there was consensus to allow buildings with very high CVRMSE values (well above ASHRAE guidance of 0.25, up to 1.0) in order to prevent the exclusion of willing participants in "bad buildings" from available P4P programs.

But for the purpose of these comparison groups, it would seem accuracy of CalTRACK models (i.e. low CVRMSE values) within the comparison group would be more important than permitting a larger comparison group that includes buildings with poor model fits. The goal of this effort is to assess the impact of an exogenous factor within the comparison group, so including just those buildings with reasonable model fits -- assuming the population is big enough -- would help improve the accuracy of that analysis.

So I would propose this method be changed to "Remove customers with daily baseline CVRMSE values in excess of 0.25."

Has such a tradeoff between this CVRMSE filter and comparison group size already been discussed and analyzed?

[mcgeeyoung]

This would be possible. We did not explicitly test CVRMSE thresholds. It would be interesting to see how it changed the variability that is seen. Our initial thinking was that you would want to apply the same restrictions on treated and untreated customers. So if you restricted CVRMSE to 0.25 for treated, you'd do the same thing for untreated. One thing to note with COVID is that it has led to more erratic energy use, so if you were to apply tighter restrictions right now, you'd likely end up with a smaller pool of qualified buildings. I think this is best left to program M&V plans to decide.