Change table card to use Predbat raw plan data

[gcoan]

For a long time there's been wack-a-mole between Predbat and the table card, every time Trefor changes the format of the Predbat HTML plan, the table card has needed tweaks to accomodate.

In a recent Predbat release Trefor has added the 'raw' attribute to predbat.plan_html which gives the raw data in structured format.

It looks like this gives all the data the table card needs, but would need more investigation to swap over to using this attribute instead of decoding the HTML.
Appreciate that this is likely to be a big change but once done could decouple the ongoing maintenance significantly

{{ state_attr('predbat.plan_html', 'raw') }}

gives:

{'rows': [{'time': '2025-10-19T10:25:00+0100', 'import_rate': 28.65, 'export_rate': 15.0, 'state': 'Demand', 'state_target': '', 'state_override': '', 'state_html': '↗', 'pv_forecast': 0.13, 'pv_forecast10': 0.1, 'pv_forecast_total': 0.13, 'load_forecast': 0.04, 'load_forecast10': 0.04, 'load_forecast_total': 0.04, 'clipped': 0, 'soc_percent': 52, 'soc_change': 0.08, 'cost_change': 0.0, 'total_cost': 0.25}, {'time': '2025-10-19T10:30:00+0100', 'import_rate': 28.65, 'export_rate': 15.0, 'state': 'Demand', 'state_target': '', 'state_override': '', 'state_html': '↗', 'pv_forecast': 0.89, 'pv_forecast10': 0.67, 'pv_forecast_total': 1.02, 'load_forecast': 0.29, 'load_forecast10': 0.32, 'load_forecast_total': 0.32999999999999996, 'clipped': 0, 'soc_percent': 53, 'soc_change': 0.55, 'cost_change': 0.0, 'total_cost': 0.25}, {'time': '2025-10-19T11:00:00+0100', 'import_rate': 28.65, 'export_rate': 15.0, 'state': 'Demand', 'state_target': '', 'state_override': '', 'state_html': '↗', 'pv_forecast': 0.91, 'pv_forecast10': 0.68, 'pv_forecast_total': 1.9300000000000002, 'load_forecast': 0.38, 'load_forecast10': 0.42, 'load_forecast_total': 0.71, 'clipped': 0, 'soc_percent': 57, 'soc_change': 0.49, 'cost_change': 0.0, 'total_cost': 0.25}, {'time': '2025-10-19T11:30:00+0100', 'import_rate': 28.65, 'export_rate': 15.0, 'state': 'Demand', 'state_target': '', 'state_override': '', 'state_html': '↗', 'pv_forecast': 0.86, 'pv_forecast10': 0.62, 'pv_forecast_total': 2.79, 'load_forecast': 0.36, 'load_forecast10': 0.4, 'load_forecast_total': 1.0699999999999998, 'clipped': 0, 'soc_percent': 60, 'soc_change': 0.46, 'cost_change': 0.0, 'total_cost': 0.25}, {'time': '2025-10-19T12:00:00+0100', 'import_rate': 28.65, 'export_rate': 15.0, 'state': 'Demand', 'state_target': '', 'state_override': '', 'state_html': '↗', 'pv_forecast': 0.8, 'pv_forecast10': 0.56, 'pv_forecast_total': 3.59, 'load_forecast': 0.5, 'load_forecast10': 0.55, 'load_forecast_total': 1.5699999999999998, 'clipped': 0, 'soc_percent': 64, 'soc_change': 0.27, 'cost_change': 0.0, 'total_cost': 0.25},
...

[pacemaker82]

It's great to see this supported - the limitation/issue is there is no colour references to Trefor's logic. So old_skool settings are out the window unless I can mimic exactly in line column by column what his logic is to allow those settings to continue. I get that for free in the HTML.

FWIW I think relatively easy to support this field otherwise.

[pacemaker82]

Be useful if the fields were documented too ... some are in the plan, some arent, and some will appear if you configure them. I dont have co2 configured, so dont see those in the raw and dont know what the JSON might look like either. I will see if I can find in his code

[pacemaker82]

Looks like this is the file that defines it: https://github.com/springfall2008/batpred/blob/e371784bca74f8c7eaf543d6fd39e7c7556e719e/apps/predbat/output.py#L1476

[gcoan]

Be useful if the fields were documented too ... some are in the plan, some arent, and some will appear if you configure them. I dont have co2 configured, so dont see those in the raw and dont know what the JSON might look like either. I will see if I can find in his code

nothing is documented !

The only documentation of the output data, the detailed writeup of the colours and symbols of the predbat plan https://springfall2008.github.io/batpred/predbat-plan-card/#understanding-the-predbat-plan are that I went through the code and reverse engineered it into documentation.

If there is a real need for it, I could do so for the raw. I have updated the docs to at least mention this new attribute now

[pacemaker82]

I should be able to cross reference Trefor's code and your doc to figure out the HTML/RGB colours. I am mostly concerned about the ones I dont see, I can figure out the ones I do.

[pacemaker82]

I have it mostly working, with the colours - I copied Trefor's same logic

Biggest issue is the "doubled up" states he does:

Take this example, in the json raw it looks pretty much the same other than the state_html, which will then lead me to doing something gross with code. His state here should be different in some way IMHO.

    {
      "time": "2025-10-19T23:00:00+0100",
      "import_rate": 29.63,
      "export_rate": 15,
      "state": "Exp",
      "state_target": "11",
      "state_override": "",
      "state_html": "Exp↘",
      "pv_forecast": 0,
      "pv_forecast10": 0,
      "pv_forecast_total": 0,
      "load_forecast": 0.22,
      "load_forecast10": 0.24,
      "load_forecast_total": 3.3100000000000005,
      "clipped": 0,
      "car_charging": 0,
      "soc_percent": 27,
      "soc_change": -1.55,
      "cost_change": -0.18,
      "total_cost": 2.33
    },
    {
      "time": "2025-10-19T23:30:00+0100",
      "import_rate": 7,
      "export_rate": 15,
      "state": "Exp",
      "state_target": "6",
      "state_override": "",
      "state_html": "Chrg&nearr;</td><td bgcolor=#FFFF00 style=\"padding: 4px;\">Exp&searr;",
      "pv_forecast": 0,
      "pv_forecast10": 0,
      "pv_forecast_total": 0,
      "load_forecast": 0.2,
      "load_forecast10": 0.22,
      "load_forecast_total": 3.5100000000000007,
      "clipped": 0,
      "car_charging": 0,
      "soc_percent": 11,
      "soc_change": -0.55,
      "cost_change": -0.09,
      "total_cost": 2.25
    },

[gcoan]

That's probably an artefact of how the split cells are coded in the html.

What would you want it to be and we can look at what can be changed.

[pacemaker82]

I think an additional state attribute would be good for the first of the split cells.

So, we know state: Exp is the second of the split cells, be good if state_split: attribute could be added and has the state for the first split if applicable. My logic would be if its empty then its not a split.

E.g.

      "state": "Exp",
      "state_split": "Chrg",
      "state_target": "6",
      "state_override": "",

[pacemaker82]

HTML Debug Mode is also not supported in terms of import/export rates

    {
      "time": "2025-10-20T10:25:00+0100",
      "import_rate": 29.63,
      "export_rate": 15,
      "state": "FrzExp",
      "state_target": "",
      "state_override": "",
      "state_html": "FrzExp→",
      "pv_forecast": 0.05,
      "pv_forecast10": 0.02,
      "pv_forecast_total": 0.05,
      "load_forecast": 0.07,
      "load_forecast10": 0.08,
      "load_forecast_total": 0.07,
      "clipped": 0,
      "car_charging": 0,
      "soc_percent": 93,
      "soc_change": -0.02,
      "cost_change": 0,
      "total_cost": 1.52
    },

[gcoan]

I'll have a look, see what I can do with the predbat code....

[pacemaker82]

Just generally, the more I look at it, the more the "raw" values isn't what I need without a lot more info. I either need the HTML values of each column, or additional info to help with how Trefor makes formatting decisions.

I'm replicating so much predbat logic at the moment to support colours, arrows, just noticed bold/italic formatting on the import/export too. If Trefor touches any of that I will be out of sync immediately.

Will think about it a bit more

[pacemaker82]

Another example where I can't do anything right now, cant even copy his logic as I dont have the _cost_threshold data.

From https://github.com/springfall2008/batpred/blob/dc18d2d9ffaae8b7b2fa4addf7f40bdec1da4890/apps/predbat/output.py#L1190

            if rate_value_import <= 0:  # colour the import rate, blue for negative, then green, yellow and red
                rate_color_import = "#74C1FF"
            elif rate_value_import <= import_cost_threshold:
                rate_color_import = "#3AEE85"
            elif rate_value_import > (import_cost_threshold * 1.5):
                rate_color_import = "#F18261"

            if rate_value_export >= (1.5 * export_cost_threshold):
                rate_color_export = "#F18261"
            elif rate_value_export >= export_cost_threshold:
                rate_color_export = "#FFFFAA"
            elif rate_value_export > (0.5 * export_cost_threshold):
                rate_color_export = "#48deff"
            else:
                rate_color_export = "#dcdcdc"

Perhaps a solution could be a dictionary of formatting per column from Trefor, where I don't need to map his logic.

        state: "Demand",
        state_format: {backgroundColor: "#FFFFFF", arrow: "→", textBold: false, textItalic: false},
        import_rate: "29.53",
        import_rate_format: {backgroundColor: "#AA0000", arrow: null, textBold: true, textItalic: false},

Looking at Trefor's code, this information is all in there, but it would be a huge rewrite I think to extract it all because he merges formatting with raw values in variables all the time, so that would take unpicking.

[gcoan]

My inclination would be to not replicate the colour attributes into the table card as that just hard codes you to a different bit of Predbat logic.

If we're going to use raw data then the raw data should be presentation agnostic so you then apply whatever style is required.

So for example the predbat code would become (rough code outline, not proper python):

            if rate_value_import <= 0:  # colour the import rate, blue for negative, then green, yellow and red
                rate_color_import = "#74C1FF"
                rate_color_label = zeronegative
            elif rate_value_import <= import_cost_threshold:
                rate_color_import = "#3AEE85"
                rate_color_label = belowthreshold
            elif rate_value_import > (import_cost_threshold * 1.5):
                rate_color_import = "#F18261"
                rate_color_label = 15abovethreshold

And we write the labels into the raw data structure.

Similarly state_direction is a text label(up, down, flat) and we do away with state_html entirely.

If everything is coded as raw json labels then predbat can change the output format as often as required with no breaking changes to the table card.

[pacemaker82]

I am honestly not sure what the difference is between what you're suggesting and what I am, other than a JSON formatting difference. Maybe I am not understanding clearly though or maybe it is just a JSON format preference.

In short..
I cant replicate OldSkool without replicating Trefor's formatting logic, OR being given Trefor's format explicitly per column, per row. I can't replicate his logic without having all the relevant data to make the logic work.

I agree, making my own formatting logic just makes another dependency on Predbat if that changes. Therefore the only alternative is for the format to be provided to me in the raw, if OldSkool setting wants to be replicated AS-IS.

Alternatively I just make my own format, its not OldSkool (deprecated), its my version and its separate and not a "mimic" of the original plan formatting, but shows the row raw data in the same way rather than with icons or whatever...

Otherwise I can predict the bug reports: "I am using oldSkool setting but X behaviour isn't the same as the original plan"

[gcoan]

I was suggesting that you make your formatting entirely separate from what Trefor does, so you can both iterate independently.

But if people want Trefor's columns just with the option of hiding/showing some and changing the order (aka old-skool passthrough) then you'll have to have both formatting independent raw data so you can style as you want, plus a HTML version of each of Trefor's columns so you can copy them in as-is. Which bloats the raw data structure significantly or you start grubbing around parsing the HTML which is where you are right now.