a few thoughts on what else might be possible

[pljakobs]

so, the new conditional defaults are great, that can remove, in my case, a bit of application code and push decisions to the compile phase.

In general, ConfigDB has come a very long way from my initial idea of "how can I get rid of those in-memory configuration structs and their multiple conversion into json for the API and storage" to something that is "just there" and makes my code much cleaner.

I have found myself wondering a few times what more it could do, and please just take this as just that: a few thougths.

• the data structure lends itself to a REST API - I think it would be really nice to use the existing json schema to autogenerate a REST API? (perhaps with some extensions to hint for specific API semantics such as what operations are allowed) I iamgine a system where I basically might hand a reference to a webserver to the class and all the json leaf objects are registered as endpoints. That would allow a developer to define the data structure just once and derive the API as well as the storage structure from that
• the same is true for an mqtt API - with a few quirks such that most want separate read and write topic subtrees - again something that might be done through hinting properties
• an ephemeral "storage" (essentially cache only) could be nice for data that doesn't need to persist but is available through the API
• add to this a system of registering callbacks in the C++ code for interactions with said APIs

This would essentially turn any application that uses this into a reactive / event driven application not dissimilar to how we write GUI applications.

Taken to the extreme: the same system could be used to describe a UI - after all, ui elements usually also have state and something that creates an input or wants to be displayed. Sure, this is way beyond the scope of ConfigDB but - it seems like a possible extension of the concept.

Thinking about this I was wondering: would I really prefer describing all this in a json schema instead of in code, but the real question is: how much effort is it to keep code, UI and API in sync? Having a single place that provides an abstract data structure and auto-generate the access semantics does sount like something that would create a lot of functionality with very little code.

[mikee47]

With REST are you thinking of perhaps Swagger / OpenAPI https://swagger.io/docs/specification/v3_0/about/ ?

[pljakobs]

Something like this, yes. But with OpenAPI, if I understand it correctly, I would still have to write the glue to the persistance module (which would then be ConfigDB)
Certainly something for me to look in to a bit deeper.

[pljakobs]

After thinking about it a bit longer, Swagger generated C++ would be a possible bolt-on, but my understanding is that it can be pretty heavy, depending on the library used.

I was thinking about something that i might set as a path handler, say for /api/v2/ (or as an mqtt subscribe for /cmd/) that would use the fact that it already has an understanding of the hierarchy and can thus read/update the object described by the path.

In my mind, it would not be too dissimilar from the json import/export functions.

If that then could also fire events / callbacks or even fire internal actions such as an mqtt publish if that is defined in the schema (I assume that would require annotations such as what methods are allowed) we'd have a very powerful state management engine that would provide a single truth via multiple interfaces.

In my mind this could make some applications (think simple sensors) very light on user code.

[pljakobs]

and a few more thoughts:

I am starting to think this should be, architecturally, not be part of ConfigDB but a (set of) closely coupled Components:

The core Idea is to have a single catchall function that can capture a http request or mqtt message with the path / topic tree conforming to the structure defined in ConfigDB.
Let's take this:

{"color":
  {
    "raw":
      {
        "r": 0,
        "g":0,
        "b":0,
        "ww":378,
        "cw":297
      },
    "hsv":
      {
        "h":0,
        "s":0,
        "v":65.98240662,
        "ct":4150
       }
    }
}

setting the color could be done by sending a POST to http://<ip>/api/color/hsv/h with content 180 or to http://<ip>/api/color/hsv/ with content {"h":180,"s":50,v:"30"}
I can probably do that today by assembling the json snippet representing the url path and content to a json doc and hand that the the ConfigDB streaming interface.
The equivalent could be constructed for an mqtt message but that's a bit less clear as mqtt does not usually carry the requested method, that would have to be inferred from the topic path (such as is it below cmnd or stat)

The easiest way for the application would probably be to have a global get/set function at the ROOT object that takes both value and path. That would necessarily be a non-typed (or just String and int64_t) parameter, though.
That way, the external code would not need to build the json object but just assemble the path and call a function.
But in a way, that's just a bit more comfortable than using the json import.

The other way round would be enabled if ConfigDB could call a function when a value has been updated. I'm thinking that would be some sort of enum (like ROOT::Color::rgb::r in the example above to allow a single callback to act as a dispatcher using a switch : case tree to dispatch the corresponding action.

[mikee47]

The JSON streaming interface uses JSONPath-like syntax (e.g. color.hsv.h but I wonder if we should standardise on JSONPointer instead. Do you rely on JSONPath in your applications?

[mikee47]

So I've dug around a bit and here's the situation so far.

As previously discussed, JSONPath is a complex query language and I'm not proposing we attempt to implement it here. Standardising on JSONPointer would be more practical. Translating to dotted form should be pretty straightforward if required.

The ABNF syntax of a JSON Pointer is:

json-pointer    = *( "/" reference-token )
reference-token = *( unescaped / escaped )
unescaped       = %x00-2E / %x30-7D / %x7F-10FFFF
    ; %x2F ('/') and %x7E ('~') are excluded from 'unescaped'
escaped         = "~" ( "0" / "1" )
    ; representing '~' and '/', respectively

Even with escaping (which we could set aside for now), it's quite doable.

Using the BasicConfig example, we can have standard JSONPointer paths:

• api/security returns {"api_secured": true}
• api/network/mqtt/server returns "mqtt.local"
• api/general/channels/0/name returns "Channel #619941660"

We can make a customisation to support array lookups like this:

• api/general/channels[name=Channel #619941660]/details/purpose returns "general"

Note that this syntax currently only supported for streaming updates.
See object array selection in https://sming.readthedocs.io/en/latest/_inc/Sming/Libraries/ConfigDB/index.html#json-update-mechanism.

I quite like the java approach using a JSONPointer class. For ConfigDB we could have:

ConfigDB::Pointer ptr("/color/hsv/h");

A Pointer can resolve to the Database root, a Store root, an Object or a simple property.
The object or property may also reside inside an array (or even nested arrays).
Thus resolving a Pointer requires some context.
This problem is addressed inside the Json streaming classes, but it could be pulled out into something more generic:

struct PointerContext
{
public:
    // Methods here for introspection
    // TODO

private:
    Database* db;
    StoreRef store;
	Object objects[JSON::StreamingParser::maxNesting];
    uint8_t nesting{};
};

PointerContext Pointer::resolve(Database &db);
PointerContext Pointer::resolve(Object& obj);

As far as handling a GET request, if the pointer resolves to a simple property then that can be returned as a String, but the others require a streaming instance.
The streaming API could be modified to accept a PointerContext, hopefully also use it internally during iteration.

[pljakobs]

(sorry for the delay)

no, I do not use jsonpath anywhere - as said, I use sparse json documents for updates (I have a streaming endpoint for two of my databases and currently config updates are just sent this way by the front end)

Let me understand the pointer logic - I would basically be able to create a pointer from a String (such as request.uri.path) and pass that to ConfigDB to get the object?

I understand the challenge of datatypes here, but at least for my use case, strings would be okay as the data would be serialized to send over network protocols anyway, having the streaming interface in parallel would be helpful to limit heap usage.

As said, the idea is to basically be able to map the internal structure (which is hopefully a logical representation of the data the application is working with) to a set of similarly structured external interfaces.

In fact, I like the idea to have external routing functions that map subtrees from the html or mqtt or any other request to the corresponding database paths. That would be helpful to build multiple access methods but keep the structure in the backend same and minimize the amount of translation / database access code.

At this point, I have two legacy APIs that use json rpc over http and mqtt, and I will probably want to keep json rpc for websocket, but since most home automation systems have moved to a state based mqtt, that seems to be the way to go.

So in the end, I guess what I'm envisioning would make ConfigDB the central marshaller for all settings and information and there would be routers for the different access protocols around it, providing the widest set of APIs.