Resolve relative paths used in flows in relation to the flow definition file

[reubenmiller]

Is your feature improvement request related to a problem? Please describe.

It is common to create a self-contained flow which includes both the flow definition file (e.g. flow.toml) and any related JavaScript code.

When packaging a flow, it is helpful to package the flow into a folder so that all of the files are self-contained and so that it is easier to identify which files belong to which flow.

Currently, if a flow's files are put under a nested directory, the flow's script paths need to be updated to reflect the nested hierarchy which is difficult to manage, and makes the flow less portable/flexible.

Describe the solution you'd like

A flow definition file (e.g. flow.toml) should be able to use relative paths when referencing JavaScript files, but the path should be relative to the flow definition file.

Having the script's path being relative to the flow definition file allows for more flexible packaging options where the "script" property does not need to be changed if the flow's files are moved into nested folders.

For example, below is an example flow definition file using a relative path to the "main.js" JavaScript file.

File: flow.toml

# example flow
input.mqtt.topics = ["foo"]

[[steps]]
script = "main.js"

Then, a flow could be installed at any nested level (as long as the JavaScript is kept alongside the flow.toml file). Below shows the example flow being stored in two different ways, and the relative path is still valid in each case:

tree /etc/tedge/mappers
└── flows
    └── flows
        ├── example
        │   ├── flow.toml
        │   └── main.js
        └── foo
            └── example
                ├── flow.toml
                └── main.js

Note: A flow definitions should still be able to support absolute paths

Describe alternatives you've considered

Additional context

[jarhodes314]

This is possibly an overly fancy solution compared to reading the path, but figment has some bit of serde trickery that allows the deserialisation of (possibly) relative paths (so the conversion between relative and absolute happens as part of the deserialisation): https://docs.rs/figment/latest/figment/value/magic/struct.RelativePathBuf.html. This is a suggestion to implement our own version of that as we don't currently depend on figment and it's a fairly significant crate for just doing this one thing.

[reubenmiller]

@jarhodes314 The RelativePathBuf has promise but it doesn't seem to work for the nested field that I need to.

I added a new dummy field to the FlowConfig just to verify if I was using the Figment crate correctly, and the following worked fine.

pub struct FlowConfig {
    name: RelativePathBuf,
    // ...rest
}

But when using RelativePathBuf in a nested struct or enum, it doesn't work, and the parsing of the toml fails...and here is the ticket saying that is not supported when used within an Enum, SergioBenitez/Figment#107

So I think I'll have to go with the boring solution.

[jarhodes314]

But when using RelativePathBuf in a nested struct or enum, it doesn't work, and the parsing of the toml fails...and here is the ticket saying that is not supported when used within an Enum, SergioBenitez/Figment#107

This seems to be an inherent issue with serde, where things like certain enum representations and struct flattening require serde to buffer the input before deserialising it. When serde does this buffering, it bypasses figment's custom deserialisation logic and as a result, we get errors as you discovered.

I have been experimenting with an alternative approach relying on thread-local storage. So when we start deserialising, we store the path in the thread-local storage, then our RelativePathBuf type reads this storage for the current file path. I've got a proof of concept locally that I can experiment with incorporating into tedge-flows and it seems to work (with sufficient guards to protect against the possible pitfalls of thread-locals).

[reubenmiller]

@jarhodes314 Though is it really with the effort? As the current code does work fine and it is very simple to reason with.

[jarhodes314]

@jarhodes314 Though is it really with the effort? As the current code does work fine and it is very simple to reason with.

Maybe not in the context of flows. For tedge-config though, it might well be now we have many mapper configs in different places

[reubenmiller]

Implemented already