workflow-controller should allow exporting the dag.

[chirag04]

eg: given a workflow with dependent steps, workflow-controller should allow statically querying the execution-plan / dag. This is much more like how you can statically analyses a DB query.

thoughts?

[sdminonne]

@chirag04 mind explain a little bit more this issue? Do we need to export the DAG (feasible with an external tool) I'm not sure I understand everything about "statically analysis a DB query"... But I'm really not a DB guy. :)

[chirag04]

This could very well be exposed as an external tool but i believe we already have this tool in our code here.

@sdminonne imagine steps like these defined in the workflow spec:

- step A
      .....
- step B
      ....
      depends on: step A
- step C
     ....
- step D
     ...
     depends on step B and step C.

if should be possible to query what the DAG will look like even before running the workflow. in this case it would be something like:

A   -----------> B    ------->  D
C ---------------------------^

I think we already compute that graph, just that we should expose that computation as an api / util. example output could be something like a json

{
    a: [ 'b' ],
    b: [ 'd' ],
    c: [ 'd' ],
    d: [],
}

where each node stores the edges it has to other nodes. This can be used to display the DAG on UI etc.

[sdminonne]

OK. Thanks. But I don't think workflow-controller should expose an API to "export" a graphviz-like representation of the workflowSpec. What if an external pod would watch the same CustomResource expose that API?
@clamoriniere1A @EricMountain-1A ideas?

[sdminonne]

Obviously the pod I'm talking would use the same code used by workflow-controller.

[clamoriniere1A]

@chirag04 also it can be done in the future workflow-controller-ui that we have in our nice to have todo list with @sdminonne

[sdminonne]

yeah but if they already have something we may expose something... Or @chirag04 may start a small project to do this.

[chirag04]

I don't think workflow-controller should expose an API to "export" a graphviz-like representation of the workflowSpec

why not? i would imagine we already have the code for that here. we could def use that for workflow-controller-ui as well.

[sdminonne]

'cause I do want all the threads and CPU horse-power for controller threadiness. Exposing an API like that it's a simple task "but" with a REST endpoint that I would keep out of the controller.

[chirag04]

sure. didn't mean to say we have to have a REST endpoint for this. could be a simple stand-alone utility/script outside workflow-controller as well. I just thought we already have that code here, it would be cool to make it a reusable for static analysis.

that said, i haven't looked at the codebase yet. i will check and see if we can make it a simple script here or outside workflow-controller.

[sdminonne]

@chirag04 to expose what I called a graphviz-like representation you just need a simple that does something like

$ kubectl get workflows -o json > workflows.json

That is going to give you a json file that you can transform with the tool you like (jq?) or python or whatever.
If you want to do something more reactive you may want to observe workflows or implement a simple informer that watch workflows and change REST endpoint accordingly to the latest changes (if any).
As soon you have a proto or similar les us know and we may eventually have a look (if you need/think/are ok).

[sdminonne]

Keep this issue open to track this discussion which is interesting for various topics.