Automation: Execution Instructions for XOR/OR Gateway, Timer Event, Starting Variables, Network Hosts, synchronous Script Functions, etc.

[iaktern]

Hotfixes - Prio 1 (on current main):

• (Prio 1) Process Deployment seems not to work without MQTT-Server (I tried direct Engine on localhost with HTTP)
• (Prio 1) Execution View: only show versioned processes for deployment. Also show hint in top of the modal: "Hint: For deployment, a process must be versioned. Only the newest version will be deployed."
• (Prio 1) Execution View seems not to be rendering in Personal Spaces. The logs say that it tries to connect to an MQTT-Server (it also tries that in Org Spaces but there the rendering finally works). But it must also work if no (MQTT-)Engine is connected. (maybe connected or easy to solve in this context: while building the MS, it also tries to connect to an MQTT-Server. but this should not be the case)
• (Prio) Process Engines View: it should be shown, if stored Engines are online/reachable with the green dot and tooltip (currently no function)
• (Prio) delete "Templates" and "Projects" element from the left navigation side
• (Prio) create a small View for the Dashboard. E.g. Number of Deployed Processes, Number of connected Process Engines, Running versus Finished versus Aborted Process Runs.
• (Prio 1, Janis) UserTasks: Possible Performers in the "Automation Property" tab should also be available in Personal Spaces (then, there is just one User to assign a task to)
• Rename "Machine" View to "Process Engines"
• Execution View "Bug": load the view with the table immediately, independent if there is a connection to any PROCEED Engine. Have a loading symbol inside the table indicating that in the background the process data is fetched from all Engines. Timeout after 15s: show empty table if not processes could be fetched.
• Deployment Bug: if you try to deploy an already deployed version, there only comes the error badge that "Something went wrong". Show "Process is already deployed in the newest version" in a yellow badge.
• XML Editor: add X to the top right of the modal to be able to close the modal (like "Cancel")
• XML Editor: if not in writing mode, there should only be one blue "OK" button in the bottom right corner which closes the modal (not "Save")
• Bug XML Editor: going to edit mode, change some thing (e.g. the text inside the documentation element) and then clicking on Save -> the Save Warning modal stays open and the process is not reloaded.
• Bug XML Editor: the user should not be able to go back to non-edit mode if he did changes in the XML -> Request modal to discard or save changes.
• Bug Copy Script and User Task: if the user copies a Script Task (with Ctrl+C and Ctrl+V), there is no new file generated and there is a prisma error on the console. => show no error on console and the script content should be copied into a new script file which should be linked by the new Script Task.

Login Modal:

• Show mandatory fields: our env variables determine how someone can login in PROCEED: either with username+password or with an email. if both is allowed, a user must fill both fields for a new user registration. 1. This is not shown in the UI, 2. the shown error is not understandable if not both infos are entered. -> correct both problems
• Make the password field a real password field where a user can also decide on the right side to make the password visible (both: for registration and for login)
• after the first login, usually users have to create a new password but both input fields are not synchronized correctly. the bottom input field checks the password with the top one, but not the other way around. -> correct this: both must always have the same content before it is possible to submit the password
• Bug in "Roles" view: the member count for @everyone does not work

Process List and Process Editor views - Table View:

• Folders: next to the "Import" button, also have an icon for creating a new folder (just an icon, not the text. but of course with tooltip.)
• Folders: add possibility to add an (optional) "ID"
• Folders: add an "Edit" possibility tho change the properties of a folder after creation (currently only remove is possible)
• Export with folders: if multiple processes and folders are selected, export should be able to export the whole folder-structure. in this way, a user can backup all processes at once
• Bug Copy Icon for copying a process: a change in the "ID" field of the copy modal is not applied after copying
• Fill Meta Panel on the right side: Next to the table on the right side, there is an expandable Meta Panel (like the Property Panel in the Editor). The fields are not filled yet -> fill it with the same data that we already have available for the table: ID, Name, etc.
• Bug - Move Modal: Moving a process into the root folder is not user-friendly. (It works by not selecting any other folder.) Add the space name as the top root element, so that this is selectable as the target location.
• Make the min-width of "ID" column a bit smaller (min width should be around 4 letters)
• Bug - Selection in Card view: by clicking on the four-square button in the top right, you get to the Card view. But instead of opening a folder or the process editor, a click only selects the elements -> correct that.

Process List view - BPMN Viewer:

• It is correct that all input fields (especially in the property panel) are deactivated, but the gray text color and background is too hard. Make it softer and more readable: the user should recognize that it is read-only, but the text should be "active" (probably black text color)
• (also for BPMN Editor) Problem: if you have many nested folders and you open a process within it, there is no button in the BPMN Viewer or Editor to get back to the folder where you just came from (back to the folder that stores the process). You can only use the browser back button, but that is not super user-friendly. Implement the following:
  • Back Button (with a left-arrow icon) in the Header bar, on the left of "Process List"
  • "Process List" in the Header bar: it should either show just the processes from the current folder of the open process OR (better, but not sure if it is easy and user-friendly) show the full hieraechical structure with folders and processes to click on
  • Clickable Breadcrump menu in Header bar: if the table is open, there is already a clickable breadcrump menu in the Header bar. Also include this if the Editor is open, on the right of "Process List" in the Header bar. (Pay attention: don't make it too long, since we also need space for the Process Name. it is automatically cropped.)
  • if you have the BPMN Viewer or Editor with a process open, then clicking on left navigation elements "Process List" or respectively "Process Editor" should not return to the root folder but instead to the folder where the open process is stored. Only if you are in the table view (i.e. there is no process open), then clicking on this navigation elements should bring you to the root level

Process Editor view - BPMN Editor:

• Version List: change sorting -> show newest versions on top
• Change title for component planned-duration-input:
  • for Property Panel: "Edit Planned Execution Duration"
  • for Intermediate and Boundary Timer Events: "Edit Wait Duration of Timer Event"
  • for Timer Start Events: "Edit Trigger Interval for Timer Start Event"
    • Icon Tooltip for Timer Start Events: "Edit Trigger Interval for Timer Start Event"
• Bug - Keyboard Shortcuts and open Property Panel: the bpmn-js keyboard shortcuts inside the BPMN Editor do currently not work, if the Property Panel is open or collapsed (Del, Copy, etc.). Maybe because of our own keyboard shortcut system. It only works if the Property Panel is closed completely, which only happens if the 'i' icon is clicked. Fix it, so that so keyboard shortcuts work if the Property Panel is open.
  • Property Panel - only two states: currently, the Property Panel has three states -> closed, open, collapsed. We only want two states: collapsed and open. To switch between both states, two options should work: a click on the 'i' icon and a click on the whole collapsed side bar (as well as the icon for collapsing if the Property Panel is open)
• Process Variable definition - harmonize the definition of process variables with the global Data Objects: add Display Name input field, make Description a bit bigger. write Display Name into the XML attribute displayName
  • for Kai G.: document displayName attribute in XML schema
• Automation Activation in Automation Tab: if Automation is activated, highlight the following automateable BPMN Elements in the "Change element" list (wrench symbol) with a blue color and put them on top of the list:
  • for Activities: User task, Script task, Sub-process (collapsed), Sub-process (expanded);
  • for Events: Empty Start, Timer Start, Intermediate throw, Timer intermediate catch, Timer boundary, Escalation boundary, Error boundary, Timer boundary (non-interrupting), Escalation boundary (non-interrupting), End, Terminate end event
  • for Gateways: XOR, AND, OR
• Automation Activation in Automation Tab: if Automation is activated AND the diagram contains BPMN Elements that are not automateable, show yellow hint text under the option "The diagram contains elements that cannot be automated. These are marked with a yellow exclamation mark."
  • if Automation is activated, show a yellow exclamation mark icon next to/on the top right corner of every BPMN Element that cannot be automated. (only Tasks, Events and Gateways. Not the attached data objects or text annotations)

UserTasks Editor:

• (Janis/Felipe) UserTask and ScriptTask Editor: "Save" button should close the Modal. It should also show the green "Saved" pop-up (currently only ScriptTask Editor)
• Bug: Cancel button and closing X show a hint about unsaved changed, although nothing was changed. -> if nothing was changed, don't show this hint.
• Hint modal about unsaved changes (Cancel button and X): Add an additional, blue Save button (saves and closes form) and rename "OK" to "Discard Changes"
• Add User Task name to title of Editor modal after "Edit User Task Form:"
• Add Default Content:
  • for User Tasks: one Header element that contains the task name and one Text element that contains the task documentation description (see later: by using the predefined process variable task-description) and one Submit button -> automatically attach this form to every UserTask even if the Form Editor was never opened.
  • 1. for Empty Start Events, if there has no process variable been defined which is required to be filled at Startup, then: do not attach/define any default form automatically until the Form Editor is not opened. But if the Form Editor is opened for the first time, attach the same default template as in the next point but with an empty variable list.
  • 2. for Empty Start Events, if there are process variables defined which are required to be filled at Startup, then: one Header element that contains the process name and a list of Input elements for every variable that needs to be filled at Startup and one Submit button => This automatically generated form should automatically be attached to the process (but only of there is at least one startup required variables). This is just the default form, it can always be edited and refined by the User
    • Problem: 1. variables were added after the Startup form was already edited by the user: add new Input fields for the new variables directly over the Submit button
    • Problem: 2. variables were deleted after the Startup form was already edited: delete the respective Input fields for the variable from the list. (-> It should not be possible for the user to delete Input fields for variables that are required to be set at startup. This should only be done automatically by the MS)
  • Remove the tile for the Submit button -> Users are not allowed to add one by themself. Also, don't allow Users to delete the existing Submit buttons within the templates -> so, there is always a Submit button inside the form
• (Janis) UserTask Editor: if the mouse is over a text field, it gets gray. super. double click on the whole gray area should also open the text edit mode (next to the left click on the edit icon)
  • for opening the text editing mode, automatically select all containing text (so that a click on a letter will override the existing text)
• (Janis) UserTask Editor: in a normal Text field, Enter should not leave the field but go to next line (next to Shift+Enter)
• if in text editing mode of any element, "ESC" key should close the text editing mode (instead of trying to close the UT editor).
• Delete icon not on top but on every element next to the edit symbol
  • also make the keyboard shortcut "Del" work for deleting
• instead of the Delete icon on top, have a Reset icon to reset the form to its default content (of course, first show a with a Hint modal with a warning)
• only for Empty Start Events Form Editor: add an icon in the top row to delete the attached form. This should only be possible if there are no variables that need to be defined at startup. this button closes the modal after deleting the start form.
• input element: remove "Type" setting and take the type from the variable definition instead.
• text element:
  • is it possible to allow markdown syntax like in our other text editors?
  • allow enumerations (e.g. with the markdown syntax)
• checkbox element improvements:
  • delete the gear symbol of each checkbox row. instead, make each checkbox row also selectable and show the settings if the row is selected
  • even when you are editing the text of a row, show the row settings (variable, adding rows, etc.) under the text editing elements
  • add icons+functionality to move the selected row one row up or down
  • additionally, show the icons for adding one new checkbox row at the bottom inside he checkbox element - inside the editor on the right side. this elements should be clickable in the editor, but it must not be rendered in the final UserTask at runtime. this is just for better visibility that someone can add new checkbox row with these icons.
  • is it possible to drag and drop checkbox elements (as new rows) into an existing checkbox group? (in the beginning, I tried that for adding new rows, because it was not intuitive for me how to add rows)
  • if the whole checkbox group is selected, also show the buttons for adding new rows on top or bottom under Settings on the left side
  • if the whole checkbox group or one row is selected, rename "Variable" to "Variable for all Checkbox Rows"
  • if one checkbox row is selected, rename "Value" to "Variable for the selected Checkbox Row"
• UserTask Editor buttons:
  • Other buttons (e.g. Cancel): add possibility to add further buttons ("Submit should always be there). These additional buttons should have multiple actions: 1. end UserTask by setting a variable (could be checked in a Gateway afterwards), 2. throw error and throw escalation events (to go to attached boundary events), 3. open a link in a new tab, (4. (low prio) trigger some internal script (which maybe calculates the remaining vacation days)
• Upload Files (PDFs, etc) and store them in variables + Engine + datastore (MS)
• Enable Scripting within the UserTask form. Example: live calc of the remaining vacation days, if the start and end are selected
• Add a new Drop-Down element in UserTask Editor for showing and selecting a list of values. For example, to select a list of available laptops. The drop-down should be connectable to an array from the process variables (type is called "List"). Options on the left side should be: "List to show" -> variable selection, "Store Selection in:" (here, the user needs to select in process variable the selection of the drop-down is stored), checkbox for "Multiple Selections are possible"
  • Extend variable definition of type "List": in the "Add a new Variable" modal, add the possibility for "List" variables to add starting/default values. Also look if adaption to the process engine are necessary for filling the array at instance start.
  • Make Global Data Objects also selectable for drop-down element: if a global data object is selected, the Display Name of all child elements on the first level under the selected Data Objects should be shown in the list. but not the value should be stored as the selection result, instead the name
• Be able to assign a employee's boss and back office as the executing person (TODO: concept/how)

Script Tasks:

• Bug: if a Script Task is interrupted/ended by a boundary event, e.g. timer boundary event, the currently existing local variables are not written into the process variables (there state gets lost)
• Editor: show in top in heading after "Edit Script Task:" the name of the ScriptTask
• Editor: improve syntax checking, so that it conforms to our api (e.g. network.post() has two optional arguments)

• JS Code improvements, so that the script task always 1. ends when all code was executed, and 2. is interruptable at any time (if a "Pause" or "Stop" is triggered):
  • offer a synchronous wait(ms) function for the user to use in script tasks. If the process has been paused by the user (via the MS) during the execution of wait(), return true, else return false. Remove await setTimeoutAsync() from editor (MS) and script execution (Engine) (Note: we still want an asynchronous version, but under a different name, see later). use a similar implementation like here: https://stackoverflow.com/questions/3969475/javascript-pause-settimeout -> if paused or stopped, this is how you can realize it
    • (Kai G.) test and put in documentation
  • change await setIntervalAsync() (MS Script Editor and Engine) to let id = setInterval(clbFunc, 10);. Return no Promise anymore, but instead the interval id which should be canceable with clearInterval(id);. This is how the normal setInterval() of Node.js or Web also works, but since we offer the functions, register that this function was called and works until it is canceled with clearInterval(id);. You can then reuse the normal setInterval()
    • a ScriptTask should not end if there are running intervals
    • setInterval() must be pauseable and stoppable, see wait()
    • (Kai G.) test and put in documentation
    • (low prio) Bug: the timer of setInterval() for the next interval only starts after the last execution ended. => if the user sets an interval for every 5 seconds, he expects the interval to run each 5s (and not 5s after the last execution)
      • (Kai G.) test and put in documentation
    • Bug: the usage of multiple setInterval() within one Script Task is currently synchronous, i.e. a long-running code (network call or wait()) blocks the start/execution of other intervals. this is wrong => if the interval code contains some of our long-running, synchronous functions (like network calls or wait()) it should only block/wait to execute the next commands that directly follow but not the parallel execution of other timers and intervals. You can test it by starting two 1s intervals and one having a wait of 5s inside the code - by using console.info() you can see in the timestamps that the 1s-interval is not executed within the 5s waiting of the other (-> is that possible to achieve?)
      • (Kai G.) test and put in documentation
  • make all of our own script functions (currently all network functions) synchronous (same name) -> no await should be needed, so don't return a promise. (Note: we still want an asynchronous version, but under a different name, see later). at runtime, since the network functions of a script task call our function implementation, the network function should still work asynchronously in the background (don't block the JS execution) but the user does not know. our implementation returns if the network request is answered, but within our implementation also have event-handlers for pausing and stopping an open request.
    • if Paused (during a network request that needs a little longer for the answer): return the answer to the ScriptTask after Pause is ended. So, store the answer if it arrived during pause.
    • if Stopped: end the script task and all open requests/promises/etc.
    • (Kai G.) document new script language in documentation
  • now, after the previous changes, everything is synchronous, which makes it is easier for us to achieve the goals 1. and 2. (Reason: we want to allow pure JavaScript/ECMAScript but not any further functions from the different runtimes like web or Node.js. And in pure JavaScript/ECMAScript, there are no async functions predefined. -> So, everything is synchronous.) But, this does not allow any parallel executions with asynchronous code anymore, if some execution needs a while (network request, wait, etc.). That's why we should should also allow our own async functions, if we know that they are still running (so, the ScriptTask will not end) and which we can interrupt with pause and stop.
    • Check if only pure JS/ECMAScript is allowed in the Engine for executing scripts? (no access to Web or Node APIs?) => goal: only pure JS and or self-implemented functions are allowed
    • offer asynchronous waitAsync(ms), network.getAsync()/postAsync()/putAsync()/deleteAsync()/headAsync() functions for the user to use in script tasks, which returns a promise. Like with the synchronous variants, make it pausable and stoppable.
      • (high prio) Bug (Engine) - Error handling: with the async network functions it seems that the error handling is not correctly implemented. Errors should be given to reject(). sometimes it seems to be done but not always. For example: it is not possible to .catch() the error if an DNS address does not exist and can not be resolved (e.g. https://abc.proceed-labs.org/). Also, other statuscodes than 2xx can be catched, but they additionally print an error message on the console (which should not be done, example: getAsync() on www.proceed-labs.org without user-agent header).
    • (Kai G.) document new script language in documentation

networkServer.getProcessInstanceListeningPath() OR networkServer.path() // return the "root" path for a process instance where it can receive requests, which should be /running-processes/<instance-uuid>/ 

// Standard way: 99% of all use cases
// open a server and wait exactly for one request. afterwards, close the server again
let {req, res} = networkServer.get|post|put|delete|head(path) // same as express, but as syncronous function (no callback, no await): https://expressjs.com/en/5x/api.html#app.METHOD
res.send('Hello World');

// 0,5% of use cases: there should also be an async variant with a promise, which can handle *one* incoming request asynchronously
// the ScriptTask should not close until this one request was handled
networkServer.getAsync|postAsync|putAsync|deleteAsync|headAsync(path).then( ... )

// 0,5% of use cases: there should be a way to create a server that handles *multiple* incoming requests -> use callback (i.e. it is async and other code can run afterwards)
// the ScriptTask should not close until the server object is closed with the script task!
let server = networkServer.get|post|put|delete|head(path, (req, res) => {
  res.send('Hello World');
  server.close(); // important to end the script task!
});

• Allow creating a network server with the previous script task code
  • Implement networkServer.getProcessInstanceListeningPath() and networkServer.path()
  • Implement that get|post|put|delete|head(path, [optional clb]) or getAsync|postAsync|putAsync|deleteAsync|headAsync(path) register a listening server in the Native Part of the Engine. After receiving a request to the instance path on /running-processes/<instance-uuid>/ or /running-processes/latest/<process-uuid>/, forward the request to all waiting ScriptTasks of the instance-uuid or of the latest instance of a process
    • usually, after one request to a specific method+path, remove it again from the open listening list. (if no callback is given as parameter.)
    • usually, 99% of all use cases will probably be with the synchronous server methods
    • if a function (callback, i.e. the functions gets asynchron) is given, a continuous server should be opened until it is closed with serverObj.close();. The script task should not close before that.
    • multiple, parallel ScriptTasks should be able to open a server for the same method+path within one process instance => send the incoming request to all listening ScriptTasks
    • multiple, Process Instances should be able to open a server for the same method+path. Usually an external requester would directly address one specific process instance, but he can also address the latest process instance => send the incoming requests to all listening ScriptTasks of the newest process instance
    • a ScriptTask is not allowed to end, if it has open listeners
  • if process is Paused: return 503 with body: "this process instance is currently paused and does not handle incoming requests"
  • if process is Stopped or no server code was created in ScriptTask: return 400 with body: "this process instance is currently paused and does not handle incoming requests"
  • req and res should be similar to the express objects: https://expressjs.com/en/5x/api.html#req.app and https://expressjs.com/en/5x/api.html#res.app BUT: we can not directly use the express objects because of the serialized IPC connection to the subprocess for a script task (which blocks sending functions). So, you don't need to implement everything, just a few things: all properties and no functions on req (read-only, thats why it is easy to serialize and transmit to the script task), these functions of res: cookie(), clearCookie(), end(), json(), location(), send(), set(), status() (maybe less, since one response method would also be sufficient. so, you could decide between end(), json(), send() )
  • (Kai G.) document the new server functions on the website

---

• JS and Blockly Editor: Improve looking up the code of other ScriptTasks: in the ScriptTask Editor, add an area to open other ScriptTasks of the same and other processes (for reading and copying code). ScriptTasks of the same process should be opened in read-write mode and from different processes only in read-only mode (show a little badge in the top for the mode).

  • (Bug: "Save All" button does (sometimes) not work to save changes made in other ScriptTasks of the same process -> but I don't know how to reproduce it. test if you can also experience it.)
  • (Low Prio) Is it possible to select and copy blocks from Blockly into another Blockly diagram?
  • The current process should always be on top of the process/file list. The other processes should be alphabetically sorted under it.
  • File List (on the left): show the name of the ScriptTask instead of the filename
  • Bug: the JS Editor windows are not separate from each other: if two variables with the same name are declared in two separate script task (e.g. let a;), the editor shows a "redeclare block-scoped variable" error

• Small UI Improvements in JS and Blockly Editor:

  • Bug: "Script syntax is valid" tag does not work -> it is also displayed if the JS code is syntactically wrong
  • JS Editor: put the "Script syntax is valid" tag to the right side, like in Blockly
  • Buttons to close the Editor Modal in the right bottom corner should be "Cancel" and "Save All". Both should close the Script Editor. "Save All" should be deactivated by default and activated if the user changes something in any editable code file. If "Cancel" is clicked and some code is not yet saved, show the existing question modal if you want to discard the changes.
    • Modal which questions the user if he wants to discard the changes: add a list of all files that have unsaved changes
  • (if possible) JS Editor: Don't be able to "Save" if there are syntactic JS errors. (But be able to save (JS) code if we don't know the objects or types. Reason: we don't have all possible objects registered in the monaco editor)
    • (if possible) JS Editor: Don't be able to "Save" if there are loose blocks in Blockly.
  • add an Editor Icon bar on top (on the left side of "Script syntax is valid" tag, above the content part of the editor). All icons should have tooltips:
    • "Save" icon to save the single open script
      • (low prio) catch "Ctrl + S" keys to also save the file
    • Blockly: make "Switch to JavaScript Editor" as an icon
    • make "Open Documentation" to an icon
    • JS: add an "Format" icon to format the code (this can already be done with "Ctrl + Shift + i")
    • JS (low prio, only if fast+easy): add an icon to show a modal with useful keyboard shortcuts coming from the monaco editor

• JS and Blockly Editor: add the ability to write code/scripts for the event that process Pause or Stop is clicked: Add two new (Tabs?/Sub-Files?) in the Script Editor to write code that is triggered if Stop or Paused is clicked (e.g. pausing or stopping the 3D printer).

  • Extend the PROCEED Engine to execute the Pause and Stop code if the command occurs.

• Engine: the setVariable() functions should consider the set format, e.g. number, date, url, etc.). maybe there can also be hints in the JS Editor, if the user tries to set a wrong format.

• (Low Prio) For every Script Task, enable the possibility to define a "Default Network Host" with IP OR URL in the Property Panel of the BPMN Editor inside the "Automation Properties"

  • Write the value into attribute proceed:defaultNetworkHost for the selected Script Task
    • Kai G.: document in XML schema
  • PROCEED Engine: always if networkRequest.get/put/post/delete/head(path[, options]) inside a script task have a path variable without a host part (e.g. just /api/users/) use the value of proceed:defaultNetworkHost as default for the target of the network requests (if it is a valid IP or domain). For example, network.get('/api/users/') should send the request to the IP 12.0.0.34 if this is set in the defaultNetworkHost attribute of the ScriptTask

Gateways:

• XOR-/OR-Gateway - setting condition instructions in the BPMN editor like in the old MS: the click on an outgoing sequence flow should show an icon in the middle if the top row to open a modal with input fields for 1. Description (mapped to name attribute of the sequence flow), 2. Condition (one-line editor), 3. Default Path (checkbox, overrides the value on the gateway if there was already another default set). The condition should be written into the BPMN like described here: https://docs.proceed-labs.org/developer/bpmn/bpmn-gateways#xor-and-or-splitting-gateways
• Splitting XOR/OR-Gateways with one Default Flow (Engine): usually all outgoing sequence flows need to have script conditions attached for execution. But it is also allowed to use a so-called Default Flow (configurable via settings)
  • Check if default flows work in the Engine: all conditions of all outgoing sequence flows must be evaluated first, before the the default flow is used -> if it not works, implement it.
  • please also implement that it is okay that the user only defines one default flow outgoing and no conditions attached on the other outgoing sequence flows. -> All sequence flows with no condition should result the GW evaluation to false
  • Splitting XOR/OR-Gateways: if no condition evaluates to true and there is not default flow, end this one token with ERROR-SEMANTIC
  • Critical Bug: if a GW condition tries to access a non-existing variable, the Engine shuts down. -> if a condition has a wrong syntax or tries to access a variable that does not exists, end this one token with ERROR-TECHNICAL
• Splitting XOR/OR-Gateways: make syntax consistent to Script Tasks with variable.get()
• Splitting XOR/OR-Gateways Editor: 1. add the possibility to see and select all existing variables (like in Script Task Editor). 2. Syntax checking should show no errors if you correctly get a variable (currently it marks it red)
• Splitting XOR/OR-Gateways: if the splitting Gateway is selected, show an icon in the top to open an GW-Editor (currently, it is only shown if an outgoing sequence flow is selected - which is the correct place but a bit user unfriendly). This GW-Modal should show and be able to edit all outgoing sequence flows (don't remove the condition editor from the outgoing sequence flow - it is also not problematic to keep it)
• Splitting XOR/OR-Gateways with full Script support: instead of only defining one condition line, allow to define complete Scripts like in the ScriptTasks
  • Adapt GW-Editor UI
  • store and serialize JS code like with Script Tasks
  • Enable execution in Engine

Automate several BPMN Elements:

• Loop Task Markers (Loop, Multiple Instance Parallel and Sequential): if the marker is selected for a task, add an icon in the top row next to the other editor icon (but with with loop or multiple instance symbol). This should open a modal where the user can set the loop characteristics. write it to the XML.
  • implement functionality in Engine
  • mark them blue in "Change element" list (wrench symbol) if executability is on activated
  • (Kai G.): documentation
• Extend the BPMN Editor with the definition of Error and Escalation Events like described here.
• Call Activities like in our docu:
  • BPMN Editor: Select other processes in the MS and link them in the XML
  • Deployment: Deploy main process and all linked processes
  • Instance View: Add icon to open Call Activities
• Optional Attribute for Activities/Tasks: In the Property Panel, under the section "Properties", add a checkbox to mark a selected Activity/Task as "Optional Task" with a question mark icon behind it explaining: "Marks a task as optional if it does not always need to be performed. For example, an employee does not always ask where an application form is kept if they already know." If activated, make the border of the selected Activity/Task dashed. Serialize this option as XML attribute proceed:optionalTask="true" inside the task element
  • Kai G.: document the XML serialization in XML schema
• Conditional Event: this is actually only a JS Script Execution
  • MS: if a Conditional Event is clicked, show icon on top and implement to open the Script Editor (JS+Blockly) for Conditional Events and store the scripts similar to Script Tasks (in the file list: show all Script Tasks and Conditional Event Code)
    • allow throw new ConditionSuccessful() keyword in monaco editor
    • (Kai G.) document throw new ConditionSuccessful() error event for Start and non-interrupting boundary events
  • Engine: implement to execute JS code for intermediate conditional events
  • Engine: implement to execute JS code for conditional interrupting boundary events. Start the conditional event script directly after the task was started. if the script ends, then stop (terminate) the attached task and trigger the token on the outgoing sequence flow
  • Engine: implement to execute JS code for conditional non-interrupting boundary events. Start the conditional event script directly after the task was started. if the conditional script ends, then trigger the token on the outgoing sequence flow but do not end the attached task. A non-interrupting boundary event can also occur multiple times. This should be done in a loop and every time the condition occurs the user/script can throw new ConditionSuccessful() error event. (or is there a better event triggering way than using an error?)
  • Engine: implement to execute JS code for conditional Start Events in the case that the process is activated. This is actually just starting the JS script. It should somehow pre-create the template for a Process Instance but without the times and ids. Every time either throw new ConditionSuccessful() is thrown (usually from inside a loop) or the Conditional Script ends, a new, complete Instance is created with new ids and the correct times. but also the set variables from the script should be transferred to the instance. (For now, the Engine should not automatically restart the script after it ends. This means, the condition can be fulfilled one time, a new process instance is created and afterwards the process is not activated anymore.)
  • MS - Engine: show and implement the activation and (automatic) deactivation of processes with start conditional events.
  • (Kai G.) document Conditional Events

Engine:

• if a Script Task has a syntactic or JS error, write the error to the engines log system. (Also, end the Script Task and this one token with ERROR-TECHNICAL. don't end the whole process - but I think, this is already the case.)
• PROCEED Engine: on a process split (OR/AND GW), the currently generated tokens ids get too long if the process contains a very big loop. Result: the log is nearly not readable anymore. Solution: the current way of generating tokens-ids should be stored in a different instance variable called tokenProcessPath (also for merges and Subprocesses, see wiki). the tokenId should always be just a seven-char string.
• If a defined Variable (in the Property Panel) has a Default Value, set this value at the beginning of the Process Instance
• If a defined Variable (in the Property Panel) is Required at Startup, don't start the Process Instance if it is not set within the start command. Send back semantic error and explanation.
• Create a list of predefined, read-only variables about the process and instance to be accessed inside the BPMN elements (probably it is enough to fill them at runtime on the Engine): process-id (definitions -> userDefinedId), process-name (definitions -> name), process-description (process -> documentation), task-id (current task), task-name (current task), task-description (documentation of current task), instance-id

Process Engines View:

• make adding Engine more user-friendly. 1. add input drop-down field in the modal for "Protocol" so that one protocol always has to be selected (default: http) 2. a placeholder in the location text input that shows the expected strings, e.g. "127.0.0.1 or my.domain.de", 3. add a "Port" input field with default to 33029 for http and 1883 for mqtt
• Process Engines View:
  • use the expendable row table to show the list of Engines with the following columns: Name, Location, Status, Protocol, Port, Actions (all fields sortable but Actions)
  • Only MQTT Servers should be expandable (not normal HTTP Engines): all discovered Process Engines connected to an MQTT-Server should be shown nested, under the MQTT Server (default: expanded)
  • MS Engines: Show Engines configured by the MS Admins (via the Admin Dashboard, not added by the Space) in a separate table and slightly grayed out (non-editable). This table should always be the last one on the site (under the other tables)
• MQTT-Server can contain multiple Engines. Especially in the case of an cloud MQTT-Server, a user should not be able to see and use all Engines of that server. (Example: PROCEED Labs will offer a public MQTT server where every user can add their own Engine for home automation. A User should not be able to see the other, registered Engines on the MQTT Server). TODO: how to do that? => insert unique ID of Engine - how to get it?

MS Admin Dashboard:

• Bug Fix: "Spaces" view on the left side is currently showing an error -> show all spaces
• "Process Engines" View with two tables: 1. first table: add/delete Process Engines that are valid for the whole MS and can be used inside all Spaces as their default automation Engine. 2. list all process engines added of all spaces with option to jump into the respective Space to manage the the engines
  • also show the connection status and more information in the tables
  • filter for Spaces should work
• "Execution View": List all Processes of every Engine of every Space. Icon for the Admins to be able to jump into the respective Space to manage the processes.
  • filter for Spaces should work
  • in the admin dashboard, the admins should not be able to see the instances or the process itself. just some meta data. the real data will only be seen if they jump into the space

Execution View:

• For deployment in the deployment table, only show processes that have a version (already done) AND that are executable. Find out if a process is executable with the isExecutable attribute of the main process element.
• For deployment in the deployment table, next (or under) the "Deploy Process" button of each row have a drop-down menu where a user can select a specific Engine to deploy to. Also add a row called "Automatic Engine Selection" which should be the visible default.
  • in the drop-down engine list, deactivate (light-gray) the Engines which already have the newest version
  • if "Automatic Engine Selection" is selected and the process already exists on one or more Engines (but in an older version), then deploy the newest version to all of these Engines.
  • if "Automatic Engine Selection" is selected and the process does not exists on any connected Engine, then deploy the newest version to a randomly Engines. But: if there are own space-related Process Engines (added in the "Process Engine" view of a space and not Engines added by the MS Admins for every space), then only randomly select one these own Engines and never the MS shared Engines.
• For deployment in the deployment table, make the table bigger and also the show the version no. and version creating date/time in columns (to know which version will be deployed)
• Execution table process deletion: before deletion, show a modal and let the user select what she wants with a drop-down list: "Delete all Stopped Processes on every Process Engine" (should delete non-running process versions and their instance data), "Delete all Processes (even if they are currently running) on every Process Engine", "Delete all Stopped Processes on a specific Process Engine" (should delete non-running process versions and their instance data), "Delete all Processes (even if they are currently running) on a specific Process Engine". if "specific" process engine is selected, show a second drop-down field to select an Engine. Implement this functionality.
• If there are processes (any version) on the connected Engines that are not yet locally stored/existing in the MS data, then download all (static) process data (not the instance data, only all versions of process diagrams, Scripts and UserTasks) to the MS database+filesystem. For the user to find these processes in the Editor UI, store the processes inside a folder called "Automatically Downloaded from Process Engines".
• Auto-Update of already deployed Processes: if a process is already deployed and a new version of this process is created, it should automatically be deployed to all Engines having this process. Do this by extending the db table for processes to store, which currently connected Engines have the process deployed. If a new version is created, deploy
• Write a generic validator that checks a process before it is deployed, see here: https://github.com/PROCEED-Labs/proceed/wiki/Process-Validation-Rules-(for-Import,-Style-Guide-and-Deployment). A process should also be deployable, if not every Task has Executions Instructions like HTML or Scripts attached. (This is useful for already starting a process if updates come later.) Show a hint in the case to the user and let him decide if he really wants to deploy the process version.
  • Quick Fix if we have no time: the deployment already fails if not every Script Task has a Script and not every UserTask has a HTML -> Show a good hint to the user why the deployment fails and what is missing.
• (TODO) users should be able to start and manage their own processes, even if they do not have the permissions for accessing the whole automation view. Some Ideas:
  • Either own automation view or every user has access to the Execution view which is filtered to only see the processes that the user started

Instance View:

• Sort the Instances: newest always first
• (High Prio) reorganize the Start/Resume/Pause/Stop Icons:
  • Add a new icon: maybe tick symbol. It should activate and deactivate a process for execution in the case that there are non-empty start events, i.e. currently only for Timer Start events. (This events gets triggered automatically and create new instances)
  • instead of having the Start icon (Plus Symbol) in the left icon group, separate it in a new 2-icon-group with the Activate icon. it should be in the middle between the filter group and the Pause/Stop group
  • make the Resume/Pause/Stop group to a 2-icon-group. Show the Resume icon (Play symbol) instead of the Pause icon if the process is paused.
• Show the editor icons of the BPMN Editor but open the editors in read-only mode. beneath the Resume/Pause/Stop icons in the middle: Script Task, User Tasks, Events, Gateways, Loops, Open Subprocesses. Next to info icon: XML icon
• Question or TODO: if the newest process version is deployed to multiple Engines, will the start of an Instance then only start one instance or multiple ones? -> should only start one instance
  • (low prio) Add an icon like the "Filter by version" icon for "Selecting a Process Engine". if the newest process version is deployed to multiple Engines, activate it. (if it is only on one Engine, the icon should not be visible.) There should be an entry "Automatic Selection" (default) and all Engine with the deployed version.
• "Submit" button of the start form needs very long and the user gets no feedback if he already clicked the button. => Close the start form and show a loading symbol.
• Bug Versioning: in the instance view, the shown version is different (automatically counted up) then what you create as a version in the process editor. Add this automatically counted version in the process editor versioning dialog as a non-editable field. Make "Name" and "Description" not required. In the Instance View, also show the "Name" next to the version no. and show the "Description" as tooltip. Extend the execution table with the self-defined "Version Name".
• Show process logs somewhere
• Show execution instructions (HTML code, scripts, etc.) somewhere
• Update the Property Panel and fill the areas with data from the instance
• Show variables and their current value
  • Implement a way to change the process variables from the MS
• in the shown process diagram of an instance, mark all the tasks with a orange dashed border that have no execution instructions. (Then, the user is able to see where the process will stop and wait.)
• Bug: the overall Instance State in the Property Panel is wrong if there is a process split and multiple tokens exist. If one token ended, the state in the Property Panel shows "Ended" even if other tokens are still active. Show the instanceState in the property panel.
• the overall Instance State in the Property Panel is "Aborted" if there was a Terminate Event. That is correct. But don't show it with a red background instead with a black background. And rename it to "Terminated"
• if a Technical Error occured inside a ScriptTask (e.g. because of a JS error), show the error message. This is currently only visible in the logs of the Engine but we also need that in the MS, so that the Script developer knows the error. Show it somewhere in the Property Panel

Better Workflow for Automation Development:

• Instance Viewer: add a button to jump directly to the BPMN Editor for editing the process
• BPMN Editor: Define processes as executable: in the Property Panel of the BPMN Editor inside the "Automation Properties" tab add a checkbox on top as the first element to "Activate Automation for the entire Process". This checkbox should be always visible (when no or any BPMN element is "selected" in the process). This checkbox should set the <process> element's attribute to isExecutable to "true" or "false" (attention: a checkbox's change must be written to the attribute of the <process> element and not to any other attribute even if some BPMN element is selected). If false, deactivate the input fields of the automation tab and hide the icon for setting the execution instructions in the top row. Do not delete any existing values. (Currently, isExecutable is always "true" and can not be set in the UI. New: the default should be "false" if creating a new process diagram.)
  • store the value if isExecutable not inside the BPMN, but instead in the db table for fast filtering. in the BPMN store a variable like we already do with other important BPMN attributes. This will be replaced if the XML is needed, e.g. Export or in the XML Viewer.
• BPMN Editor: If process is set as executable, only show BPMN Elements that can already be used in automation in PROCEED (plus all annotation elements that have no execution meaning)
• BPMN Editor: If process is set as executable, show a new button in the versioning dialog of the BPMN Editor to "Version and Deploy" directly from the BPMN editor -> TODO: what is shown after clicking this button (probably: Engine selection and Instance View)
• BPMN Editor: If process is set as executable, show a new button in the BPMN Editor to "Test Deploy and Start" of the latest process diagram. Show an Engine selection dialog. This directly deploys the latest version, starts an Instance and show the Instance Viewer. (Deploying latest is an exception and should not be possible via the Execution View. There, only versioned processes should be deployable.)
• BPMN Editor: if clicking a Task (currently only UserTask and ScriptTask useful): in the Automation Tab, add a checkbox to "Skip this Task during Execution (useful for development)". if multiple tasks are selected, also show this checkbox to skip/enable all of them at once. Serialize in BPMN as attribute skipExecution="true|false"
  • Engine: implement in the Engine that these tasks are skipped and not executed
  • (Kai G.) document in XML Schema
• Script Task Editor: add a play button to directly execute the opened script task. Therefore, only deploy the script task to the Engine and execute it. The script task should not be visible in the list of deployed processes of the engine (e.g. in execution view). This execution should not produce a full "process" instance - just a task instance. Add a button to stop the execution. The Script Task should also be removed from the Engine directly after the execution (if successful or not)
  • If the Script Task uses variables, show the variable pop-up (from non-start events) to fill all variables. Transmit the variables to the Engine and start the script task execution
• Instance View: Right-click on a Script Task should show an option to "Start Script Task only". Implement in the Engine that only the selected Script Task can be executed (from the already deployed process). Stop button should be activated if the script is running. This execution should not produce a full "process" instance - just a task instance.
  • If the Script Task uses variables, show the variable pop-up (from non-start events) to fill all variables. Transmit the variables to the Engine and start the script task execution

Intermediate and Boundary Timer Events

• - setting duration in the BPMN editor like in the old MS: the click on an Intermediate or Boundary Timer Event should show an icon in the middle of the top row to open the Duration modal (use existing duration modal of the property panel). The condition should be written into the BPMN like described here: https://docs.proceed-labs.org/developer/bpmn/bpmn-timer#durations

Non-typed Start Events

• Connect Non-typed Start Events to User Task Editor for defining a UI. Similar Serialization and Storage as for UserTasks: in <process ... proceed:uiForNontypedStartEventsFileName="" >
  • Process Start: if a User wants to create a new Instance, show the User Task in the MS before Starting the Process. Click on "Start" triggers a process Start on the Engine
  • (Kai G.): document serialization in XML Schema
• Set Process Variables when Starting a Process Instance:
  • Define Process Variables in the BPMN Editor: in the Property Panel of the BPMN Editor inside the "Automation Properties" tab add a table to define Process Variables with Name, Data Type (Text, Number, On/Off), Default Value. Use List and Design like with Milestones.
  • Defined Variables are valid for the whole Process and all FlowNodes
  • Serialization in BPMN (spec is not enough)
  • (Kai G.): document serialization in XML Schema

MQTT Connection in MS:

• have an MQTT Subscriber functionality in the backend that is triggered by the frontend and running the whole time as long as there is at least one MQTT Engine registered under Process Engines
  • continuously ping from Frontend the MQTT Subscriber (this maybe prevents Google Cloud Run from shutting it down very fast)
  • for Streaming use SSE in Next.js. Problem: Google Cloud Run can shut down the connection anytime. Therefore, die Frontend maybe misses some data. Two Solution possible: 1. if SSE Connection closes, reestablish a new one, 2. preemptively close the SSE connection and reestablish a new one
  • Subscription Problem: Google Cloud Run can shut down the connection anytime and we miss some data.
  • At MQTT Subscriber startup and after reestablishing the connection to an MQTT Server, fetch the following data via the pms/engine//api/ of every Engine and store it in the DB:
    • all deployed processes: only fetch the BPMN of the newest version, store the number of versions and the Engine-ID where they are deployed
    • Instance data: all instance data of running and ended instances of the newest process version. Store it without the log part inside of the instance object. Therefore improve Engine Endpoint:
      • add query to Engine endpoint /process/{definitions}/instance/{instanceId}?log=false and document in wiki
      • /process endpoint to only list an array of processes with some few meta data like described in the wiki (no bpmn or instance data => pay attention, where the endpoint is already used in PROCEED)
      • /process/<id>/versions endpoint to only list an array of processes with some few meta data like id and creation-date. update wiki
    • only fetch and store the data, if not already in db
  • Subscribe to the following topics:
    • ...
  • Store the following subscription data in the DB:
    • ...
  • Streaming the following data to the frontend with SSE:
    • process log