New Feature: objective formulas (draft)

[michaellans]

This draft PR lets users add formulas as badger objectives.

It also enables modifying the badger environments to specify observables as a dictionary with default rules, but should be backwards-compatible with all current environments

Key Changes:

• Gives users the ability to create and add formulas of observables in the GUI

  • Formulas can contain observables as well as other formulas, and use functions from python.statistics, python.math and numpy libraries, numbers, and standard math symbols
  • Formulas are written as text and can be renamed and edited. Variable names in formulas (including other formulas) should be backticked, for example mean(`x1`)
  • "Add Formula" button opens a new BadgerFormulaDialog QDialog window
  • Checks that only allowed operators and functions are used, and that the expression is syntactically valid
    • see formula_utils.py validate_formula
    • Adapted from https://github.com/slaclab/trace/blob/main/trace/utilities/formula_validation.py
    • This raises an error if it finds disallowed AST node types or operators, complex nested expressions, very long lists/tuples, non-allowed functions, or undefined symbols, or if the expression cannot be parsed as valid Python.
  • Formulas are stored as a dictionary of {name: {"formula_str": equation as text, "variable_mapping": dict of name: formula_str or None}}
    • The variable mapping maps formula names like x1 with a PV name or another formula.

• Redesigned ObjectiveTable to separate data from UI representation. Replaced with ObjectiveListView:

  • ObjectiveListView is a QScrollArea composed of a series of ObjectiveRowWidgets. The ObjectiveListView handles displaying the table, filtering, adding new items, updating items, and connecting external signals. It stores a list of all ObjectiveItems (for each row widget), and a list of currently displayed rows.
  • Each ObjectiveRowWidget represents one row in the table as a horizontal layout with the UI elements, and an ObjectiveItem stores the data
  • When a new badger routine is created, the selected items are returned as a dictionary of {item.name: {"rule": item.rule, etc. }}

• Modified Badger Routine class to map formula names to an expanded formula string in terms of base variables

  • formulas are unpacked by recursively substituting backticked variable names within the formula with their mapped variable or formula name. This returns a forward mapping of name to expanded formula, a reverse mapping, and list of base observables

Still to do:

• Refactor ObjectiveListView into a parent ObservableListView class and ObjectiveListView and ConstraintListView children, this should be pretty straightforward
• Some backend improvements to the formula logic
  • One area of improvement is how formulas are stored and expanded, and processed. See yaml screenshot for current example.
  • Currently formulas are expanded recursively using the variable mapping, which can probably be simplified
  • How to handle scalar vs buffer data needs to be thought out a bit more, I've partially implemented this as "statistics" for non-formula objectives but left this disabled by default for the PR. In the current state it adds quite a bit of complexity.
  • Better formula validation and processing, including security
  • Modifying environments/interfaces to request lists of PVs from epics rather than iterating through each would speed up performance, especially on initialization.
• A couple small bugs
  • formula validation fails if you reference another formula AND separately add one of the variables within that formula, I think this is an issue with the backticking/sanitization parsing
  • detect/prevent circular dependencies when formulas and mappings are created
• Alongside formulas, this enables some changes to the badger environments, including specifying default rules for observables, and defining formulas in environments.
• Deleting formulas or user-added observables
• The formulas are all stored as text and then expanded when the routine runs, then passed to the environment as something like mean(`some:pv`) + 2. Environment.get_observables interprets any observable name with backticks as a formula, and processes it using Formula.interpret_expression(). I think this whole process can likely be improved/simplified and mapped out more clearly.
• There are a few different changes here, the objective table redesign could be separated from the implementation of formulas.

• Example of how formulas and variable mappings are currently stored: