Revise documentation

doc/calibration_tools.rst

@@ -0,0 +1,62 @@
+.. _calibration_tools:
+
+Calibration tools
+=================
+
+Typically, users are expected to use PEtab Select via a calibration tool. Hence, we recommend
+reading the model selection documentation page for the chosen calibration tool.
+
+Listed below are the supported calibration tools. If you want to use an alternative calibration
+tool, then see :doc:`examples/other_model_types` for an example of that.
+
+
+BasiCO (COPASI)
+---------------
+
+.. image:: https://raw.githubusercontent.com/copasi/basico/refs/heads/master/docs/_static/COPASI_Conly_176x176.png
+   :alt: COPASI logo
+   :height: 80px
+   :target: https://basico.readthedocs.io/
+
+`BasiCO <https://basico.readthedocs.io/>`__ is a Python-based tool that is an interface to
+`COPASI <https://copasi.org/>`__.
+
+- `Documentation <https://basico.readthedocs.io/>`__
+- `Model selection guide <https://basico.readthedocs.io/en/latest/notebooks/Working_with_PEtab.html#Model-Selection>`__
+- `SBML support <https://sbml.bioquant.uni-heidelberg.de/Submission/Details/62>`__ (via COPASI)
+
+Data2Dynamics
+-------------
+
+`Data2Dynamics <https://github.com/Data2Dynamics/d2d>`__ is a MATLAB-based tool.
+
+- `Documentation <https://github.com/Data2Dynamics/d2d>`__
+- `Model selection guide <https://github.com/Data2Dynamics/d2d/wiki/Model-selection-with-PEtab-Select>`__
+
+PEtab.jl
+--------
+
+.. image:: https://raw.githubusercontent.com/sebapersson/PEtab.jl/main/docs/src/assets/logo_light.png
+   :alt: PEtab.jl logo
+   :height: 80px
+   :target: https://sebapersson.github.io/PEtab.jl
+
+`PEtab.jl <https://sebapersson.github.io/PEtab.jl>`__ is a Julia-based tool.
+
+- `Documentation <https://sebapersson.github.io/PEtab.jl>`__
+- `Model selection guide <https://sebapersson.github.io/PEtab.jl/stable/pest_select/>`__
+- `SBML support <https://sebapersson.github.io/SBMLImporter.jl/stable/support>`__
+
+pyPESTO
+-------
+
+.. image:: https://raw.githubusercontent.com/ICB-DCM/pyPESTO/main/doc/logo/logo.png
+   :alt: pyPESTO logo
+   :height: 80px
+   :target: https://pypesto.readthedocs.io/
+
+`pyPESTO <https://pypesto.readthedocs.io/>`__ is a Python-based tool.
+
+- `Documentation <https://pypesto.readthedocs.io/>`__
+- `Model selection guide <https://pypesto.readthedocs.io/en/latest/example/model_selection.html>`__
+- `SBML support <https://sbml.bioquant.uni-heidelberg.de/Submission/Details/71>`__ (via AMICI)

doc/examples.rst

@@ -1,19 +1,34 @@
 Examples
 ========
 
-There are currently notebooks aimed at users and developers.
+For users
+---------
 
-Users that are working with PEtab problems should consult the model selection documentation of the calibration tool they wish to use. In this case, there is no need to use the PEtab Select package directly to perform model selection. After model selection, results can be analyzed with PEtab Select. See the API documentation for :mod:`petab_select.analyze`, or the "Visualization gallery" notebook.
+The examples for users cover setting up the PEtab Select files for different model classes, and visualization of model selection results.
 
-Users who wish to apply model selection methods using a calibration tool that doesn't support PEtab can refer to the "Model selection with non-SBML models" notebook for a demonstration using statsmodels.
+If you use one of the supported calibration tools, then we recommend using PEtab Select via the interface provided by the :ref:`calibration_tools`. Otherwise, see the example with an unsupported calibration tool.
 
-Developers wishing to implement model selection in their tool via PEtab Select can consult the notebooks on workflows ("Example usage with the CLI" and "Example usage with Python 3") or "FAMoS in PEtab Select".
+After model selection, results can be analyzed with PEtab Select. See the API documentation for :mod:`petab_select.analyze`, or the "Visualization gallery" notebook.
 
 .. toctree::
    :maxdepth: 1
 
-   examples/visualization.ipynb
-   examples/other_model_types.ipynb
-   examples/example_cli_famos.ipynb
-   examples/workflow_cli.ipynb
-   examples/workflow_python.ipynb
+   An ordinary differential equation model with time-series data. <examples/ode_timeseries.rst>
+   A polynomial (non-SBML) model with an unsupported calibration tool. <examples/other_model_types.ipynb>
+   A model space with multiple structurally different models. <examples/multiple_supersets.rst>
+   A large-scale, real-world problem with steady-state data. <examples/test_case_0009.rst>
+   A demonstration of all visualizations of model selection results. <examples/visualization.ipynb>
+
+For developers
+--------------
+
+Developers that want to implement model selection in their tool via PEtab Select can consult the notebooks for the CLI or Python interface. The FAMoS notebook is a complicated example that ensures your tool communicates with PEtab Select correctly over multiple iterations of model selection.
+
+.. toctree::
+   :maxdepth: 1
+
+   Example for the Python interface <examples/workflow_python.ipynb>
+   Example for the CLI <examples/workflow_cli.ipynb>
+   Example with the FAMoS method <examples/example_cli_famos.ipynb>
+
+Developers may also benefit from viewing the notebook that demonstrates an `alternative calibration tool <examples/other_model_types.ipynb>`_.

doc/examples/multiple_supersets.rst

@@ -0,0 +1,95 @@
+.. _multiple_supersets:
+
+Multiple superset models
+========================
+
+The PEtab Select model space table is designed to express large model spaces based on a single superset model, concisely. However, model selection problem can also involve multiple superset models that are structurally different. For example, one superset may be a linear model, and the other superset may be an ordinary differential equation (ODE) model. To express this concisely, we suggest using two model space tables; one per superset.
+
+We illustrate this with two of the other examples in this documentation, specifically:
+
+- :ref:`ode_timeseries_example`: an ODE model with parameters ``k1`` through ``k3``, and
+- :doc:`other_model_types`: a polynomial model with parameters ``k1`` through ``k5``.
+
+Example file layout
+-------------------
+
+The PEtab Select problem YAML file and the two model space files are kept in a top directory. The PEtab problems for each superset model are kept in separate directories.
+
+.. code-block:: text
+
+   model_selection_problem/
+   ├── petab_select_problem.yaml
+   ├── model_space_polynomial.tsv
+   ├── model_space_ode_timeseries.tsv
+   ├── polynomial_petab_files/
+   │   └── ...
+   └── ode_timeseries_petab_files/
+       └── ...
+
+The PEtab Select problem YAML
+-----------------------------
+
+The PEtab Select problem YAML lists both model space files.
+
+.. code-block:: yaml
+
+   format_version: 1.0.0
+   criterion: AIC
+   method: brute_force
+   model_space_files:
+   - model_space_polynomial.tsv
+   - model_space_ode_timeseries.tsv
+
+The model space tables
+----------------------
+
+The model space can be defined in a single model space table; however, this can lead to sparse tables. Hence, we suggest splitting the table into one per superset.
+
+``model_space_polynomial.tsv``:
+
+.. list-table::
+   :header-rows: 1
+
+   * - model_subspace_id
+     - model_subspace_petab_yaml
+     - k0
+     - k1
+     - k2
+     - k3
+     - k4
+     - k5
+   * - polynomial
+     - polynomial_petab_files/problem.yaml
+     - 0;estimate
+     - 0;estimate
+     - 0;estimate
+     - 0;estimate
+     - 0;estimate
+     - 0;estimate
+
+
+``model_space_ode_timeseries.tsv``:
+
+.. list-table::
+   :header-rows: 1
+
+   * - model_subspace_id
+     - model_subspace_petab_yaml
+     - k1
+     - k2
+     - k3
+   * - ode_timeseries
+     - ode_timeseries_petab_files/petab_problem.yaml
+     - 0;estimate
+     - 0;estimate
+     - 0;estimate
+
+Together, these two model spaces define :math:`2^6 + 2^3 = 72` candidate models.
+
+Notes
+-----
+
+- **Method** The stepwise methods (``forward``, ``backward``)
+  move between models will not visit model subspaces with a different ``model_subspace_petab_yaml`` than the
+  initial model. Hence, multiple stepwise searches initialized in the model subspaces associated with each superset model are required to explore the full space. ``brute_force`` will search all models regardless.
+- **Criteria** The information criteria (AIC etc.) require that models are compared on the same dataset. Hence, the PEtab problems inside ``polynomial_petab_files/`` and ``ode_timeseries_petab_files/`` should use the same data.

doc/examples/ode_timeseries.rst

@@ -0,0 +1,116 @@
+.. _ode_timeseries_example:
+
+Example: ODE with timeseries data
+=================================
+
+In this example, we provide the PEtab and PEtab Select files for a model selection problem involving an ordinary differential equation (ODE) model and timeseries data. `All files are available for download <https://github.com/PEtab-dev/petab_select/tree/main/doc/examples/ode_timeseries>`_.
+
+
+The ODE model
+-------------
+
+The model is describes the dynamics of two biochemical species: ``x1`` and ``x2``. There are three hypothetical reactions:
+
+1. import of ``x1`` at a constant rate :math:`k_1` (:math:`\varnothing \rightarrow x_1`),
+2. conversion of ``x1`` into ``x2`` at rate :math:`k_2 x_1` (:math:`x_1 \rightarrow x_2`), and
+3. export of both species through their interaction at rate :math:`k_3 x_1 x_2` (:math:`x_1 + x_2 \rightarrow \varnothing`).
+
+Assuming mass-action kinetics, the the ordinary differential equation (ODE) model is
+
+.. math::
+
+   \frac{\mathrm{d}x_1}{\mathrm{d}t} &= k_1 - k_2 x_1 - k_3 x_1 x_2, \\
+   \frac{\mathrm{d}x_2}{\mathrm{d}t} &= k_2 x_1 - k_3 x_1 x_2,
+
+where :math:`t` is time.  The initial condition is :math:`x_1(0) = x_2(0) = 0`.
+
+Hence, the three hypothetical reactions are each controlled by a single parameter (:math:`k_1`, :math:`k_2`, or :math:`k_3`) and the model selection problem is to identify the raections (parameters) that are sufficient to explain the data.
+
+In the files, the model is encoded in SBML, named ``model.xml``.
+
+The data
+--------
+
+The (synthetic) experiment was conducted by collecting measurements of :math:`x_2` at six time points.
+
+The PEtab measurement table is:
+
+.. csv-table::
+   :file: ode_timeseries/measurements.tsv
+   :delim: tab
+   :header-rows: 1
+
+The observable model
+--------------------
+
+As :math:`x_2` is observed with measurement noise, we include a noise model.
+
+The observable is
+
+.. math::
+
+   y_{\text{obs\_x2}} = x_2.
+
+The measurements are assumed to be the observable plus some Gaussian-distributed noise:
+
+.. math::
+
+   \bar{y} = y_{\text{obs\_x2}} + \varepsilon, \qquad
+   \varepsilon \sim \mathcal{N}(0, \sigma_{x2}^2),
+
+where the Gaussian standard deviation :math:`\sigma_{x2}` is an estimated parameted.
+This Gaussian noise model is what defines the likelihood that PEtab Select uses
+to compute model selection criteria.
+
+The PEtab observable table is:
+
+.. csv-table::
+   :file: ode_timeseries/observables.tsv
+   :delim: tab
+   :header-rows: 1
+
+The estimated parameters
+------------------------
+
+Each of the estimated parameters are specified in the PEtab parameters table.
+
+.. csv-table::
+   :file: ode_timeseries/parameters.tsv
+   :delim: tab
+   :header-rows: 1
+
+This table is adapted by PEtab Select to represent the different models in the model space. For example, the model ``M1_0`` in the model space table does not estimate :math:`k_2` and instead sets it to ``0``. PEtab Select implements this by changing the ``nominalValue`` of ``k2`` to ``0``, and turning off parameter estimation for ``k2`` by setting ``estimate`` to ``0``.
+
+The model selection problem
+---------------------------
+
+The PEtab parameter table (``parameters.tsv``) contains the rate constants
+:math:`k_1, k_2, k_3` (and the noise parameter :math:`\sigma_{x2}`, which is
+always estimated).
+
+The model selection problem is to determine which of the parameters :math:`k_1`, :math:`k_2`, and :math:`k_3` should be estimated in order to explain the data well.
+
+This is encoded in the model space file. Each row is a
+candidate model (a model subspace), and the columns ``k1``, ``k2``, ``k3`` give
+the value each parameter takes: either a fixed value, or ``estimate`` to indicate that the parameter should be estimated.
+
+.. csv-table::
+   :file: ode_timeseries/model_space.tsv
+   :delim: tab
+   :header-rows: 1
+
+For example, ``M1_0`` fixes all three rate constants to 0 (in this model, this means the associated reactions are turn off),
+``M1_7`` estimates all three, and the middle rows encode models that estimate different subsets. When a parameter is not estimated, it is fixed
+to a value (e.g. ``k1 = 0.2``, ``k2 = 0.1``, ``k3 = 0``).
+
+In this example, we implement a single model per model subspace. A more concise definition of the model space is presented in :doc:`other_model_types`.
+
+The PEtab Select problem YAML combines all files and remaining information required to define the model selection problem.
+
+.. literalinclude:: ode_timeseries/petab_select_problem.yaml
+   :language: yaml
+
+Here, models are compared by the Akaike information criterion (``AIC``), and the
+model space is explored with the ``forward`` method — starting from the
+smallest model and iteratively adding estimated parameters as long as the AIC
+improves.

doc/examples/model_selection/calibrated_M1_4.yaml → doc/examples/ode_timeseries/calibrated_M1_4.yaml

@@ -6,7 +6,7 @@ model_subspace_indices:
 criteria:
   AIC: 15
 model_hash: M1_4-000
-model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+model_subspace_petab_yaml: petab_problem.yaml
 estimated_parameters:
   k2: 0.15
   k3: 0.0

doc/examples/model_selection/calibrated_M1_7.yaml → doc/examples/ode_timeseries/calibrated_M1_7.yaml

@@ -6,7 +6,7 @@ model_subspace_indices:
 criteria:
   AIC: 20
 model_hash: M1_7-000
-model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+model_subspace_petab_yaml: petab_problem.yaml
 estimated_parameters:
   k1: 0.25
   k2: 0.1

doc/examples/model_selection/calibrated_models_1.yaml → doc/examples/ode_timeseries/calibrated_models_1.yaml

@@ -6,7 +6,7 @@
   criteria:
     AIC: 180
   model_hash: M1_0-000
-  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   estimated_parameters: {}
   iteration: 1
   model_id: M1_0-000
@@ -23,7 +23,7 @@
   criteria:
     AIC: 100
   model_hash: M1_1-000
-  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   estimated_parameters: {}
   iteration: 1
   model_id: M1_1-000
@@ -41,7 +41,7 @@
   criteria:
     AIC: 50
   model_hash: M1_2-000
-  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   estimated_parameters: {}
   iteration: 1
   model_id: M1_2-000

doc/examples/ode_timeseries/measurements.tsv

@@ -0,0 +1,7 @@
+observableId	simulationConditionId	measurement	time
+obs_x2		0	0
+obs_x2		0.19421762	1
+obs_x2		0.0484032	5
+obs_x2		0.61288016	10
+obs_x2		4.07930835	30
+obs_x2		10.12008893	60

doc/examples/ode_timeseries/observables.tsv

@@ -0,0 +1,2 @@
+observableId	observableFormula	noiseFormula
+obs_x2	x2	sigma_x2

doc/examples/ode_timeseries/parameters.tsv

@@ -0,0 +1,5 @@
+parameterId	parameterName	parameterScale	lowerBound	upperBound	nominalValue	estimate
+k1	k_{1}	lin	0	1e3	0.2	1
+k2	k_{2}	lin	0	1e3	0.1	1
+k3	k_{3}	lin	0	1e3	0	1
+sigma_x2	\sigma_{x2}	lin	1e-5	1e3	0.15	1