diff --git a/doc/examples.rst b/doc/examples.rst index f053079..a4fb82c 100644 --- a/doc/examples.rst +++ b/doc/examples.rst @@ -17,6 +17,7 @@ After model selection, results can be analyzed with PEtab Select. See the API do A polynomial (non-SBML) model with an unsupported calibration tool. A model space with multiple structurally different models. A large-scale, real-world problem with steady-state data. + Model selection with multi-parameter hypotheses (metaparameters). A demonstration of all visualizations of model selection results. For developers diff --git a/doc/examples/metaparameters.rst b/doc/examples/metaparameters.rst new file mode 100644 index 0000000..31ea3db --- /dev/null +++ b/doc/examples/metaparameters.rst @@ -0,0 +1,77 @@ +Example: metaparameters +======================= + +Metaparameters are a PEtab Select feature that enables model selection with +model hypotheses that involve multiple parameters. + +An example use case would be a model selection problem where one hypothesis is that +a process with a Michaelis-Menten kinetic exists. Since the Michaelis-Menten kinetic :math:`\frac{V_{\mathrm{max}}\, s}{K_m + s}` (parameters :math:`V_{\mathrm{max}}` and :math:`K_m`, and state variable :math:`s`) involves two parameters, both need to be set to be estimated simultaneously to include the process in the model, or set to ``0`` simultaneously to turn the process off and ensure the correct number of parameters are counted when computing model selection criteria such as the AIC. + +We show an example of this below. As metaparameters only affect the PEtab Select files, we only show those. + +Without metaparameters +---------------------- + +As ``Vmax`` and ``Km`` need to be toggled on or off simultaneously, without metaparameters, we require two model space rows. + +.. list-table:: + :header-rows: 1 + + * - model_subspace_id + - model_subspace_petab_yaml + - k1 + - Vmax + - Km + - k3 + * - M_mm_off + - petab_problem.yaml + - 0;estimate + - 0 + - 0 + - 0;estimate + * - M_mm_on + - petab_problem.yaml + - 0;estimate + - estimate + - estimate + - 0;estimate + +With metaparameters +------------------- + +Instead, we can define a metaparameter ``mm`` in the problem YAML, which represents both ``Vmax`` and ``Km``. + +.. code-block:: yaml + + format_version: 1.0.0 + criterion: AIC + method: forward + model_space_files: + - model_space.tsv + metaparameters: + mm: + - Vmax + - Km + +This enables concise specification in the model space table. ``Vmax`` and ``Km`` are assigned the same value as ``mm``. + +.. list-table:: + :header-rows: 1 + + * - model_subspace_id + - model_subspace_petab_yaml + - k1 + - mm + - k3 + * - M + - petab_problem.yaml + - 0;estimate + - 0;estimate + - 0;estimate + +Notes +----- + +During model selection steps (e.g. with the ``forward`` and ``backward`` methods), a metaparameter is treated as a single parameter. This enables model selection in terms of selection of model hypotheses, rather than the individual parameters associated with hypotheses. + +However, during computation of criteria such as the AIC, then the correct number of estimated parameters is used, i.e., with ``Vmax`` and ``Km`` in the example above if ``mm_R2`` is set to be estimated. diff --git a/doc/index.rst b/doc/index.rst index 40de99d..e440059 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -41,7 +41,7 @@ Python package or command line interface. An example of this is provided for the For users, in most cases, model selection is performed entirely via one of the calibration tools, -rather than the PEtab Select package directly. Hence, we recommend reading the `documentation from your calibration tool on model selection `_. +rather than the PEtab Select package directly. Hence, we recommend reading the :doc:`documentation from your calibration tool on model selection `. After model selection, the analysis and visualization methods in the PEtab Select package can be used directly with the results from your calibration tool. diff --git a/doc/problem_definition.rst b/doc/problem_definition.rst index 90f5444..2d3e7ad 100644 --- a/doc/problem_definition.rst +++ b/doc/problem_definition.rst @@ -22,7 +22,8 @@ A YAML file with a description of the model selection problem. criterion: # string. method: # string. model_space_files: # list[string]. Filenames. - candidate_space_arguments: # list[string] (optional). Filenames. + candidate_space_arguments: # dict[string] (optional). + metaparameters: # dict[string, list[string]] (optional). - ``format_version``: The version of the model selection extension format (e.g. ``1``) @@ -32,11 +33,15 @@ A YAML file with a description of the model selection problem. (e.g. ``forward``) - ``model_space_files``: The filenames of model space files. - ``candidate_space_arguments``: Additional arguments used to generate - candidate models during model selection. For example, an initial candidate - model can be specified with the following code, where + candidate models during model selection, used by the model selection methods. + For example, an initial candidate + model can be specified with the code below, where ``predecessor_model.yaml`` is a valid :ref:`model file `. Additional arguments are provided in the documentation of the ``CandidateSpace`` class, and an example is provided in - `test case 0009 `_. + `test case 0009 `_. +- ``metaparameters``: Metaparameter definitions (optional). Each metaparameter + (key) groups multiple PEtab parameters (value, a list of PEtab parameter + IDs), to represent a multi-parameter hypothesis. See :ref:`section-metaparameters`. .. code-block:: yaml @@ -117,6 +122,51 @@ can be specified like - 0;estimate - 10;20;estimate +.. _section-metaparameters: + +Metaparameters +^^^^^^^^^^^^^^ + +PEtab Select supports multi-parameter hypotheses via metaparameters. + +For example, a model selection problem might include the research question: Should I include a Michaelis-Menten kinetic in my model to help explain the data? The Michaelis-Menten kinetic looks like :math:`\frac{V_{\mathrm{max}}s}{K_M s}`, where :math:`s` is some state variable (e.g. a biochemical species), :math:`V_{\mathrm{max}}` and :math:`K_M` are parameters. Hence, in order to toggle the Michaelis-Menten kinetic "on", both :math:`V_{\mathrm{max}}` and :math:`K_M` need to be set to be estimated simultaneously. To achieve this, one can specify a metaparameter, which is a group of parameters, e.g. ``meta_mm={V_max, K_M}``, and specify in the model space table that this metaparameter can take the values ``0`` or ``estimate``. + +The definition of metaparameters is in the PEtab Select problem YAML file, in a ``metaparameters`` section. For example, to define a metaparameter named ``reaction1`` that groups the real parameters ``k1`` and ``k2``, the following could be added to the PEtab Select problem YAML: + +.. code-block:: yaml + + metaparameters: + reaction1: + - k1 + - k2 + +The metaparameter ID can then be used as a single column in a model space file, +exactly like a normal parameter: + +.. list-table:: + :header-rows: 1 + + * - model_subspace_id + - model_subspace_petab_yaml + - reaction1 + - p3 + * - subspace1 + - petab_problem.yaml + - 0;estimate + - estimate + +Here, the value of the ``reaction1`` column is applied to all parameters in its group +(``k1`` and ``k2``). When ``reaction1`` is ``estimate``, both ``k1`` +and ``k2`` are estimated; when ``reaction1`` is ``0``, both are fixed to ``0``. + +During model selection, a metaparameter is treated as a single parameter, i.e., +toggling it on or off corresponds to a single ``forward`` or ``backward`` move, respectively, even +though it involves multiple parameters being turned on or off. + +Model selection criteria like the AIC require the actual number of estimated parameters in the model. Hence, ``reaction1`` adds two parameters (``k1`` and ``k2``) to the count when computing the criteria. + +If a parameter is in a metaparameter group, it cannot also appear as a column in the model space table. + .. _section-model-yaml: 3. Model(s) (Predecessor models / model interchange / report) @@ -150,6 +200,7 @@ Brief format description iteration: # int (optional). model_id: # str (optional). parameters: # dict[str, float | int | "estimate"] (required). Parameter ID => parameter value or "estimate". + metaparameters: # dict[str, list[str]] (optional). Metaparameter ID => Parameter IDs. predecessor_model_hash: # str (optional). - ``model_subspace_id``: Same as in the model space files. @@ -157,10 +208,11 @@ Brief format description - ``criteria``: The value of the criterion by which model selection was performed, at least. Optionally, other criterion values too. - ``model_hash``: The model hash, generated by the PEtab Select package. The format is ``[MODEL_SUBSPACE_ID]-[MODEL_SUBSPACE_INDICES_HASH]``. If all parameters are in the model are defined like ``0;estimate``, then the hash is a string of ``1`` and ``0``, for parameters that are estimated or not, respectively. - ``model_subspace_petab_yaml``: Same as in model space files. -- ``estimated_parameters``: Parameter estimates, including all estimated parameters that are not in the model selection problem; i.e., parameters that are set to be estimated in the model subspace PEtab problem but don't appear in the column header of the model space file. +- ``estimated_parameters``: Parameter estimates, including all estimated parameters that are not in the model selection problem; i.e., parameters are also included here that don't appear in the model space file, if they are set to be estimated in the model subspace PEtab problem. - ``iteration``: The iteration of model selection in which this model appeared. - ``model_id``: The model ID. -- ``parameters``: The parameter combination from the model space file that defines this model (either values or ``"estimate"``). Not the calibrated values, which are in ``estimated_parameters``! +- ``parameters``: The parameter combination from the model space file that defines this model (either values or ``"estimate"``). Not the calibrated values, which are in ``estimated_parameters``! Keys may be metaparameter IDs (see ``metaparameters``). +- ``metaparameters``: The metaparameter definitions used by this model (see :ref:`section-metaparameters`). Each metaparameter (key) is a group of parameters (value). If a metaparameter appears in ``parameters``, the value there is applied to all of the parameters within the metaparameter's group, when generating the model-specific PEtab problem. - ``predecessor_model_hash``: The hash of the model that preceded this model during the model selection process. Will be ``virtual_initial_model-`` if the model had no predecessor model. Schema diff --git a/doc/standard/model.yaml b/doc/standard/model.yaml index 304f466..e7ae1b7 100644 --- a/doc/standard/model.yaml +++ b/doc/standard/model.yaml @@ -58,6 +58,16 @@ properties: type: string title: Parameters type: object + metaparameters: + anyOf: + - additionalProperties: + items: + type: string + type: array + type: object + - type: 'null' + default: null + title: Metaparameters predecessor_model_hash: $ref: '#/$defs/ModelHash' default: null diff --git a/doc/standard/models.yaml b/doc/standard/models.yaml index ba04503..8f39238 100644 --- a/doc/standard/models.yaml +++ b/doc/standard/models.yaml @@ -57,6 +57,16 @@ $defs: type: string title: Parameters type: object + metaparameters: + anyOf: + - additionalProperties: + items: + type: string + type: array + type: object + - type: 'null' + default: null + title: Metaparameters predecessor_model_hash: $ref: '#/$defs/ModelHash' default: null diff --git a/doc/standard/problem.yaml b/doc/standard/problem.yaml index ceba7a9..7986b35 100644 --- a/doc/standard/problem.yaml +++ b/doc/standard/problem.yaml @@ -17,6 +17,13 @@ properties: candidate_space_arguments: title: Candidate Space Arguments type: object + metaparameters: + additionalProperties: + items: + type: string + type: array + title: Metaparameters + type: object required: - criterion - method diff --git a/petab_select/candidate_space.py b/petab_select/candidate_space.py index f3ed08a..6f44f27 100644 --- a/petab_select/candidate_space.py +++ b/petab_select/candidate_space.py @@ -594,15 +594,35 @@ def distances_in_estimated_parameters( "This check is based on the PEtab YAML file location." ) - # All parameters from the PEtab problem are used in the computation. + # Distances are computed in terms of estimated parameters. Each + # metaparameter is treated as a single parameter here, even though it + # involves multiple parameters. This is because the distance is on the + # level of model hypotheses, rather than parameters directly. + estimated1 = set(model1.get_estimated_parameter_ids()) + if model0.hash == VIRTUAL_INITIAL_MODEL_HASH: - parameter_ids = list( - get_petab_parameters(model1._model_subspace_petab_problem) - ) if self.method == Method.FORWARD: - parameters0 = np.array([0 for _ in parameter_ids]) + # The virtual initial model estimates no parameters. + estimated0 = set() elif self.method == Method.BACKWARD: - parameters0 = np.array([ESTIMATE for _ in parameter_ids]) + # The virtual initial model estimates all parameters. + metaparameter_members = { + member + for members in (model1.metaparameters or {}).values() + for member in members + } + # Includes metaparameters + estimated0 = set(model1.parameters) + # Hence, we exclude parameters here that are already included + # via a metaparameter + for parameter_id in get_petab_parameters( + model1._model_subspace_petab_problem + ): + if ( + parameter_id not in model1.parameters + and parameter_id not in metaparameter_members + ): + estimated0.add(parameter_id) else: raise NotImplementedError( "Distances for the virtual initial model have not yet been " @@ -610,27 +630,13 @@ def distances_in_estimated_parameters( "developers." ) else: - parameter_ids = list( - get_petab_parameters(model0._model_subspace_petab_problem) - ) - parameters0 = np.array( - model0.get_parameter_values(parameter_ids=parameter_ids) - ) - parameters1 = np.array( - model1.get_parameter_values(parameter_ids=parameter_ids) - ) - - # TODO change to some numpy elementwise operation - estimated0 = np.array([p == ESTIMATE for p in parameters0]).astype(int) - estimated1 = np.array([p == ESTIMATE for p in parameters1]).astype(int) + estimated0 = set(model0.get_estimated_parameter_ids()) - difference = estimated1 - estimated0 - - # Changes to(/from) estimated from(/to) not estimated - l1 = np.abs(difference).sum() + # Parameters whose estimated status differs between the two models. + l1 = len(estimated1.symmetric_difference(estimated0)) # Change in the number of estimated parameters. - size = np.sum(difference) + size = len(estimated1) - len(estimated0) # TODO constants? e.g. Distance.L1 and Distance.Size distances = { @@ -1322,6 +1328,8 @@ def get_most_distant( Next we check if this model is contained in any subspace. If so we choose it. If not we choose the model in a subspace that has least distance to this complement model. + + Also metaparameters not yet supported here. """ most_distance = 0 most_distant_indices = [] diff --git a/petab_select/constants.py b/petab_select/constants.py index 7ffb2f5..543f05e 100644 --- a/petab_select/constants.py +++ b/petab_select/constants.py @@ -72,6 +72,7 @@ class Criterion(str, Enum): # Problem MODEL_SPACE_FILES = "model_space_files" +METAPARAMETERS = "metaparameters" PROBLEM = "problem" VERSION = "version" diff --git a/petab_select/model.py b/petab_select/model.py index 165e4be..2ce31e6 100644 --- a/petab_select/model.py +++ b/petab_select/model.py @@ -311,6 +311,8 @@ class ModelBase(VirtualModelBase): For example, fixes parameters to certain values, or sets them to be estimated. """ + metaparameters: dict[str, list[str]] | None = Field(default=None) + """Metaparameter definitions used by this model.""" predecessor_model_hash: ModelHash = Field(default=None) """The predecessor model hash.""" @@ -557,15 +559,26 @@ def to_petab( """ petab_problem = petab.Problem.from_yaml(self.model_subspace_petab_yaml) + # Get all parameter values that should be implemented in the PEtab parameter + # table. + metaparameters = self.metaparameters or {} + parameters = { + petab_parameter_id: parameter_value + for parameter_id, parameter_value in self.parameters.items() + for petab_parameter_id in metaparameters.get( + parameter_id, [parameter_id] + ) + } + if set_estimated_parameters is None and self.estimated_parameters: set_estimated_parameters = True if set_estimated_parameters: - # Check that estimates are provided for all estimated parameters + # Check that estimates are provided for all estimated parameters. required_estimates = set() for parameter_id in petab_problem.x_ids: - if parameter_id in self.parameters: - if self.parameters[parameter_id] == ESTIMATE: + if parameter_id in parameters: + if parameters[parameter_id] == ESTIMATE: required_estimates.add(parameter_id) elif parameter_id in petab_problem.x_free_ids: required_estimates.add(parameter_id) @@ -580,7 +593,7 @@ def to_petab( f"`{missing_estimates}`." ) - for parameter_id, parameter_value in self.parameters.items(): + for parameter_id, parameter_value in parameters.items(): # If the parameter is to be estimated. if parameter_value == ESTIMATE: petab_problem.parameter_df.loc[parameter_id, ESTIMATE] = 1 @@ -642,12 +655,25 @@ def get_estimated_parameter_ids(self, full: bool = True) -> list[str]: full: Whether to provide all IDs, including additional parameters that are not part of the model selection problem but estimated. + + Returns: + The estimated parameter IDs. Note that metaparameters are returned, + not the parameters in their group. """ estimated_parameter_ids = [] if full: - estimated_parameter_ids = ( - self._model_subspace_petab_problem.x_free_ids - ) + # Exclude PEtab parameters that are represented by a metaparameter; + # they are reported via their metaparameter ID below instead. + metaparameter_members = { + member + for members in (self.metaparameters or {}).values() + for member in members + } + estimated_parameter_ids = [ + parameter_id + for parameter_id in self._model_subspace_petab_problem.x_free_ids + if parameter_id not in metaparameter_members + ] # Add additional estimated parameters, and collect fixed parameters, # in this model's parameterization. diff --git a/petab_select/model_space.py b/petab_select/model_space.py index 83633f7..e29b6c7 100644 --- a/petab_select/model_space.py +++ b/petab_select/model_space.py @@ -47,6 +47,7 @@ def __init__( def load( data: TYPE_PATH | pd.DataFrame | list[TYPE_PATH | pd.DataFrame], root_path: TYPE_PATH = None, + metaparameters: dict[str, list[str]] | None = None, ) -> ModelSpace: """Load a model space from dataframe(s) or file(s). @@ -57,6 +58,9 @@ def load( Any paths in dataframe will be resolved relative to this path. Paths in TSV files will be resolved relative to the directory of the TSV file. + metaparameters: + Keys are metaparameter IDs, values are groups of parameter IDs + that the metaparameter represents. Returns: The model space. @@ -80,6 +84,7 @@ def load( ModelSubspace.from_definition( definition=definition, root_path=root_path, + metaparameters=metaparameters, ) ) model_space = ModelSpace(model_subspaces=model_subspaces) diff --git a/petab_select/model_subspace.py b/petab_select/model_subspace.py index 6b0b3cb..e5088e7 100644 --- a/petab_select/model_subspace.py +++ b/petab_select/model_subspace.py @@ -47,6 +47,8 @@ class ModelSubspace: exclusions: Hashes of models that have been previously submitted to a candidate space for consideration (:meth:`CandidateSpace.consider`). + metaparameters: + The metaparameter mappings that exist in the subspace. """ def __init__( @@ -55,10 +57,17 @@ def __init__( petab_yaml: str | Path, parameters: TYPE_PARAMETER_OPTIONS_DICT, exclusions: list[Any] | None | None = None, + metaparameters: dict[str, list[str]] | None = None, ): self.model_subspace_id = model_subspace_id self.petab_yaml = Path(petab_yaml) self.parameters = parameters + # Store metaparameters that exist in this subspace + self.metaparameters = { + metaparameter_id: member_ids + for metaparameter_id, member_ids in (metaparameters or {}).items() + if metaparameter_id in self.parameters + } self.exclusions = set() if exclusions is not None: @@ -75,6 +84,37 @@ def __init__( "(e.g. in the model space table)." ) + # Validate metaparameters. + for metaparameter_id, member_ids in self.metaparameters.items(): + if not member_ids: + raise ValueError( + f"The metaparameter does not represent any parameters: `{metaparameter_id}`" + ) + missing_ids = [ + member_id + for member_id in member_ids + if member_id not in self.petab_problem.x_ids + ] + if missing_ids: + raise ValueError( + f"The metaparameter `{metaparameter_id}` represents the " + f"following parameters that are not in the PEtab problem " + f"`{self.petab_yaml}`: `{missing_ids}`." + ) + member_ids_as_columns = [ + member_id + for member_id in member_ids + if member_id in self.parameters + ] + if member_ids_as_columns: + raise ValueError( + f"The metaparameter `{metaparameter_id}` represents " + "parameters that also appear as their own columns in the " + f"model space: {member_ids_as_columns}. A parameter that " + "belongs to a metaparameter must only be controlled via " + "the metaparameter column." + ) + def check_compatibility_stepwise_method( self, candidate_space: CandidateSpace, @@ -718,6 +758,7 @@ def reset( def from_definition( definition: dict[str, str] | pd.Series, root_path: TYPE_PATH = None, + metaparameters: dict[str, list[str]] | None = None, ) -> "ModelSubspace": """Create a :class:`ModelSubspace` from a definition. @@ -728,6 +769,8 @@ def from_definition( values. root_path: Any paths will be resolved relative to this path. + metaparameters: + Metaparameter definitions. Returns: The model subspace. @@ -753,6 +796,7 @@ def from_definition( model_subspace_id=model_subspace_id, petab_yaml=petab_yaml, parameters=parameters, + metaparameters=metaparameters, ) def to_definition(self, root_path: TYPE_PATH | None = None) -> pd.Series: @@ -802,6 +846,7 @@ def indices_to_model(self, indices: list[int]) -> Model | None: model_subspace_indices=indices, model_subspace_petab_yaml=self.petab_yaml, parameters=self.indices_to_parameters(indices), + metaparameters=(self.metaparameters or None), _model_subspace_petab_problem=self.petab_problem, ) if self.excluded(model): @@ -879,11 +924,25 @@ def parameters_to_model( def parameters_all(self) -> TYPE_PARAMETER_DICT: """Get all parameters, including those only in the PEtab problem. + If there are metaparameters, they are returned directly, not their represented parameters. + Parameter values in the PEtab problem are overwritten by the model subspace values. """ + metaparameter_members = { + member_id + for member_ids in self.metaparameters.values() + for member_id in member_ids + } + petab_parameters = { + parameter_id: parameter_value + for parameter_id, parameter_value in get_petab_parameters( + self.petab_problem, as_lists=True + ).items() + if parameter_id not in metaparameter_members + } return { - **get_petab_parameters(self.petab_problem, as_lists=True), + **petab_parameters, **self.parameters, } @@ -968,11 +1027,20 @@ def must_estimate(self) -> list[str]: @property def must_estimate_all(self) -> list[str]: - """All parameters that must be estimated in this subspace.""" + """All parameters that must be estimated in this subspace. + + If there are metaparameters, they are returned directly, not their represented parameters. + """ + metaparameter_members = { + member_id + for member_ids in self.metaparameters.values() + for member_id in member_ids + } must_estimate_petab = [ parameter_id for parameter_id in self.petab_problem.x_free_ids if parameter_id not in self.parameters + and parameter_id not in metaparameter_members ] return [*must_estimate_petab, *self.must_estimate] diff --git a/petab_select/problem.py b/petab_select/problem.py index 4ba467c..f07e5f7 100644 --- a/petab_select/problem.py +++ b/petab_select/problem.py @@ -90,6 +90,12 @@ class Problem(BaseModel): These are forwarded to the candidate space constructor. """ + metaparameters: dict[str, list[str]] = Field(default_factory=dict) + """Metaparameters. + + Keys are metaparameter IDs, and values are the groups of parameter IDs that + the metaparameters represent. + """ _compare: Callable[[Model, Model], bool] = PrivateAttr(default=None) """The method by which models are compared.""" @@ -119,7 +125,8 @@ def _check_input( [ root_path / model_space_file for model_space_file in problem.model_space_files - ] + ], + metaparameters=problem.metaparameters, ) if PREDECESSOR_MODEL in problem.candidate_space_arguments: diff --git a/test/metaparameters/__init__.py b/test/metaparameters/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/test/metaparameters/test_metaparameters.py b/test/metaparameters/test_metaparameters.py new file mode 100644 index 0000000..ae9c83d --- /dev/null +++ b/test/metaparameters/test_metaparameters.py @@ -0,0 +1,226 @@ +"""Tests for metaparameters.""" + +from pathlib import Path + +import pytest + +from petab_select import Problem +from petab_select.constants import ESTIMATE, Criterion +from petab_select.model import Model + +PETAB_YAML = ( + Path(__file__).parent.parent + / "test_data" + / "ode_timeseries" + / "petab_problem.yaml" +) + +METAPARAMETERS = {"mp1": ["k1", "k2"]} + + +def write_problem(directory: Path, model_space_row: str) -> Problem: + """Write and load a PEtab Select problem that uses a metaparameter.""" + model_space_tsv = ( + "model_subspace_id\tmodel_subspace_petab_yaml\tmp1\tk3\n" + f"{model_space_row}\n" + ) + (directory / "model_space.tsv").write_text(model_space_tsv) + problem_yaml = ( + "format_version: 1.0.0\n" + "criterion: AIC\n" + "method: forward\n" + "model_space_files:\n" + "- model_space.tsv\n" + "metaparameters:\n" + " mp1:\n" + " - k1\n" + " - k2\n" + ) + problem_path = directory / "petab_select_problem.yaml" + problem_path.write_text(problem_yaml) + return Problem.from_yaml(problem_path) + + +@pytest.fixture +def problem(tmp_path) -> Problem: + return write_problem(tmp_path, f"M\t{PETAB_YAML}\t0;estimate\t0;estimate") + + +def test_problem_metaparameters(problem): + """Metaparameters are parsed from the problem YAML.""" + assert problem.metaparameters == METAPARAMETERS + + +def test_subspace_metaparameters(problem): + """The model subspace represents metaparameters. + + i.e. the parameters in the metaparameter group should not exist + in the subspace explicitly, only via the metaparameter. + """ + subspace = problem.model_space.model_subspaces["M"] + assert subspace.metaparameters == METAPARAMETERS + + # The metaparameter, and not its parameters, exist + assert "mp1" in subspace.parameters_all + assert "k1" not in subspace.parameters_all + assert "k2" not in subspace.parameters_all + + # The metaparameter can be "estimated" + assert "mp1" in subspace.can_estimate + # `sigma_x2` is estimated in the PEtab problem and is not grouped. + assert subspace.must_estimate_all == ["sigma_x2"] + + +def test_model_metaparameter_expansion(problem): + """Metaparameters expand to their member PEtab parameters.""" + subspace = problem.model_space.model_subspaces["M"] + model = subspace.parameters_to_model({"mp1": ESTIMATE, "k3": 0}) + + # Compact representation (as in the model space file). + assert model.parameters == {"mp1": ESTIMATE, "k3": 0} + # The generated PEtab correctly assigns the metaparameter value to its parameters. + parameter_df = model.to_petab()["petab_problem"].parameter_df + assert parameter_df.loc["k1", ESTIMATE] == 1 + assert parameter_df.loc["k2", ESTIMATE] == 1 + assert parameter_df.loc["k3", ESTIMATE] == 0 + + +def test_estimated_parameter_ids_are_compact(problem): + """A metaparameter is a single estimated parameter.""" + subspace = problem.model_space.model_subspaces["M"] + model = subspace.parameters_to_model({"mp1": ESTIMATE, "k3": 0}) + # `mp1` (not `k1`/`k2`) is reported, plus the always-estimated `sigma_x2`. + assert set(model.get_estimated_parameter_ids()) == {"mp1", "sigma_x2"} + + +def test_n_estimated_counts_members(problem): + """The number of estimated parameters counts metaparameter members.""" + subspace = problem.model_space.model_subspaces["M"] + + model_on = subspace.parameters_to_model({"mp1": ESTIMATE, "k3": 0}) + petab_problem_on = model_on.to_petab()["petab_problem"] + # Estimating `mp1` estimates both `k1` and `k2` (plus `sigma_x2`). + assert set(petab_problem_on.x_free_ids) == {"k1", "k2", "sigma_x2"} + + problem.model_space.reset_exclusions() + model_off = subspace.parameters_to_model({"mp1": 0, "k3": 0}) + petab_problem_off = model_off.to_petab()["petab_problem"] + # Fixing `mp1` fixes both `k1` and `k2` (to ``0``). + assert set(petab_problem_off.x_free_ids) == {"sigma_x2"} + assert petab_problem_off.parameter_df.loc["k1", ESTIMATE] == 0 + assert petab_problem_off.parameter_df.loc["k2", ESTIMATE] == 0 + assert petab_problem_off.parameter_df.loc["k1", "nominalValue"] == 0 + assert petab_problem_off.parameter_df.loc["k2", "nominalValue"] == 0 + + +def test_aic_uses_real_parameter_count(problem): + """The AIC uses the real (expanded) number of estimated parameters.""" + subspace = problem.model_space.model_subspaces["M"] + model = subspace.parameters_to_model({"mp1": ESTIMATE, "k3": 0}) + model.set_criterion(Criterion.NLLH, 10.0) + # AIC = 2 * (n_estimated + NLLH) = 2 * (3 + 10) = 26. + assert model.get_criterion(Criterion.AIC) == 26.0 + + +def test_forward_move_is_single_step(problem): + """Estimating a metaparameter is a single forward move.""" + subspace = problem.model_space.model_subspaces["M"] + predecessor = subspace.parameters_to_model({"mp1": 0, "k3": 0}) + + candidate_space = problem.new_candidate_space( + predecessor_model=predecessor + ) + problem.model_space.reset_exclusions() + problem.model_space.search(candidate_space) + + candidate_parameters = [m.parameters for m in candidate_space.models] + # Both single-parameter moves are equally minimal forward steps: + # estimating the metaparameter, or estimating `k3`. + assert {"mp1": ESTIMATE, "k3": 0} in candidate_parameters + assert {"mp1": 0, "k3": ESTIMATE} in candidate_parameters + assert len(candidate_parameters) == 2 + + +def test_backward_move_is_single_step(problem): + """Fixing a metaparameter is a single backward move.""" + subspace = problem.model_space.model_subspaces["M"] + predecessor = subspace.parameters_to_model( + {"mp1": ESTIMATE, "k3": ESTIMATE} + ) + + candidate_space = problem.new_candidate_space( + method="backward", + predecessor_model=predecessor, + ) + problem.model_space.reset_exclusions() + problem.model_space.search(candidate_space) + + candidate_parameters = [m.parameters for m in candidate_space.models] + assert {"mp1": 0, "k3": ESTIMATE} in candidate_parameters + assert {"mp1": ESTIMATE, "k3": 0} in candidate_parameters + assert len(candidate_parameters) == 2 + + +def test_yaml_round_trip(problem, tmp_path): + """A model with a metaparameter survives a YAML round trip.""" + subspace = problem.model_space.model_subspaces["M"] + model = subspace.parameters_to_model({"mp1": ESTIMATE, "k3": 0}) + + model_yaml = tmp_path / "model.yaml" + model.to_yaml(model_yaml) + reloaded = Model.from_yaml(model_yaml) + + assert reloaded.metaparameters == METAPARAMETERS + assert reloaded.parameters == {"mp1": ESTIMATE, "k3": 0} + # The reloaded model still expands ``mp1`` into ``k1`` and ``k2``. + petab_problem = reloaded.to_petab()["petab_problem"] + assert set(petab_problem.x_free_ids) == {"k1", "k2", "sigma_x2"} + assert len(petab_problem.x_free_indices) == 3 + + +def test_invalid_metaparameter_member(tmp_path): + """A metaparameter referencing an unknown PEtab parameter is rejected.""" + model_space_tsv = ( + "model_subspace_id\tmodel_subspace_petab_yaml\tmp1\n" + f"M\t{PETAB_YAML}\t0;estimate\n" + ) + (tmp_path / "model_space.tsv").write_text(model_space_tsv) + problem_yaml = ( + "format_version: 1.0.0\n" + "criterion: AIC\n" + "method: forward\n" + "model_space_files:\n" + "- model_space.tsv\n" + "metaparameters:\n" + " mp1:\n" + " - k1\n" + " - does_not_exist\n" + ) + problem_path = tmp_path / "petab_select_problem.yaml" + problem_path.write_text(problem_yaml) + with pytest.raises(ValueError, match="not in the PEtab problem"): + Problem.from_yaml(problem_path) + + +def test_metaparameter_member_as_column_is_rejected(tmp_path): + """A metaparameter member must not also be its own model space column.""" + model_space_tsv = ( + "model_subspace_id\tmodel_subspace_petab_yaml\tmp1\tk1\n" + f"M\t{PETAB_YAML}\t0;estimate\t0;estimate\n" + ) + (tmp_path / "model_space.tsv").write_text(model_space_tsv) + problem_yaml = ( + "format_version: 1.0.0\n" + "criterion: AIC\n" + "method: forward\n" + "model_space_files:\n" + "- model_space.tsv\n" + "metaparameters:\n" + " mp1:\n" + " - k1\n" + " - k2\n" + ) + problem_path = tmp_path / "petab_select_problem.yaml" + problem_path.write_text(problem_yaml) + with pytest.raises(ValueError, match="also appear as their own columns"): + Problem.from_yaml(problem_path) diff --git a/test/problem/expected_output/petab_select_problem.yaml b/test/problem/expected_output/petab_select_problem.yaml index 360def4..38b7a91 100644 --- a/test/problem/expected_output/petab_select_problem.yaml +++ b/test/problem/expected_output/petab_select_problem.yaml @@ -4,3 +4,4 @@ method: forward model_space_files: - model_space.tsv candidate_space_arguments: {} +metaparameters: {}