DAG scopes

[NicoLaval]

Hi @noahboerger,

We have an issue regarding this kind of script with the DAG feature.

If we don't disable the $vtl.engine.use_dag property, the test fails because of scope issue for LOOP variable.

What do you think about?

[noahboerger]

Thank you, we will take care of it.
It does not seem to be an easy case, as we have to distinguish here if a VarId is having an external scope (e.g. Scalar coming from outside) or if it is an component from the dataset.

[noahboerger]

Before implementing it, I think we need to exactly clarify the expected behavior. Let us use the example from the unit test:

TEMP_RACINE := MULTIMODE [keep interrogationId, FOO];
RACINE := union(TEMP_RACINE, TEMP_RACINE) ;
TEMP_LOOP := MULTIMODE [keep interrogationId, LOOP, LOOP.FOO1]
                       [filter LOOP <> "\"]
                       [rename LOOP.FOO1 to FOO1];
LOOP <- union(TEMP_LOOP, TEMP_LOOP);

The expected behavior is not fully clear for us, because of the following reasons:

• The VTL Syntax does not allow varIds in calc/filter expressions. The reason why LOOP is parsed into a varId here (and added as dependency for the DAG) is because of the adjusted G4 grammar in Trevas, otherwise only components of the dataset would be allowed in such expressions.
  • Nevertheless we do think that it is a real need, to allow scalars in such expression (see also Usage of scalars #355), but are not sure if this is really covered by the current VTL standard. Can you confirm this?
  • Therefore out of my point of view the scoping rule, we assume here, is not defined in the VTL manuals. Am I missing something or do you have a reference therefore?
• Is the DAG sth that should be fully resolvable by a script or is it also dependent on the data structures? Should it rely on scoping rules?

Considering e.g.

ds_calc := ds[filter Me_1 = scalar_value];
scalar_value := max(ds_calc)#Me_1; // according to https://github.com/sdmx-twg/vtl/pull/602/files to resolve a scalar from a dataset

--> The DAG would be different for the following cases (if we allow such scoping rules):

• scalar_values is existent in ds: The script would be valid.
• scalar_value is not existent in ds: This would lead to an error because of the cyclic dependency

Should we adress this at TF level?

Of course we do not want to intervene you developments until we have a clear solution herefore. Therefore we excluded the creation of dependencies on statements where in the original VTL syntax only components would be allowed (e.g. calc/filter statements) - as Trevas currently does not support such expressions as pointed out via #355.

See also #461.

[hadrienk]

The components branches are removed from the grammar in our implementation. The reasoning is that detecting if we are in or out of a dataset context is trivial to do in the implementation rather than using a duplicate expression branch.

To me this sounds like an ambiguity that should be detected and trigger a failure. Having the same variable name in a dataset in the context of a VTL expression that does not have a fixed ordering is bound to fail.

[NicoLaval]

Really interesting example @noahboerger:

Inspired by closures, I would say:

// Case 1
ds_calc := ds[filter Me_1 = scalar_value]; // if scalar_value is a component of ds, scalar_value has to be this component
scalar_value := max(ds_calc)#Me_1;

// Case 2
ds_calc := ds[filter Me_1 = scalar_value]; // if scalar_value is NOT a component of ds, scalar_value has to be the root scalar
scalar_value := max(ds_calc)#Me_1;

But, if you don't have dataset metadata when you run the DAG algorithme, you are not able to apply these rules :/

@noahboerger I agree to have a more global issue on the TWG repo about this kind of use cases.

@hadrienk, same in our implementation? case 2 is not supported I think.

[noahboerger]

Thank you for your feedback, I have addressed it as proposed via sdmx-twg/vtl#653 in the TF repo.