Consistency Rules not working on Dataplex

[pbaquedanov]

Hello!

I'm using Data Quality Tasks on Dataplex and i'm using a rule of consistency (using a sql case when), and i get an error and looking for a example of rules of this dimension (consistency) i can't find an example of it.

This is my .yaml file:

rules:
CONSISTENCY_MINSS:
rule_type: CUSTOM_SQL_EXPR
dimension: consistency
params:
custom_sql_arguments:
- ref_data_dataset
- ref_data_table_id
custom_sql_expr: |-
case
when ($column IN (0,10,100)) or ($column is null) then true
else false end
from data a
inner join $ref_data_project.$ref_data_dataset.$ref_data_table_id b
on a.loc = b.loc
rule_dimensions:

• consistency
  row_filters:
  _PARTITIONTIME:
  filter_sql_expr: |-
  DATE(_PARTITIONTIME) = '2024-03-03'
  rule_bindings:
  MY-RULE-BINDING-CONSISTENCY-MINSS:
  entity_uri: bigquery://projects/my-gcp-project-ENV/locations/US/datasets/my_gcp_dataset_ENV/tables/my_bigquery_tablename_ENV
  column_id: column_name
  row_filter_id: _PARTITIONTIME
  rule_ids:
  • CONSISTENCY_MINSS:
    • ref_data_project: df-datalake-transformed-ENV
    • ref_data_dataset: my_gcp_dataset_ENV
    • ref_data_table_id: my_gcp_table_ENV
      metadata:
      manual_column_id: minss

This is the error in the output on dataproc:

metadata_registry_defaults:
dataplex:
projects:
locations:
lakes:
zones:

2024-03-12 18:35:10 gdpic-srvls-batch-9e830eba-2731-47cc-beeb-3fdf1ea8b20f-m clouddq[80] ERROR Failed to resolve Rule Binding ID 'MY-RULE-BINDING-CONSISTENCY-MINSS' with error:
Failed to resolve rule_id 'CONSISTENCY_MINSS' in rule_binding_id 'MY-RULE-BINDING-CONSISTENCY-MINSS' with error:
'list' object has no attribute 'get'
Traceback (most recent call last):
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/dq_rule_binding.py", line 213, in resolve_rule_sql_expr
rule.resolve_sql_expr()
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/dq_rule.py", line 122, in resolve_sql_expr
self.rule_sql_expr = self.rule_type.to_sql(self.params).safe_substitute()
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/rule_type.py", line 187, in to_sql
return to_sql_custom_sql_expr(params)
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/rule_type.py", line 96, in to_sql_custom_sql_expr
if params.get("rule_binding_arguments", {}).get(argument, None) is None:
AttributeError: 'list' object has no attribute 'get'

[thinhha]

Hi @pbaquedanov, the input arguments for the rule_binding should be an object not a list. All you have to do is remove the hyphen before the arguments.

Please see example rules here
https://github.com/GoogleCloudPlatform/cloud-data-quality/blob/main/docs/examples/advanced_rules/USERMANUAL.md