Merge branch 'master' into feat/stats-for-dbt

dlawin · web-flow · commit ffa03c60d920 · 2023-08-11T14:19:54.000-06:00
diff --git a/README.md b/README.md
@@ -1,25 +1,65 @@
-<p align="left">
+<p align="center">
     <a href="https://datafold.com/"><img alt="Datafold" src="https://user-images.githubusercontent.com/1799931/196497110-d3de1113-a97f-4322-b531-026d859b867a.png" width="30%" /></a>
 </p>
 
-<h1 align="left">
-data-diff: compare datasets fast, within or across SQL databases
-</h1>
+<h2 align="center">
+data-diff: Compare datasets fast, within or across SQL databases
 
+![data-diff-logo](docs/data-diff-logo.png)
+</h2>
 <br>
 
+# Use Cases
+
+## Data Migration & Replication Testing
+Compare source to target and check for discrepancies when moving data between systems:
+- Migrating to a new data warehouse (e.g., Oracle > Snowflake)
+- Converting SQL to a new transformation framework (e.g., stored procedures > dbt)
+- Continuously replicating data from an OLTP DB to OLAP DWH (e.g., MySQL > Redshift)
+
+
+## Data Development Testing
+Test SQL code and preview changes by comparing development/staging environment data to production:
+1. Make a change to some SQL code
+2. Run the SQL code to create a new dataset
+3. Compare the dataset with its production version or another iteration
+
+  <p align="left">
+  <img alt="dbt" src="https://seeklogo.com/images/D/dbt-logo-E4B0ED72A2-seeklogo.com.png" width="10%" />
+  </p>
+  
+<details>
+<summary> data-diff integrates with dbt Core to seamlessly compare local development to production datasets
+
+ </summary>
+
+![data-development-testing](docs/development_testing.png)
+
+</details>
+
+> [dbt Cloud users should check out Datafold's out-of-the-box deployment testing integration](https://www.datafold.com/data-deployment-testing)
+
+:eyes: **Watch [4-min demo video](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
+
+**[Get started with data-diff & dbt](https://docs.datafold.com/development_testing/open_source)**
+
+Also available in a [VS Code Extension](https://marketplace.visualstudio.com/items?itemName=Datafold.datafold-vscode)
+
+Reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) for advice and support
+
+
 # How it works
 
 When comparing the data, `data-diff` utilizes the resources of the underlying databases as much as possible. It has two primary modes of comparison:
 
-## joindiff
+## `joindiff`
 - Recommended for comparing data within the same database
 - Uses the outer join operation to diff the rows as efficiently as possible within the same database
 - Fully relies on the underlying database engine for computation
 - Requires both datasets to be queryable with a single SQL query
 - Time complexity approximates JOIN operation and is largely independent of the number of differences in the dataset
   
-## hashdiff
+## `hashdiff`
 - Recommended for comparing datasets across different databases
 - Can also be helpful in diffing very large tables with few expected differences within the same database
 - Employs a divide-and-conquer algorithm based on hashing and binary search
@@ -52,59 +92,32 @@ data-diff \
 Check out [documentation](https://docs.datafold.com/reference/open_source/cli) for the full command reference.
 
 
-# Use cases
-
-## Data Migration & Replication Testing
-Compare source to target and check for discrepancies when moving data between systems:
-- Migrating to a new data warehouse (e.g., Oracle > Snowflake)
-- Converting SQL to a new transformation framework (e.g., stored procedures > dbt)
-- Continuously replicating data from an OLTP DB to OLAP DWH (e.g., MySQL > Redshift)
-
-
-## Data Development Testing
-Test SQL code and preview changes by comparing development/staging environment data to production:
-1. Make a change to some SQL code
-2. Run the SQL code to create a new dataset
-3. Compare the dataset with its production version or another iteration
-
-  <p align="left">
-  <img alt="dbt" src="https://seeklogo.com/images/D/dbt-logo-E4B0ED72A2-seeklogo.com.png" width="10%" />
-  </p>
-  
-`data-diff` integrates with dbt Core and dbt Cloud to seamlessly compare local development to production datasets. 
-
-:eyes: **Watch [4-min demo video](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
-
-**[Get started with data-diff & dbt](https://docs.datafold.com/development_testing/open_source)**
-
-Reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) for advice and support
-
 # Supported databases
 
 
 | Database      | Status | Connection string |
 |---------------|-------------------------------------------------------------------------------------------------------------------------------------|--------|
-| PostgreSQL >=10 |  💚    | `postgresql://<user>:<password>@<host>:5432/<database>`                                                                        |
-| MySQL         |  💚    | `mysql://<user>:<password>@<hostname>:5432/<database>`                                                                              |
-| Snowflake     |  💚    | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
-| BigQuery      |  💚    | `bigquery://<project>/<dataset>`                                                                                                    |
-| Redshift      |  💚    | `redshift://<username>:<password>@<hostname>:5439/<database>`                                                                       |
-| Oracle        |  💛    | `oracle://<username>:<password>@<hostname>/database`                                                                                |
-| Presto        |  💛    | `presto://<username>:<password>@<hostname>:8080/<database>`                                                                         |
-| Databricks    |  💛    | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>`                                                      |
-| Trino         |  💛    | `trino://<username>:<password>@<hostname>:8080/<database>`                                                                          |
-| Clickhouse    |  💛    | `clickhouse://<username>:<password>@<hostname>:9000/<database>`                                                                     |
-| Vertica       |  💛    | `vertica://<username>:<password>@<hostname>:5433/<database>`                                                                        |
-| DuckDB        |  💛    |                                                                                                                                     |
+| PostgreSQL >=10 |  🟢   | `postgresql://<user>:<password>@<host>:5432/<database>`                                                           |
+| MySQL         |  🟢     | `mysql://<user>:<password>@<hostname>:5432/<database>`                                                                              |
+| Snowflake     |  🟢     | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
+| BigQuery      |  🟢     | `bigquery://<project>/<dataset>`                                                                                                    |
+| Redshift      |  🟢     | `redshift://<username>:<password>@<hostname>:5439/<database>`                                                                       |
+| Oracle        |  🟡   | `oracle://<username>:<password>@<hostname>/database`                                                                                |
+| Presto        |  🟡   | `presto://<username>:<password>@<hostname>:8080/<database>`                                                                         |
+| Databricks    |  🟡   | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>`                                                      |
+| Trino         |  🟡   | `trino://<username>:<password>@<hostname>:8080/<database>`                                                                          |
+| Clickhouse    |  🟡   | `clickhouse://<username>:<password>@<hostname>:9000/<database>`                                                                     |
+| Vertica       |  🟡   | `vertica://<username>:<password>@<hostname>:5433/<database>`                                                                        |
+| DuckDB        |  🟡   |                                                                                                                                     |
 | ElasticSearch |  📝    |                                                                                                                                     |
 | Planetscale   |  📝    |                                                                                                                                     |
 | Pinot         |  📝    |                                                                                                                                     |
 | Druid         |  📝    |                                                                                                                                     |
 | Kafka         |  📝    |                                                                                                                                     |
 | SQLite        |  📝    |                                                                                                                                     |
 
-* 💚: Implemented and thoroughly tested.
-* 💛: Implemented, but not thoroughly tested yet.
+* 🟢: Implemented and thoroughly tested.
+* 🟡: Implemented, but not thoroughly tested yet.
 * ⏳: Implementation in progress.
 * 📝: Implementation planned. Contributions welcome.
 
diff --git a/data_diff/config.py b/data_diff/config.py
@@ -4,6 +4,12 @@
 import toml
 
 
+_ARRAY_FIELDS = (
+    "key_columns",
+    "columns",
+)
+
+
 class ConfigParseError(Exception):
     pass
 
@@ -38,6 +44,11 @@ def _apply_config(config: Dict[str, Any], run_name: str, kw: Dict[str, Any]):
         for index in "12":
             run_args[index] = {attr: kw.pop(f"{attr}{index}") for attr in ("database", "table")}
 
+    # Make sure array fields are decoded as list, since array fields in toml are decoded as list, but TableSegment object requires tuple type.
+    for field in _ARRAY_FIELDS:
+        if isinstance(run_args.get(field), list):
+            run_args[field] = tuple(run_args[field])
+
     # Process databases + tables
     for index in "12":
         try:
diff --git a/data_diff/sqeleton/databases/postgresql.py b/data_diff/sqeleton/databases/postgresql.py
@@ -1,3 +1,4 @@
+from typing import List
 from ..abcs.database_types import (
     DbPath,
     JSON,
@@ -92,6 +93,10 @@ def quote(self, s: str):
     def to_string(self, s: str):
         return f"{s}::varchar"
 
+    def concat(self, items: List[str]) -> str:
+        joined_exprs = " || ".join(items)
+        return f"({joined_exprs})"
+
     def _convert_db_precision_to_digits(self, p: int) -> int:
         # Subtracting 2 due to wierd precision issues in PostgreSQL
         return super()._convert_db_precision_to_digits(p) - 2
diff --git a/data_diff/tracking.py b/data_diff/tracking.py
@@ -42,7 +42,7 @@ def bool_ask_for_email() -> bool:
     """
     Checks the .datadiff.toml profile file for the asked_for_email key
 
-    Returns False immediately if --no-tracking
+    Returns False immediately if --no-tracking or not in an interactive terminal
 
     If found, return False (already asked for email)
 
@@ -51,7 +51,8 @@ def bool_ask_for_email() -> bool:
     Returns:
         bool: decision on whether to prompt the user for their email
     """
-    if g_tracking_enabled:
+    console = get_console()
+    if g_tracking_enabled and console.is_interactive:
         profile = _load_profile()
 
         if "asked_for_email" not in profile:
diff --git a/data_diff/version.py b/data_diff/version.py
@@ -1 +1 @@
-__version__ = "0.8.1"
+__version__ = "0.8.3"
diff --git a/docs/data-diff-logo.png b/docs/data-diff-logo.png
diff --git a/docs/development_testing.png b/docs/development_testing.png
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "data-diff"
-version = "0.8.1"
+version = "0.8.3"
 description = "Command-line tool and Python library to efficiently diff rows across two different databases."
 authors = ["Datafold <data-diff@datafold.com>"]
 license = "MIT"
@@ -36,7 +36,7 @@ cryptography = {version="*", optional=true}
 trino = {version="^0.314.0", optional=true}
 presto-python-client = {version="*", optional=true}
 clickhouse-driver = {version="*", optional=true}
-duckdb = {version="^0.7.0", optional=true}
+duckdb = {version="*", optional=true}
 dbt-artifacts-parser = {version="^0.4.0"}
 dbt-core = {version="^1.0.0"}
 keyring = "*"
diff --git a/tests/test_config.py b/tests/test_config.py
@@ -15,6 +15,8 @@ def test_basic(self):
 
             [run.default]
             update_column = "timestamp"
+            key_columns = ["id"]
+            columns = ["name", "age"]
             verbose = true
             threads = 2
 
@@ -39,6 +41,8 @@ def test_basic(self):
         assert res["table2"] == "rating_del1"
         assert res["threads1"] == 11
         assert res["threads2"] == 22
+        assert res["key_columns"] == ("id",)
+        assert res["columns"] == ("name", "age")
 
         res = apply_config_from_string(config, "pg_pg", {"update_column": "foo", "table2": "bar"})
         assert res["update_column"] == "foo"
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
@@ -70,3 +70,47 @@ def test_uuid(self):
         self.connection.query(self.table_src.drop(True))
         self.connection.query(self.table_dst.drop(True))
         mysql_conn.query(self.table_dst.drop(True))
+
+
+class Test100Fields(unittest.TestCase):
+    def setUp(self) -> None:
+        self.connection = get_conn(db.PostgreSQL)
+
+        table_suffix = random_table_suffix()
+
+        self.table_src_name = f"src{table_suffix}"
+        self.table_dst_name = f"dst{table_suffix}"
+
+        self.table_src = table(self.table_src_name)
+        self.table_dst = table(self.table_dst_name)
+
+    def test_100_fields(self):
+        self.connection.query('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";', None)
+
+        columns = [f"col{i}" for i in range(100)]
+        fields = " ,".join(f'"{field}" TEXT' for field in columns)
+
+        queries = [
+            self.table_src.drop(True),
+            self.table_dst.drop(True),
+            f"CREATE TABLE {self.table_src_name} (id uuid DEFAULT uuid_generate_v4 (), {fields})",
+            commit,
+            self.table_src.insert_rows([[f"{x * y}" for x in range(100)] for y in range(10)], columns=columns),
+            commit,
+            self.table_dst.create(self.table_src),
+            commit,
+            self.table_src.insert_rows([[1 for x in range(100)]], columns=columns),
+            commit,
+        ]
+
+        for query in queries:
+            self.connection.query(query)
+
+        a = TableSegment(self.connection, self.table_src.path, ("id",), extra_columns=tuple(columns))
+        b = TableSegment(self.connection, self.table_dst.path, ("id",), extra_columns=tuple(columns))
+
+        differ = HashDiffer()
+        diff = list(differ.diff_tables(a, b))
+        id_ = diff[0][1][0]
+        result = (id_,) + tuple("1" for x in range(100))
+        self.assertEqual(diff, [("-", result)])

Original file line number	Diff line number	Diff line change
`@@ -1 +1 @@`
`1`		`-__version__ = "0.8.1"`
	`1`	`+__version__ = "0.8.3"`