Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions yaml/README_Iceberg_To_MySql_Yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ on [Metadata Annotations](https://github.com/GoogleCloudPlatform/DataflowTemplat
* **jdbcType**: Specifies the type of JDBC source. An appropriate default driver will be packaged. For example, `mysql`.
* **writeStatement**: The SQL query for inserting records, with placeholders for values. For example, `INSERT INTO my_table (col1, col2) VALUES(?, ?)`.
* **batchSize**: The number of records to group together for each write. For example, `1000`. Defaults to: 1000.
* **autosharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.
* **autoSharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.
* **configProperties**: A map of properties to pass to the Hadoop Configuration. For example, `{"fs.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"}`.
* **drop**: A list of field names to drop. Mutually exclusive with 'keep' and 'only'. For example, `["field_to_drop_1", "field_to_drop_2"]`.
* **filter**: A filter expression to apply to records from the Iceberg table. For example, `age > 18`.
Expand Down Expand Up @@ -144,7 +144,7 @@ export CONNECTION_INIT_SQL=<connectionInitSql>
export JDBC_TYPE=mysql
export WRITE_STATEMENT=<writeStatement>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>
export CONFIG_PROPERTIES=<configProperties>
export DROP=<drop>
export FILTER=<filter>
Expand All @@ -165,7 +165,7 @@ gcloud dataflow flex-template run "iceberg-to-mysql-yaml-job" \
--parameters "location=$LOCATION" \
--parameters "writeStatement=$WRITE_STATEMENT" \
--parameters "batchSize=$BATCH_SIZE" \
--parameters "autosharding=$AUTOSHARDING" \
--parameters "autoSharding=$AUTO_SHARDING" \
--parameters "table=$TABLE" \
--parameters "catalogName=$CATALOG_NAME" \
--parameters "catalogProperties=$CATALOG_PROPERTIES" \
Expand Down Expand Up @@ -207,7 +207,7 @@ export CONNECTION_INIT_SQL=<connectionInitSql>
export JDBC_TYPE=mysql
export WRITE_STATEMENT=<writeStatement>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>
export CONFIG_PROPERTIES=<configProperties>
export DROP=<drop>
export FILTER=<filter>
Expand All @@ -220,7 +220,7 @@ mvn clean package -PtemplatesRun \
-Dregion="$REGION" \
-DjobName="iceberg-to-mysql-yaml-job" \
-DtemplateName="Iceberg_To_MySql_Yaml" \
-Dparameters="jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,driverClassName=$DRIVER_CLASS_NAME,driverJars=$DRIVER_JARS,connectionProperties=$CONNECTION_PROPERTIES,connectionInitSql=$CONNECTION_INIT_SQL,jdbcType=$JDBC_TYPE,location=$LOCATION,writeStatement=$WRITE_STATEMENT,batchSize=$BATCH_SIZE,autosharding=$AUTOSHARDING,table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP" \
-Dparameters="jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,driverClassName=$DRIVER_CLASS_NAME,driverJars=$DRIVER_JARS,connectionProperties=$CONNECTION_PROPERTIES,connectionInitSql=$CONNECTION_INIT_SQL,jdbcType=$JDBC_TYPE,location=$LOCATION,writeStatement=$WRITE_STATEMENT,batchSize=$BATCH_SIZE,autoSharding=$AUTO_SHARDING,table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP" \
-f yaml
```

Expand Down Expand Up @@ -279,7 +279,7 @@ resource "google_dataflow_flex_template_job" "iceberg_to_mysql_yaml" {
# jdbcType = "mysql"
# writeStatement = "<writeStatement>"
# batchSize = "1000"
# autosharding = "<autosharding>"
# autoSharding = "<autoSharding>"
# configProperties = "<configProperties>"
# drop = "<drop>"
# filter = "<filter>"
Expand Down
12 changes: 6 additions & 6 deletions yaml/README_Iceberg_To_PostgreSQL_Yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ on [Metadata Annotations](https://github.com/GoogleCloudPlatform/DataflowTemplat
* **postgresTable**: The name of the database table. For example, `public.my_table`.
* **query**: The SQL query/statement to execute on the source/sink. For example, `SELECT * FROM my_table WHERE status = 'active'`.
* **batchSize**: The number of records to group together for each write. For example, `1000`. Defaults to: 1000.
* **autosharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.
* **autoSharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.



Expand Down Expand Up @@ -140,7 +140,7 @@ export CONNECTION_PROPERTIES=<connectionProperties>
export POSTGRES_TABLE=<postgresTable>
export QUERY=<query>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>

gcloud dataflow flex-template run "iceberg-to-postgresql-yaml-job" \
--project "$PROJECT" \
Expand All @@ -160,7 +160,7 @@ gcloud dataflow flex-template run "iceberg-to-postgresql-yaml-job" \
--parameters "postgresTable=$POSTGRES_TABLE" \
--parameters "query=$QUERY" \
--parameters "batchSize=$BATCH_SIZE" \
--parameters "autosharding=$AUTOSHARDING"
--parameters "autoSharding=$AUTO_SHARDING"
```

For more information about the command, please check:
Expand Down Expand Up @@ -195,7 +195,7 @@ export CONNECTION_PROPERTIES=<connectionProperties>
export POSTGRES_TABLE=<postgresTable>
export QUERY=<query>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>

mvn clean package -PtemplatesRun \
-DskipTests \
Expand All @@ -204,7 +204,7 @@ mvn clean package -PtemplatesRun \
-Dregion="$REGION" \
-DjobName="iceberg-to-postgresql-yaml-job" \
-DtemplateName="Iceberg_To_PostgreSQL_Yaml" \
-Dparameters="table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP,jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,connectionProperties=$CONNECTION_PROPERTIES,postgresTable=$POSTGRES_TABLE,query=$QUERY,batchSize=$BATCH_SIZE,autosharding=$AUTOSHARDING" \
-Dparameters="table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP,jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,connectionProperties=$CONNECTION_PROPERTIES,postgresTable=$POSTGRES_TABLE,query=$QUERY,batchSize=$BATCH_SIZE,autoSharding=$AUTO_SHARDING" \
-f yaml
```

Expand Down Expand Up @@ -263,7 +263,7 @@ resource "google_dataflow_flex_template_job" "iceberg_to_postgresql_yaml" {
# postgresTable = "<postgresTable>"
# query = "<query>"
# batchSize = "1000"
# autosharding = "<autosharding>"
# autoSharding = "<autoSharding>"
}
}
```
12 changes: 6 additions & 6 deletions yaml/README_Iceberg_To_SqlServer_Yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on [Metadata Annotations](https://github.com/GoogleCloudPlatform/DataflowTemplat
* **jdbcType**: Specifies the type of JDBC source. An appropriate default driver will be packaged. For example, `mssql`.
* **query**: The SQL query for inserting records, with placeholders for values. For example, `INSERT INTO my_table (col1, col2) VALUES(?, ?)`.
* **batchSize**: The number of records to group together for each write. For example, `1000`. Defaults to: 1000.
* **autosharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.
* **autoSharding**: If true, a dynamic number of shards will be used for writing. For example, `False`.



Expand Down Expand Up @@ -148,7 +148,7 @@ export CONNECTION_INIT_SQL=<connectionInitSql>
export JDBC_TYPE=mssql
export QUERY=<query>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>

gcloud dataflow flex-template run "iceberg-to-sqlserver-yaml-job" \
--project "$PROJECT" \
Expand All @@ -172,7 +172,7 @@ gcloud dataflow flex-template run "iceberg-to-sqlserver-yaml-job" \
--parameters "location=$LOCATION" \
--parameters "query=$QUERY" \
--parameters "batchSize=$BATCH_SIZE" \
--parameters "autosharding=$AUTOSHARDING"
--parameters "autoSharding=$AUTO_SHARDING"
```

For more information about the command, please check:
Expand Down Expand Up @@ -211,7 +211,7 @@ export CONNECTION_INIT_SQL=<connectionInitSql>
export JDBC_TYPE=mssql
export QUERY=<query>
export BATCH_SIZE=1000
export AUTOSHARDING=<autosharding>
export AUTO_SHARDING=<autoSharding>

mvn clean package -PtemplatesRun \
-DskipTests \
Expand All @@ -220,7 +220,7 @@ mvn clean package -PtemplatesRun \
-Dregion="$REGION" \
-DjobName="iceberg-to-sqlserver-yaml-job" \
-DtemplateName="Iceberg_To_SqlServer_Yaml" \
-Dparameters="table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP,jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,driverClassName=$DRIVER_CLASS_NAME,driverJars=$DRIVER_JARS,connectionProperties=$CONNECTION_PROPERTIES,connectionInitSql=$CONNECTION_INIT_SQL,jdbcType=$JDBC_TYPE,location=$LOCATION,query=$QUERY,batchSize=$BATCH_SIZE,autosharding=$AUTOSHARDING" \
-Dparameters="table=$TABLE,catalogName=$CATALOG_NAME,catalogProperties=$CATALOG_PROPERTIES,configProperties=$CONFIG_PROPERTIES,drop=$DROP,filter=$FILTER,keep=$KEEP,jdbcUrl=$JDBC_URL,username=$USERNAME,password=$PASSWORD,driverClassName=$DRIVER_CLASS_NAME,driverJars=$DRIVER_JARS,connectionProperties=$CONNECTION_PROPERTIES,connectionInitSql=$CONNECTION_INIT_SQL,jdbcType=$JDBC_TYPE,location=$LOCATION,query=$QUERY,batchSize=$BATCH_SIZE,autoSharding=$AUTO_SHARDING" \
-f yaml
```

Expand Down Expand Up @@ -283,7 +283,7 @@ resource "google_dataflow_flex_template_job" "iceberg_to_sqlserver_yaml" {
# jdbcType = "mssql"
# query = "<query>"
# batchSize = "1000"
# autosharding = "<autosharding>"
# autoSharding = "<autoSharding>"
}
}
```
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,12 @@ public interface IcebergToMySQLYaml {

@TemplateParameter.Boolean(
order = 12,
name = "autosharding",
name = "autoSharding",
optional = true,
description = "If true, enables using a dynamically determined number of shards to write.",
helpText = "If true, a dynamic number of shards will be used for writing.",
example = "False")
Boolean getAutosharding();
Boolean getAutoSharding();

@TemplateParameter.Text(
order = 13,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,10 @@ public interface IcebergToPostgreSQLYaml {

@TemplateParameter.Boolean(
order = 15,
name = "autosharding",
name = "autoSharding",
optional = true,
description = "If true, enables using a dynamically determined number of shards to write.",
helpText = "If true, a dynamic number of shards will be used for writing.",
example = "False")
Boolean getAutosharding();
Boolean getAutoSharding();
}
Original file line number Diff line number Diff line change
Expand Up @@ -220,10 +220,10 @@ public interface IcebergToSQLServerYaml {

@TemplateParameter.Boolean(
order = 19,
name = "autosharding",
name = "autoSharding",
optional = true,
description = "If true, enables using a dynamically determined number of shards to write.",
helpText = "If true, a dynamic number of shards will be used for writing.",
example = "False")
Boolean getAutosharding();
Boolean getAutoSharding();
}
2 changes: 1 addition & 1 deletion yaml/src/main/python/options/mysql_options.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ options:
type: integer
default: 1000
- order: 4
name: "autosharding"
name: "autoSharding"
description: "If true, enables using a dynamically determined number of shards to write."
help: "If true, a dynamic number of shards will be used for writing."
example: false
Expand Down
2 changes: 1 addition & 1 deletion yaml/src/main/python/options/postgres_options.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ options:
type: integer
default: 1000
- order: 2
name: "autosharding"
name: "autoSharding"
description: "If true, enables using a dynamically determined number of shards to write."
help: "If true, a dynamic number of shards will be used for writing."
example: false
Expand Down
2 changes: 1 addition & 1 deletion yaml/src/main/python/options/sqlserver_options.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ options:
type: integer
default: 1000
- order: 4
name: "autosharding"
name: "autoSharding"
description: "If true, enables using a dynamically determined number of shards to write."
help: "If true, a dynamic number of shards will be used for writing."
example: false
Expand Down
2 changes: 1 addition & 1 deletion yaml/src/main/yaml/IcebergToMySQL.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ pipeline:
url: "{{ jdbcUrl }}"
username: "{{ username }}"
password: "{{ password }}"
auto_sharding: {{ autosharding }}
auto_sharding: {{ autoSharding }}
connection_init_sql: {{ connectionInitSql }}
connection_properties: {{ connectionProperties }}
query: {{ insertQuery }}
Expand Down
2 changes: 1 addition & 1 deletion yaml/src/main/yaml/IcebergToPostgreSQL.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ pipeline:
connection_properties: {{ connectionProperties }}
table: {{ postgresTable }}
batch_size: {{ batchSize }}
autosharding: {{ autosharding }}
auto_sharding: {{ autoSharding }}

options:
streaming: false
Expand Down
2 changes: 1 addition & 1 deletion yaml/src/main/yaml/IcebergToSQLServer.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ pipeline:
query: {{ query }}
table: {{ location }}
batch_size: {{ batchSize }}
autosharding: {{ autosharding }}
auto_sharding: {{ autoSharding }}

options:
streaming: false
Loading