Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/content/_data/config/pipes/spark/batch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,8 @@ input:

output:
type: com.amadeus.dataio.pipes.spark.batch.StorageOutput
fields:
- name: mode
mandatory: "Yes"
description: Specifies the writing behavior when data or table already exists.
example: mode = "append"
2 changes: 1 addition & 1 deletion docs/content/_data/config/pipes/spark/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,4 @@ fields:
description: Spark options, as key = value pairs. The list of available options is available in the official <a href="https://spark.apache.org/docs/3.5.0/api/java/org/apache/spark/sql/DataFrameReader.html" target="_blank">Spark API documentation</a> for the DataFrameReader
example: options { header = true }

fields_warning: The date_filter field is never mandatory, but be aware that omitting it could result in processing years of data.
fields_warning: The date_filter field is never mandatory, but be aware that omitting it could result in processing years of data.
4 changes: 4 additions & 0 deletions docs/content/_data/config/pipes/spark/streaming.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,7 @@ output:
mandatory: "Yes"
description: Controls the amount of time before returning from the streaming query, in hours.
example: timeout = 24
- name: mode
mandatory: "Yes"
description: Specifies the writing behavior when data or table already exists.
example: mode = "append"