Skip to content

Commit e9706c9

Browse files
Update other files with the new processor configuration options
1 parent 7d5619d commit e9706c9

File tree

2 files changed

+20
-4
lines changed

2 files changed

+20
-4
lines changed

content/integrate/redis-data-integration/data-pipelines/pipeline-config.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -144,9 +144,6 @@ processors:
144144
# Time (in ms) after which data will be read from stream even if
145145
# read_batch_size was not reached.
146146
# duration: 100
147-
# Data type to use in Redis target database: `hash` for Redis Hash,
148-
# `json` for JSON (which requires the RedisJSON module).
149-
# target_data_type: hash
150147
# The batch size for writing data to the target Redis database. Should be
151148
# less than or equal to the read_batch_size.
152149
# write_batch_size: 200
@@ -155,8 +152,26 @@ processors:
155152
# Max size of the deduplication set (default: 1024).
156153
# dedup_max_size: <DEDUP_MAX_SIZE>
157154
# Error handling strategy: ignore - skip, dlq - store rejected messages
158-
# in a dead letter queue
155+
# in a dead letter queue.
159156
# error_handling: dlq
157+
# Dead letter queue max messages per stream.
158+
# dlq_max_messages: 1000
159+
# Data type to use in Redis target database: `hash` for Redis Hash,
160+
# `json` for JSON (which requires the RedisJSON module).
161+
# target_data_type: hash
162+
# Number of processes to use when syncing initial data.
163+
# initial_sync_processes: 4
164+
# Checks if the batch has been written to the replica shard.
165+
# wait_enabled: false
166+
# Timeout in milliseconds when checking write to the replica shard.
167+
# wait_timeout: 1000
168+
# Ensures that a batch has been written to the replica shard and keeps
169+
# retrying if not.
170+
# retry_on_replica_failure: true
171+
# Enable merge as the default strategy to writing JSON documents.
172+
# json_update_strategy: merge
173+
# Use native JSON merge if the target RedisJSON module supports it.
174+
# use_native_json_merge: true
160175
```
161176

162177
## Sections

content/integrate/redis-data-integration/reference/config-yaml-reference.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ Configuration settings that control how data is processed, including batch sizes
5353
| **dlq_max_messages**<br/>(DLQ message limit) | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream<br/>Default: `1000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/> | |
5454
| **target_data_type**<br/>(Target Redis data type) | `string` | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)<br/>Default: `"hash"`<br/>Pattern: `^\${.*}$\|hash\|json`<br/> | |
5555
| **json_update_strategy** | `string` | (DEPRECATED)<br/>Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy.<br/>Default: `"replace"`<br/>Pattern: `^\${.*}$\|replace\|merge`<br/> | |
56+
| **use_native_json_merge**<br/>(Use native JSON merge) | `boolean` | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:<br/><br/>**Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)<br/><br/>**New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))<br/><br/>**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.<br/>Default: `true`<br/> | |
5657
| **initial_sync_processes** | `integer`, `string` | Number of parallel processes for performing initial data synchronization<br/>Default: `4`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `32`<br/> | |
5758
| **idle_sleep_time_ms**<br/>(Idle sleep interval) | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle<br/>Default: `200`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `999999`<br/> | |
5859
| **idle_streams_check_interval_ms**<br/>(Idle streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle<br/>Default: `1000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `999999`<br/> | |

0 commit comments

Comments
 (0)