Refine alerts and tasks (#507)

chenziliang · web-flow · commit 1d871007229c · 2025-10-14T14:23:01.000-07:00
diff --git a/docs/alert.md b/docs/alert.md
@@ -1,68 +1,136 @@
 # Alert
 
-Timeplus alerts enable you to monitor your streaming data and automatically trigger actions when specific conditions are met. When your streaming queries detect events of interest, alerts can notify stakeholders via email or Slack, send data to downstream systems like Apache Kafka, or execute custom Python functions for automated responses.
+## Overview
 
-## Create New Alert
+A **Timeplus Alert** continuously monitors events in a source stream using a **streaming query**. When the specified conditions are met, it triggers a [Python UDF](/py-udf) to **interact with external systems** — for example, sending notifications to **Slack**, **Email**, or other services.
+
+Alerts are often used in combination with [Scheduled Tasks](/task) and [Materialized Views](/materialized-view) to form a **complete, automated data pipeline**, as illustrated below:
+
+![AlertPipeline](/img/alert-pipeline.png)
+
+## Create Alert
 
-### Syntax
 ```sql
-CREATE [OR REPLACE] ALERT [IF NOT EXISTS] [database.]alert_name
-BATCH <N> EVENTS WITH TIMEOUT <nUnit>
-LIMIT <N> ALERTS PER <nUnit>
-CALL <python_udf_name>
-AS <select_query>;
+CREATE ALERT [IF NOT EXISTS] <db.alert-name>
+BATCH <N> EVENTS WITH TIMEOUT <interval>
+LIMIT <M> ALERTS PER <interval>
+CALL <python-udf-name>
+AS <streaming-select-query>;
 ```
 
-For example:
+:::info
+The **streaming select query** in an **Alert** must be a **simple** `SELECT` statement that consumes data directly from a stream.
+Stateful operations such as **joins** or **aggregations** are **not supported** within Alerts.
+
+If your use case requires joins, aggregations, or other complex logic, create a **Materialized View** first to perform those computations and **materialize** the results into a target stream.
+Then, the Alert can consume that **pre-computed stream** to trigger external actions.
+:::
+
+### `BATCH N EVENTS WITH TIMEOUT interval`
+
+Defines how events are **batched** before invoking the alert action, improving efficiency and throughput.  
+Batching can also help with **alert suppression**, reducing redundant or noisy alerts.
+
+The alert is triggered when **either** of the following conditions is met:
+- The batch accumulates **`N` events**, or  
+- The specified **timeout interval** elapses.
+
+**Example:**
+
 ```sql
-CREATE ALERT default.test
 BATCH 10 EVENTS WITH TIMEOUT 5s
-LIMIT 1 ALERTS PER 15s
-CALL alert_action_proton_new_star
-AS SELECT actor FROM github_events WHERE repo='timeplus-io/proton' AND type='WatchEvent'
 ```
 
-### Limitations
-* The alerts only run on the metadata leader node.
-* The return value of the Python UDF is ignored.
-* The select query cannot include any aggregation or JOIN. You can create a materialized view with complex JOIN or aggregation logic to cache the alert events and `SELECT` the target stream of the materialized view in the alert definition.
-* Check `system.stream_state_log` for the alert states or logs.
-* The checkpoints of the alerts are available in `system.alert_ckpt_log` stream with the `_tp_sn` column.
+In this example, the alert will invoke the configured Python UDF **as soon as**:
+
+- 10 events are collected, or
+- 5 seconds pass — whichever happens first.
+
+### `LIMIT M ALERTS PER interval`
+
+Defines **alert suppression** rules to prevent excessive notifications.  
+This limits the invocation of the configured Python UDF to at most **`M` alerts** within each specified **`interval`**.
 
-### Python UDF
-You can import Python libraries and build the custom alert action via [Python UDF](/py-udf). The return value doesn't matter. Here is an example to send events to a specific Slack channel via Slack webhook:
+**Example:**
 
 ```sql
-CREATE OR REPLACE FUNCTION alert_action_proton_new_star(actor string) RETURNS string LANGUAGE PYTHON AS $$
+LIMIT 1 ALERTS PER 10s
+```
+
+This example restricts the system to trigger **no more than one alert every 10 seconds**,
+even if multiple batches or events meet the alert conditions during that period.
+
+### `CALL python-udf-name`
+
+Specifies the **Python UDF** to invoke when events are emitted from the streaming query and both the **batching** and **suppression** conditions are met.  
+
+The Python UDF must have a compatible function signature (input parameters) that matches the **projection output** of the streaming query (`SELECT` clause).
+
+:::info
+Currently, only **Python UDFs** are supported.
+:::
+
+### Checkpoint
+
+Timeplus automatically **checkpoints** the sequence numbers (or offsets) of the source stream.  
+This ensures that upon recovery, **duplicate alerts are not triggered**.
+
+:::info
+Alert checkpoints are stored in the `system.alert_ckpt_log` **Mutable Stream**.
+:::
+
+### Example
+
+```sql
+CREATE FUNCTION send_star_events_to_slack(actor string) 
+RETURNS string 
+LANGUAGE PYTHON AS $$
 import json
 import requests
-def alert_action_proton_new_star(value):
-    for i in range(len(value)):
-        github_id=value[i]
-        requests.post("https://hooks.slack.com/services/T123/B456/other_id", data=json.dumps({"text": f"New 🌟 for Timeplus Proton from https://github.com/{github_id}"}))
+
+def send_star_events_to_slack(value):
+    for github_id in value:
+        requests.post(
+            "https://hooks.slack.com/services/T123/B456/other_id",
+            data=json.dumps({
+                "text": f"New 🌟 for Timeplus Proton from https://github.com/{github_id}"
+            })
+        )
     return value
 $$
+
+CREATE ALERT default.watch_event_alert
+BATCH 10 EVENTS WITH TIMEOUT 5s
+LIMIT 1 ALERTS PER 15s
+CALL send_star_events_to_slack
+AS 
+SELECT actor 
+FROM github_events 
+WHERE repo = 'timeplus-io/proton' AND type = 'WatchEvent';
 ```
-Please note, similar to regular Python UDF, the input parameter of the Python UDF is an array, instead of a single event. The return value can be anything but you can return the input value, so that you can test the Python UDF via `SELECT udf_name(input)`.
+
+**Explanation**:
+
+- The alert continuously monitors the `github_events` stream in the background.
+- When a new `WatchEvent` occurs for the GitHub repository `timeplus-io/proton`, the Python UDF `send_star_events_to_slack` is triggered.
+- The UDF posts a formatted message to a Slack webhook, notifying the team of new stars.
+- To avoid alert flooding, the alert is **rate-limited** to **1 alert every 15 seconds**.
+- The alert also **batches events** for efficiency — the UDF is invoked when either **10 events** accumulate or **5 seconds** have passed, whichever comes first.
 
 ## List Alerts
+
 ```sql
-SHOW ALERTS [FROM database_name] [SETTINGS verbose=true]
+SHOW ALERTS [FROM db] [SETTINGS verbose=true]
 ```
 
-Without `SETTINGS verbose=true`, it lists the alert name and its UUID. With `SETTINGS verbose=true`, the following columns are added:
-* version
-* last_modified
-* last_modified_by
-* created
-* created_by
+## Show Alert
 
-## Show Alert Definition
 ```sql
-SHOW CREATE ALERT [database.]alert_name [SETTINGS show_multi_versions=true]
+SHOW CREATE ALERT <db.alert-name> [SETTINGS show_multi_versions=true]
 ```
 
-## Drop Alerts
+## Drop Alert
+
 ```sql
-DROP ALERT [database.]alert_name
+DROP ALERT <db.alert-name>
 ```
diff --git a/docs/task.md b/docs/task.md
@@ -1,8 +1,12 @@
-# Overview
+# Task
 
-A **Timeplus scheduled task** runs a historical query periodically according to its schedule and persists the query results to a target Timeplus native stream or external system (e.g., ClickHouse).  When combined with Python UDFs, scheduled tasks can move data between external systems and Timeplus, or between different external systems.
+## Overview
 
-Scheduled tasks complement **Timeplus Materialized Views** which run streaming queries and continuously materialize results to target streams or external systems.
+A **Timeplus Scheduled Task** runs a historical query periodically according to its schedule and persists the query results to a target Timeplus native stream or external system (e.g., ClickHouse).  When combined with Python UDFs, scheduled tasks can move data between external systems and Timeplus, or between different external systems.
+
+Scheduled tasks complement **Timeplus Materialized Views** which run streaming queries and continuously materialize results to target streams or external systems. Tasks are often used in combination with [Alerts](/alert) and [Materialized Views](/materialized-view) to form a **complete, automated data pipeline**, as illustrated below:
+
+![AlertPipeline](/img/alert-pipeline.png)
 
 ## Create Task
 
@@ -15,15 +19,18 @@ AS
   <Historical SELECT query>;
 ```
 
-**SCHEDULE interval** : The interval at which the task runs.
+### `SCHEDULE interval`
+
+The interval at which the task runs.
 Tasks are scheduled via a centralized scheduler to prevent overlap: the next run starts only after the previous run completes.
 
-**TIMEOUT interval** : The maximum allowed execution time for the task.
-If the task exceeds this interval, the scheduler aborts it to prevent indefinite execution
+### `TIMEOUT interval`
+
+The maximum allowed execution time for the task. If the task exceeds this interval, the scheduler aborts it to prevent indefinite execution
 
 Once created, a task is automatically scheduled in the Timeplus cluster. The scheduler selects the best candidate node in the cluster to execute the task.
 
-**Example**:
+### Example
 
 To periodically collect the Timeplus node statuses and persist the results into a target stream:
 
@@ -40,38 +47,36 @@ AS
   SELECT cluster_id, node_id, node_state FROM system.cluster;
 ```
 
-The *node_states* stream will be populated every 5 seconds with the current status of cluster nodes.
+The `node_states` stream will be populated every 5 seconds with the current status of cluster nodes.
 
-## Show Task
+## List Tasks
 
 ```sql
 -- List all tasks
 SHOW TASKS;
+```
+
+## Show Task
 
+```sql
 -- Show DDL definition of a task
 SHOW CREATE TASK <db.task-name>;
 ```
 
-## Pause Task
-
-To pause a task:
+## Drop Task
 
 ```sql
-SYSTEM PAUSE TASK <db.task-name>;
+DROP TASK <db.task-name>;
 ```
 
-## Resume Task
-
-To resume a paused task:
+## Pause Task
 
 ```sql
-SYSTEM RESUME TASK <db.task-name>;
+SYSTEM PAUSE TASK <db.task-name>;
 ```
 
-## Delete Task
-
-To delete a task:
+## Resume Task
 
 ```sql
-DELETE TASK <db.task-name>;
+SYSTEM RESUME TASK <db.task-name>;
 ```
diff --git a/static/img/alert-pipeline.png b/static/img/alert-pipeline.png
