Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions addons/monitoring/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Database Monitoring Add-On
## Purpose
The add-on tracks database connection usage and sends an email alert when usage reaches 70% of `max_connections`. Alerts are stateful (sent once per state change) to avoid spam. Emails include detailed metrics.

## Key features
- Monitors connection usage (Usage = Threads / max_connections).
- Stateful notifications (one email per state change):
- `OK` — Usage < 70% (back to normal).
- `THRESHOLD` — Usage ≥ 70% (threshold exceeded).
- `STATUS_ERROR` — failed to get `mysqladmin status`.
- `MAXCONN_ERROR` — failed to get `max_connections` (`SHOW VARIABLES`).
- Metrics from `mysqladmin status` and `SHOW VARIABLES LIKE 'max_connections'`.
- Configurable schedule (Quartz cron) via add-on settings: 5, 10, 15, 20, 30, 40, 50 minutes.
- Runs on all `sqldb` nodes (group execution).

## How it works
1. Install:
- Downloads `/usr/local/sbin/db-monitoring.sh` to all `sqldb` nodes.
- Creates a runner script `db-monitoring.js` that invokes the shell script on the `sqldb` group, passing `USER_SESSION` and `USER_EMAIL`.
- Installs a system cron job `/etc/cron.d/db-monitoring` and sets interval via `setSchedulerInterval` (every `N` minutes): `*/N * * * * root /usr/local/sbin/db-monitoring.sh check`.
2. Runtime:
- Reads DB credentials from `/.jelenv`: `REPLICA_USER`/`REPLICA_PSWD`.
- Collects metrics: `mysqladmin status` and `SHOW VARIABLES LIKE 'max_connections'`.
- Calculates usage and determines state (OK/THRESHOLD/ERROR).
- Stores the last state in `/var/tmp/db-monitoring.status` and sends email only on state changes.
- On state change, triggers platform event `onCustomNodeEvent [name:executeScript]`, which calls the runner script and sends the email.
- Logs to `/var/log/db-monitoring.log`.

## Email content and metrics
Emails are HTML with bold labels and `<br/>` line breaks. Included:
- Status:
- Uptime — node uptime (days/hours/minutes).
- Threads — current number of active connections.
- Slow queries — number of slow queries.
- Open tables — tables currently open.
- Queries per second avg — average queries per second since start.
- max_connections — maximum concurrent connections.
- Current threads (connections) — current connections count.
- Usage — share of used connections (percent).
- Timestamp — report time.

## Logs and artifacts
- Monitoring log: `/var/log/db-monitoring.log` (start/finish, email send, errors).
- State file: `/var/tmp/db-monitoring.status` (last state to suppress duplicate emails).
- Cron: `/etc/cron.d/db-monitoring` (interval managed by the add-on).

## Configuration
- Monitoring interval is controlled in the add-on settings (5/10/15/20/30/40/50 minutes).
- The add-on updates cron with `setSchedulerInterval` to `*/N * * * *`.
- Email sending uses platform messaging API and requires valid `session` and `userEmail`, which are passed by the add-on during event handling.

154 changes: 154 additions & 0 deletions addons/monitoring/manifest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
type: update
name: Database Monitoring
id: db-monitoring

description:
text: The Database Monitoring add-on tracks MySQL/MariaDB/Percona connection usage and sends single-on-state-change email alerts. It periodically collects mysqladmin status and max_connections, alerts when usage reaches 70%, and includes detailed metrics with human‑readable uptime. The check runs online without stopping database services and supports configurable intervals.
short: Email alerts for DB connection usage with detailed metrics.

logo: /images/database-monitoring.png

baseUrl: https://raw.githubusercontent.com/sych74/mysql-cluster/JE-66040/addons/monitoring

mixins:
- https://cdn.jsdelivr.net/gh/jelastic-jps/mysql-cluster@3.0.0/scripts/common.yml

targetNodes:
nodeType:
- mysql
- mariadb-dockerized
- mariadb
- perconadb

settings:
main:
submitUnchanged: true
fields:
- name: monitorInterval
caption: Monitoring interval
type: list
editable: false
values:
- value: 5
caption: Every 5 minutes
- value: 10
caption: Every 10 minutes
- value: 15
caption: Every 15 minutes
- value: 20
caption: Every 20 minutes
- value: 30
caption: Every 30 minutes
- value: 40
caption: Every 40 minutes
- value: 50
caption: Every 50 minutes
default: 10

- name: user
caption: DB User
type: string
required: true
- name: password
caption: DB Password
type: string
inputType: password
required: true

buttons:
- name: Configure
caption: Configure
action: configure
settings: main
loadingText: Configuring...
successText: The monitoring configs have been updated successfully.

globals:
scriptSufix: db-monitoring
scriptName: ${env.name}-${globals.scriptSufix}
random: ${fn.random}

onAfterClone:
install: ${baseUrl}/manifest.yml?_r=${fn.random}
envName: ${event.response.env.envName}
nodeGroup: ${targetNodes.nodeGroup}
settings:
install: true
monitorInterval: ${settings.monitorInterval}

onInstall:
- getReplicaUser
- downloadScripts
- createScript
- setSchedulerInterval

onUninstall:
- cleanupMonitoring

onBeforeDelete:
- cleanupMonitoring

onCustomNodeEvent [name:executeScript]:
script: |
let URL = "${platformUrl}${globals.scriptName}?appid=" + appid + "&token=${globals.random}&envName=${env.name}&uid=${user.uid}&session=" + session + "&userEmail=${user.email}";
let Transport = com.hivext.api.core.utils.Transport;
resp = new Transport().get(URL);
resp = JSON.parse(resp);
if (resp.response && resp.response != 0) {
return new Transport().get("${platformUrl}/1.0/environment/jerror/rest/jerror?appid=" + appid + "&actionname=db-monitoring&callparameters=" + URL + "&email=${user.email}&errorcode=4121&errormessage=" + encodeURIComponent(resp.response.message) + "&priority=high");
}
return { result: 0 }

actions:

downloadScripts:
- cmd[sqldb]: |-
curl -fsSL ${baseUrl}/scripts/db-monitoring.sh -o /usr/local/sbin/db-monitoring.sh
chmod +x /usr/local/sbin/db-monitoring.sh
user: root

createScript:
script: |
let Response = com.hivext.api.Response;
let Transport = com.hivext.api.core.utils.Transport;
let StrSubstitutor = org.apache.commons.lang3.text.StrSubstitutor;
let scriptBody = new Transport().get("${baseUrl}/scripts/db-monitoring.js");
scriptBody = new StrSubstitutor({token: "${globals.random}"}, "${", "}").replace(scriptBody);

var resp = api.dev.scripting.GetScript(appid, session, "${globals.scriptName}");
if (resp.result == Response.OK) {
api.dev.scripting.DeleteScript(appid, session, "${globals.scriptName}");
}
resp = api.dev.scripting.CreateScript(appid, session, "${globals.scriptName}", "js", scriptBody);
if (resp.result != 0) return resp;
java.lang.Thread.sleep(1000);
resp = api.dev.scripting.Build(appid, session, "${globals.scriptName}");
if (resp.result != 0) return resp;

let command = "sed -ri 's|PLATFORM_DOMAIN=.*|PLATFORM_DOMAIN=\"${platformUrl}\"|g' /usr/local/sbin/db-monitoring.sh; " +
"sed -ri 's|USER_SCRIPT_PATH=.*|USER_SCRIPT_PATH=\"${platformUrl}${globals.scriptName}\"|g' /usr/local/sbin/db-monitoring.sh";
return api.env.control.ExecCmdByGroup("${env.name}", session, "sqldb", toJSON([{ command: command }]), true, false, "root");

setSchedulerInterval:
- cmd[sqldb]: bash /usr/local/sbin/db-monitoring.sh setSchedulerTimeout --interval=${settings.monitorInterval}
user: root

cleanupMonitoring:
script: |
let envName = "${env.name}";
var resp = api.dev.scripting.GetScript(appid, session, "${globals.scriptName}");
if (resp.result == com.hivext.api.Response.OK) {
api.dev.scripting.DeleteScript(appid, session, "${globals.scriptName}");
}

resp = api.env.control.ExecCmdByGroup(envName, session, "sqldb", toJSON([
{command: "rm -f /etc/cron.d/db-monitoring || true"},
{command: "rm -f /usr/local/sbin/db-monitoring.sh || true"},
{command: "systemctl reload crond || service cron reload || true"}
]), true, false, "root");
if (resp.result != 0) return resp;

return { result: 0 };

configure:
- setSchedulerInterval
41 changes: 41 additions & 0 deletions addons/monitoring/scripts/db-monitoring.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
//@reg(envName, token, uid)

var ROOT = "root";
var envName = getParam("envName", "${env.envName}");
var Response = com.hivext.api.Response;
var SQLDB = "sqldb";

function run() {
var tokenParam = String(getParam("token", "")).replace(/\s/g, "");
if (!session && tokenParam != "${token}") {
return {
result: Response.PERMISSION_DENIED,
error: "wrong token",
type: "error",
message: "Token [" + tokenParam + "] does not match",
response: { result: Response.PERMISSION_DENIED }
};
}
var info = jelastic.env.control.GetEnvInfo(envName, session);
if (info.result != 0) return info;

var resp;
var userEmail = getParam("userEmail", "");
var userSession = getParam("session", session);
api.marketplace.console.WriteLog(appid, session, "DB Monitoring: sendEmail started for env " + envName);
var command = "/usr/local/sbin/db-monitoring.sh sendEmail '" + userSession + "' '" + userEmail + "'";

resp = api.env.control.ExecCmdByGroup(envName, session, SQLDB, toJSON([{ command: command }]), true, false, ROOT);
if (resp.result != 0) return resp;

api.marketplace.console.WriteLog(appid, session, "DB Monitoring: sendEmail completed");
return { result: 0 };
}

try {
return run();
} catch (ex) {
return { result: com.hivext.api.Response.ERROR_UNKNOWN, error: "Error: " + toJSON(ex) };
}


Loading