diff --git a/.github/workflows/xray-config-test.yml b/.github/workflows/xray-config-test.yml index f9b9294..2073e71 100644 --- a/.github/workflows/xray-config-test.yml +++ b/.github/workflows/xray-config-test.yml @@ -2,9 +2,9 @@ name: Xray config (Ansible + xray -test) on: push: - branches: [main, master] + branches: [main, master, develop] pull_request: - branches: [main, master] + branches: [main, master, develop] jobs: test: diff --git a/README.md b/README.md index 0648f3e..fdf11ce 100644 --- a/README.md +++ b/README.md @@ -5,18 +5,20 @@ Languages: **English** | [Русский](README.ru.md) [![CI](https://github.com/AlchemyLink/Raven-server-install/actions/workflows/xray-config-test.yml/badge.svg)](https://github.com/AlchemyLink/Raven-server-install/actions/workflows/xray-config-test.yml) [![License: MPL 2.0](https://img.shields.io/badge/License-MPL_2.0-brightgreen.svg)](LICENSE) -Ansible playbooks for deploying a self-hosted VPN server stack based on [Xray-core](https://github.com/XTLS/Xray-core) and [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe). +Ansible playbooks for deploying a production-ready self-hosted VPN server stack based on [Xray-core](https://github.com/XTLS/Xray-core) and [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe). Designed for censorship circumvention with traffic indistinguishable from regular HTTPS. **What you get:** -- Xray-core with VLESS + XTLS-Reality and VLESS + XHTTP inbounds -- Optional post-quantum VLESS Encryption (mlkem768x25519plus) +- Xray-core with VLESS + XTLS-Reality (TCP) and VLESS + XHTTP (HTTP/2) inbounds +- nginx SNI routing on port 443 — all VPN traffic goes through standard HTTPS port +- Optional post-quantum VLESS Encryption (mlkem768x25519plus, Xray-core ≥ 26.x) - Optional Hysteria2 via [sing-box](https://github.com/SagerNet/sing-box) - [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe) — subscription server: auto-discovers users, serves client configs via personal URLs -- nginx TLS frontend on EU VPS (`nginx_frontend` role) -- nginx relay + TCP stream proxy on RU VPS for routing through a second server (`relay` role) +- [xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter) + VictoriaMetrics + Grafana — monitoring with per-user and per-inbound traffic dashboards +- nginx TLS frontend on EU VPS (`nginx_frontend` role) with PROXY protocol for real client IPs +- nginx SNI relay on RU VPS — hides EU server IP from clients (`relay` role) - Systemd services with config validation before every reload -- Ad and tracker blocking via geosite routing rules +- Ad and tracker blocking via geosite routing rules (`geosite:category-ads-all`) - BBR congestion control and sysctl tuning (`srv_prepare` role) --- @@ -44,48 +46,50 @@ This repo supports two deployment topologies: ### Single-server (minimal) -One VPS running Xray + Raven-subscribe + nginx frontend. +One VPS running Xray + Raven-subscribe + nginx frontend. All traffic enters on port 443 — nginx routes by SNI. ``` -Client ──VLESS+Reality──► VPS:443 (Xray) -Client ──VLESS+XHTTP────► VPS:443 (nginx) ──► VPS:2053 (Xray) -Client ──subscription───► VPS:443 (nginx) ──► VPS:8080 (Raven) +Client ──VLESS+Reality──► VPS:443 (nginx SNI) ──► VPS:4443 (Xray) +Client ──VLESS+XHTTP────► VPS:443 (nginx SNI) ──► VPS:2053 (Xray) +Client ──subscription───► VPS:443 (nginx SNI) ──► VPS:8443 (nginx HTTPS) ──► Raven:8080 ``` ### Dual-server with RU relay (recommended for CIS users) EU VPS runs Xray + nginx_frontend + Raven-subscribe. -RU VPS runs a relay that hides the EU IP from clients. +RU VPS runs an SNI relay that hides the EU IP from clients and passes traffic through. ``` -EU VPS (media.example.com) RU VPS (example.com) -┌───────────────────────────┐ ┌─────────────────────────────┐ -│ Xray :443 TCP │ │ nginx relay │ -│ nginx XHTTP :443 HTTPS │◄─────│ my.example.com → EU:8443 │ -│ nginx stream:8445 TCP │◄─────│ :8444 TCP → EU:8445 TCP │ -│ Raven :8080 local │ └─────────────────────────────┘ -│ nginx front :8443 HTTPS │ ▲ -└───────────────────────────┘ │ - clients +EU VPS RU VPS (example.com) +┌────────────────────────────────┐ ┌─────────────────────────────────────┐ +│ nginx stream :443 (SNI routing)│ │ nginx stream :443 (SNI routing) │ +│ SNI dest.com → Xray :4443 │◄──│ SNI dest.com → EU:443 │ +│ SNI adobe.com → Xray :2053 │◄──│ SNI adobe.com → EU:443 │ +│ SNI my.domain → nginx :8443 │ │ SNI my.domain → local nginx :8443 │ +│ │ │ → EU:8443 → Raven :8080 │ +│ Raven-subscribe :8080 (local) │ └─────────────────────────────────────┘ +└────────────────────────────────┘ ▲ + clients ``` **Client connection flow:** ``` -VLESS Reality: client → RU:8444 (TCP relay) → EU:8445 (nginx stream) → Xray:443 -VLESS XHTTP: client → EU:443 (nginx HTTPS) → Xray:2053 -Subscription: client → my.example.com (RU relay) → EU:8443 → Raven:8080 +VLESS Reality: client → RU:443 (SNI relay) → EU:443 (nginx SNI) → Xray:4443 +VLESS XHTTP: client → RU:443 (SNI relay) → EU:443 (nginx SNI) → Xray:2053 +Subscription: client → my.example.com:443 → RU nginx → EU:8443 → Raven:8080 ``` ### Role map | Role | VPS | Playbook | What it does | |------|-----|----------|--------------| -| `srv_prepare` | EU | `role_xray.yml` | BBR, sysctl, system user | +| `srv_prepare` | EU | `role_xray.yml` | BBR, sysctl tuning, system user `xrayuser` | | `xray` | EU | `role_xray.yml` | Xray binary + split config in `/etc/xray/config.d/` | | `raven_subscribe` | EU | `role_raven_subscribe.yml` | Subscription server, gRPC sync with Xray | -| `nginx_frontend` | EU | `role_nginx_frontend.yml` | nginx TLS proxy + TCP stream relay (port 8443/8445) | +| `nginx_frontend` | EU | `role_nginx_frontend.yml` | nginx SNI routing on :443, HTTPS proxy on :8443, PROXY protocol | +| `monitoring` | EU | `role_monitoring.yml` | xray-stats-exporter + VictoriaMetrics + Grafana | | `sing-box-playbook` | EU | `role_sing-box.yml` | sing-box + Hysteria2 (optional) | -| `relay` | RU | `role_relay.yml` | nginx reverse proxy + TCP stream relay (port 8444) | +| `relay` | RU | `role_relay.yml` | nginx SNI relay on :443 — forwards all VPN traffic to EU | --- @@ -253,24 +257,37 @@ Listens on `127.0.0.1:8080`, proxied by nginx_frontend. ### `nginx_frontend` role -Deploys nginx on the EU VPS as a TLS reverse proxy. Responsibilities: +Deploys nginx on the EU VPS as a TLS frontend and SNI router. Port 443 handles all traffic. +- **Stream SNI routing on :443** — reads SNI from TLS ClientHello, routes by hostname: + - SNI `xhttp-dest.com` → Xray XHTTP `:2053` + - SNI `your-domain.com` → nginx HTTPS `:8443` (Raven-subscribe) + - Default (any other SNI) → Xray VLESS Reality `:4443` +- **PROXY protocol** — passes real client IP to all upstreams (Xray uses `xver: 2`) +- **HTTPS on :8443** — proxies `/sub/`, `/c/`, `/api/` → Raven-subscribe `:8080` - Obtains Let's Encrypt certificate for `nginx_frontend_domain` -- Listens on port **8443** (port 443 is taken by Xray VLESS Reality) -- Proxies XHTTP path → Xray `:2053` -- Proxies subscription/API paths → Raven-subscribe `:8080` -- **TCP stream relay**: port 8445 → `127.0.0.1:443` (passes VLESS Reality through nginx) + +**Important:** When deploying nginx_frontend and Xray inbounds together, always deploy **Xray first** (`--tags xray_inbounds`), then nginx. nginx sends PROXY protocol headers immediately — Xray must be ready to accept them. --- ### `relay` role -Deploys nginx on the RU VPS as a relay. Responsibilities: +Deploys nginx on the RU VPS as an SNI relay. Responsibilities: -- Obtains Let's Encrypt certificates for `relay_domain` and `relay_sub_my` -- Serves a static stub site on `relay_domain` (camouflage) +- **Stream SNI routing on :443** — forwards all VPN traffic to EU VPS:443 by default +- Serves a static stub site on `relay_domain` (camouflage, Let's Encrypt cert) - Proxies `my.relay_domain` → EU VPS nginx_frontend `:8443` (Raven-subscribe) -- **TCP stream relay**: port 8444 → EU VPS `:8445` (VLESS Reality passthrough) + +--- + +### `monitoring` role + +Deploys the full monitoring stack on the EU VPS: + +- **[xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter)** — Prometheus exporter for per-user and per-inbound traffic metrics +- **VictoriaMetrics** — Prometheus-compatible time series database +- **Grafana** — dashboards for traffic, server health, Raven-subscribe status, and alerting rules --- @@ -384,20 +401,21 @@ singbox: | Variable | Default | Description | |----------|---------|-------------| -| `nginx_frontend_domain` | `media.example.com` | EU VPS domain — set to your domain | -| `nginx_frontend_listen_port` | `8443` | nginx HTTPS listen port (not 443 — taken by Xray) | -| `nginx_frontend_xhttp_port` | `2053` | Xray XHTTP upstream port | -| `nginx_frontend_xhttp_path` | `/api/v3/data-sync` | XHTTP path (must match xray config) | -| `nginx_frontend_reality_port` | `8445` | TCP stream relay port for Reality | +| `nginx_frontend_domain` | `media.example.com` | EU VPS domain — used for TLS cert and SNI routing | +| `nginx_frontend_listen_port` | `8443` | nginx HTTPS internal port (proxied from :443 stream) | +| `nginx_frontend_raven_port` | `8080` | Raven-subscribe upstream port | +| `nginx_frontend_stream_xhttp_sni` | `www.adobe.com` | SNI that routes to Xray XHTTP inbound | +| `nginx_frontend_stream_xhttp_port` | `2053` | Xray XHTTP inbound port | +| `nginx_frontend_stream_reality_port` | `4443` | Xray VLESS Reality inbound port (default SNI target) | ### relay (`roles/relay/defaults/main.yml`) | Variable | Default | Description | |----------|---------|-------------| -| `relay_domain` | `example.com` | RU VPS domain — set to your domain | -| `relay_upstream_raven_port` | `8443` | EU nginx_frontend port (must match `nginx_frontend_listen_port`) | -| `relay_stream_port` | `8444` | RU relay TCP port for Reality (exposed to clients) | -| `relay_upstream_xray_port` | `8445` | EU nginx stream port (must match `nginx_frontend_reality_port`) | +| `relay_domain` | `example.com` | RU VPS domain — stub site + SNI routing | +| `relay_sub_my` | `my.example.com` | Subdomain proxied to EU Raven-subscribe | +| `relay_upstream_host` | `EU_VPS_IP` | EU server IP (set in secrets.yml) | +| `relay_upstream_raven_port` | `8443` | EU nginx HTTPS port for Raven-subscribe | | `relay_stub_title` | `Welcome` | Stub site page title | | `relay_stub_description` | `Personal website` | Stub site meta description | @@ -409,11 +427,39 @@ Point the following DNS A records to the correct servers: | Domain | → | Server | Purpose | |--------|---|--------|---------| -| `media.example.com` | → | EU VPS IP | nginx_frontend (XHTTP, Raven) | -| `example.com` | → | RU VPS IP | Relay stub site | -| `my.example.com` | → | RU VPS IP | Relay → Raven-subscribe | +| `media.example.com` | → | EU VPS IP | nginx_frontend (SNI routing, TLS cert) | +| `example.com` | → | RU VPS IP | Relay stub site (camouflage) | +| `my.example.com` | → | RU VPS IP | Relay → Raven-subscribe (subscription links) | + +Clients connect to the RU VPS on port 443 for all protocols — no additional DNS records needed for VPN traffic. + +--- + +## Monitoring (optional) -The RU VPS TCP relay for Reality (port 8444) works by IP — no DNS record needed. +The `monitoring` role deploys a full observability stack on the EU VPS: + +```bash +ansible-playbook roles/role_monitoring.yml -i roles/hosts.yml --vault-password-file vault_password.txt +``` + +**Grafana dashboards included:** +- **Xray — per-user traffic** — upload/download timeseries, top users, per-inbound breakdown (Reality vs XHTTP) +- **Servers EU/RU — status** — CPU, RAM, network, disk, Xray health, Raven-subscribe latency + +**Alerting rules** (Grafana alerts via VictoriaMetrics): +- Xray down +- Raven-subscribe down +- EU/RU server unreachable +- Disk usage > 85% + +To deploy only the binary for [xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter): + +```bash +ansible-playbook roles/role_monitoring.yml -i roles/hosts.yml --vault-password-file vault_password.txt \ + --tags xray_stats_exporter \ + -e "xray_stats_exporter_local_binary=/path/to/xray-stats-exporter" +``` --- @@ -497,6 +543,7 @@ ansible-playbook tests/playbooks/render_conf.yml ## Related Projects - [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe) — subscription server (Go): auto-discovers users from Xray config, syncs via gRPC API, serves personal subscription URLs in Xray JSON / sing-box JSON / share link formats +- [xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter) — Prometheus exporter for per-user and per-inbound Xray traffic metrics - [Xray-core](https://github.com/XTLS/Xray-core) — the VPN core - [sing-box](https://github.com/SagerNet/sing-box) — alternative VPN core (Hysteria2) diff --git a/README.ru.md b/README.ru.md index 3bcfe11..5b9a211 100644 --- a/README.ru.md +++ b/README.ru.md @@ -5,18 +5,20 @@ [![CI](https://github.com/AlchemyLink/Raven-server-install/actions/workflows/xray-config-test.yml/badge.svg)](https://github.com/AlchemyLink/Raven-server-install/actions/workflows/xray-config-test.yml) [![License: MPL 2.0](https://img.shields.io/badge/License-MPL_2.0-brightgreen.svg)](LICENSE) -Ansible-плейбуки для развёртывания самохостинг VPN-стека на основе [Xray-core](https://github.com/XTLS/Xray-core) и [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe). +Ansible-плейбуки для развёртывания production-ready самохостинг VPN-стека на основе [Xray-core](https://github.com/XTLS/Xray-core) и [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe). Весь трафик неотличим от обычного HTTPS. **Что вы получаете:** -- Xray-core с inbound'ами VLESS + XTLS-Reality и VLESS + XHTTP -- Опциональное пост-квантовое VLESS Encryption (mlkem768x25519plus) +- Xray-core с inbound'ами VLESS + XTLS-Reality (TCP) и VLESS + XHTTP (HTTP/2) +- nginx SNI routing на порту 443 — весь VPN-трафик идёт через стандартный HTTPS-порт +- Опциональное пост-квантовое VLESS Encryption (mlkem768x25519plus, Xray-core ≥ 26.x) - Опциональный Hysteria2 через [sing-box](https://github.com/SagerNet/sing-box) - [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe) — сервер подписок: автоматически находит пользователей, раздаёт клиентские конфиги по персональным ссылкам -- nginx TLS frontend на EU VPS (роль `nginx_frontend`) -- nginx relay + TCP stream proxy на RU VPS для маршрутизации через второй сервер (роль `relay`) +- [xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter) + VictoriaMetrics + Grafana — мониторинг с дашбордами трафика по пользователям и протоколам +- nginx TLS frontend на EU VPS с SNI routing и PROXY protocol для реальных IP клиентов +- nginx SNI relay на RU VPS — скрывает EU сервер от клиентов (роль `relay`) - systemd-сервисы с валидацией конфига перед каждым перезапуском -- Блокировка рекламы и публичных трекеров через правила маршрутизации geosite +- Блокировка рекламы и публичных трекеров (`geosite:category-ads-all`) - BBR и тюнинг sysctl (роль `srv_prepare`) --- @@ -44,48 +46,50 @@ Ansible-плейбуки для развёртывания самохостин ### Один сервер (минимальный вариант) -Один VPS с Xray + Raven-subscribe + nginx. +Один VPS с Xray + Raven-subscribe + nginx. Весь трафик через порт 443 — nginx маршрутизирует по SNI. ``` -Клиент ──VLESS+Reality──► VPS:443 (Xray) -Клиент ──VLESS+XHTTP────► VPS:443 (nginx) ──► VPS:2053 (Xray) -Клиент ──подписка───────► VPS:443 (nginx) ──► VPS:8080 (Raven) +Клиент ──VLESS+Reality──► VPS:443 (nginx SNI) ──► VPS:4443 (Xray) +Клиент ──VLESS+XHTTP────► VPS:443 (nginx SNI) ──► VPS:2053 (Xray) +Клиент ──подписка───────► VPS:443 (nginx SNI) ──► VPS:8443 (nginx HTTPS) ──► Raven:8080 ``` ### Два сервера с RU-relay (рекомендуется для пользователей из СНГ) EU VPS: Xray + nginx_frontend + Raven-subscribe. -RU VPS: relay — скрывает EU IP от клиентов. +RU VPS: SNI relay — скрывает EU IP от клиентов, пробрасывает трафик насквозь. ``` -EU VPS (media.example.com) RU VPS (example.com) -┌───────────────────────────┐ ┌─────────────────────────────┐ -│ Xray :443 TCP │ │ nginx relay │ -│ nginx XHTTP :443 HTTPS │◄─────│ my.example.com → EU:8443 │ -│ nginx stream:8445 TCP │◄─────│ :8444 TCP → EU:8445 TCP │ -│ Raven :8080 local │ └─────────────────────────────┘ -│ nginx front :8443 HTTPS │ ▲ -└───────────────────────────┘ │ - клиенты +EU VPS RU VPS (example.com) +┌────────────────────────────────┐ ┌─────────────────────────────────────┐ +│ nginx stream :443 (SNI routing)│ │ nginx stream :443 (SNI routing) │ +│ SNI dest.com → Xray :4443 │◄──│ SNI dest.com → EU:443 │ +│ SNI adobe.com → Xray :2053 │◄──│ SNI adobe.com → EU:443 │ +│ SNI my.domain → nginx :8443 │ │ SNI my.domain → local nginx :8443 │ +│ │ │ → EU:8443 → Raven :8080 │ +│ Raven-subscribe :8080 (локал.) │ └─────────────────────────────────────┘ +└────────────────────────────────┘ ▲ + клиенты ``` **Маршруты подключения клиентов:** ``` -VLESS Reality: клиент → RU:8444 (TCP relay) → EU:8445 (nginx stream) → Xray:443 -VLESS XHTTP: клиент → EU:443 (nginx HTTPS) → Xray:2053 -Подписка: клиент → my.example.com (RU relay) → EU:8443 → Raven:8080 +VLESS Reality: клиент → RU:443 (SNI relay) → EU:443 (nginx SNI) → Xray:4443 +VLESS XHTTP: клиент → RU:443 (SNI relay) → EU:443 (nginx SNI) → Xray:2053 +Подписка: клиент → my.example.com:443 → RU nginx → EU:8443 → Raven:8080 ``` ### Карта ролей | Роль | VPS | Плейбук | Что делает | |------|-----|---------|-----------| -| `srv_prepare` | EU | `role_xray.yml` | BBR, sysctl, системный пользователь | +| `srv_prepare` | EU | `role_xray.yml` | BBR, sysctl, системный пользователь `xrayuser` | | `xray` | EU | `role_xray.yml` | Бинарь Xray + split-конфиг в `/etc/xray/config.d/` | | `raven_subscribe` | EU | `role_raven_subscribe.yml` | Сервер подписок, gRPC-синхронизация с Xray | -| `nginx_frontend` | EU | `role_nginx_frontend.yml` | nginx TLS proxy + TCP stream relay (порты 8443/8445) | +| `nginx_frontend` | EU | `role_nginx_frontend.yml` | nginx SNI routing на :443, HTTPS прокси на :8443, PROXY protocol | +| `monitoring` | EU | `role_monitoring.yml` | xray-stats-exporter + VictoriaMetrics + Grafana | | `sing-box-playbook` | EU | `role_sing-box.yml` | sing-box + Hysteria2 (опционально) | -| `relay` | RU | `role_relay.yml` | nginx reverse proxy + TCP stream relay (порт 8444) | +| `relay` | RU | `role_relay.yml` | nginx SNI relay на :443 — весь VPN-трафик на EU | --- @@ -253,24 +257,37 @@ ansible-playbook roles/role_xray.yml -i roles/hosts.yml --vault-password-file va ### Роль `nginx_frontend` -Деплоит nginx на EU VPS как TLS reverse proxy. Функции: +Деплоит nginx на EU VPS как TLS frontend и SNI router. Порт 443 обрабатывает весь трафик. +- **Stream SNI routing на :443** — читает SNI из TLS ClientHello, маршрутизирует по имени: + - SNI `xhttp-dest.com` → Xray XHTTP `:2053` + - SNI `your-domain.com` → nginx HTTPS `:8443` (Raven-subscribe) + - Default (любой другой SNI) → Xray VLESS Reality `:4443` +- **PROXY protocol** — передаёт реальный IP клиента всем upstream'ам (Xray использует `xver: 2`) +- **HTTPS на :8443** — проксирует `/sub/`, `/c/`, `/api/` → Raven-subscribe `:8080` - Получает Let's Encrypt сертификат для `nginx_frontend_domain` -- Слушает на порту **8443** (порт 443 занят Xray VLESS Reality) -- Проксирует XHTTP path → Xray `:2053` -- Проксирует пути подписки/API → Raven-subscribe `:8080` -- **TCP stream relay**: порт 8445 → `127.0.0.1:443` (проброс VLESS Reality через nginx) + +**Важно:** При одновременном деплое nginx_frontend и Xray inbounds — сначала деплоить **Xray** (`--tags xray_inbounds`), потом nginx. nginx сразу начинает отправлять PROXY protocol заголовки — Xray должен быть готов их принять. --- ### Роль `relay` -Деплоит nginx на RU VPS как relay. Функции: +Деплоит nginx на RU VPS как SNI relay. Функции: -- Получает Let's Encrypt сертификаты для `relay_domain` и `relay_sub_my` -- Отдаёт статический stub-сайт на `relay_domain` (маскировка) +- **Stream SNI routing на :443** — по умолчанию весь VPN-трафик → EU VPS:443 +- Отдаёт статический stub-сайт на `relay_domain` (маскировка, сертификат Let's Encrypt) - Проксирует `my.relay_domain` → EU VPS nginx_frontend `:8443` (Raven-subscribe) -- **TCP stream relay**: порт 8444 → EU VPS `:8445` (проброс VLESS Reality) + +--- + +### Роль `monitoring` + +Деплоит полный стек мониторинга на EU VPS: + +- **[xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter)** — Prometheus exporter для метрик трафика по пользователям и inbound'ам +- **VictoriaMetrics** — Prometheus-совместимая TSDB +- **Grafana** — дашборды трафика, состояния серверов, Raven-subscribe, и правила алертинга --- @@ -384,20 +401,21 @@ singbox: | Переменная | По умолчанию | Описание | |-----------|--------------|---------| -| `nginx_frontend_domain` | `media.example.com` | Домен EU VPS — заменить на свой | -| `nginx_frontend_listen_port` | `8443` | Порт HTTPS nginx (не 443 — занят Xray) | -| `nginx_frontend_xhttp_port` | `2053` | Порт upstream Xray XHTTP | -| `nginx_frontend_xhttp_path` | `/api/v3/data-sync` | Путь XHTTP (должен совпадать с конфигом Xray) | -| `nginx_frontend_reality_port` | `8445` | Порт TCP stream relay для Reality | +| `nginx_frontend_domain` | `media.example.com` | Домен EU VPS — используется для TLS сертификата и SNI routing | +| `nginx_frontend_listen_port` | `8443` | Внутренний порт nginx HTTPS (проксируется из :443 через stream) | +| `nginx_frontend_raven_port` | `8080` | Порт upstream Raven-subscribe | +| `nginx_frontend_stream_xhttp_sni` | `www.adobe.com` | SNI для маршрутизации на Xray XHTTP inbound | +| `nginx_frontend_stream_xhttp_port` | `2053` | Порт Xray XHTTP inbound | +| `nginx_frontend_stream_reality_port` | `4443` | Порт Xray VLESS Reality inbound (цель по умолчанию для SNI) | ### relay (`roles/relay/defaults/main.yml`) | Переменная | По умолчанию | Описание | |-----------|--------------|---------| -| `relay_domain` | `example.com` | Домен RU VPS — заменить на свой | -| `relay_upstream_raven_port` | `8443` | Порт nginx_frontend на EU (должен совпадать с `nginx_frontend_listen_port`) | -| `relay_stream_port` | `8444` | TCP порт RU relay для Reality (открытый для клиентов) | -| `relay_upstream_xray_port` | `8445` | Порт nginx stream на EU (должен совпадать с `nginx_frontend_reality_port`) | +| `relay_domain` | `example.com` | Домен RU VPS — stub-сайт и SNI routing | +| `relay_sub_my` | `my.example.com` | Поддомен, проксируемый на EU Raven-subscribe | +| `relay_upstream_host` | `EU_VPS_IP` | IP EU сервера (задать в secrets.yml) | +| `relay_upstream_raven_port` | `8443` | Порт nginx HTTPS на EU для Raven-subscribe | | `relay_stub_title` | `Welcome` | Заголовок страницы stub-сайта | | `relay_stub_description` | `Personal website` | Мета-описание stub-сайта | @@ -409,11 +427,11 @@ singbox: | Домен | → | Сервер | Назначение | |-------|---|--------|-----------| -| `media.example.com` | → | IP EU VPS | nginx_frontend (XHTTP, Raven) | -| `example.com` | → | IP RU VPS | Stub-сайт relay | -| `my.example.com` | → | IP RU VPS | Relay → Raven-subscribe | +| `media.example.com` | → | IP EU VPS | nginx_frontend (SNI routing, TLS сертификат) | +| `example.com` | → | IP RU VPS | Stub-сайт relay (маскировка) | +| `my.example.com` | → | IP RU VPS | Relay → Raven-subscribe (ссылки подписки) | -TCP relay Reality (порт 8444 на RU VPS) работает по IP — DNS-запись не нужна. +Клиенты подключаются к RU VPS на порт 443 для всех протоколов — дополнительные DNS-записи для VPN-трафика не нужны. --- @@ -494,9 +512,30 @@ ansible-playbook tests/playbooks/render_conf.yml --- +## Мониторинг (опционально) + +Роль `monitoring` разворачивает полный стек наблюдаемости на EU VPS: + +```bash +ansible-playbook roles/role_monitoring.yml -i roles/hosts.yml --vault-password-file vault_password.txt +``` + +**Включённые дашборды Grafana:** +- **Xray — трафик по пользователям** — timeseries upload/download, топ пользователей, разбивка по inbound (Reality vs XHTTP) +- **Серверы EU/RU — состояние** — CPU, RAM, сеть, диск, здоровье Xray, latency Raven-subscribe + +**Правила алертинга:** +- Xray недоступен +- Raven-subscribe недоступен +- EU/RU сервер недоступен +- Диск заполнен > 85% + +--- + ## Связанные проекты - [Raven-subscribe](https://github.com/AlchemyLink/Raven-subscribe) — сервер подписок (Go): автоматически находит пользователей из конфигов Xray, синхронизирует через gRPC API, раздаёт персональные ссылки подписки в форматах Xray JSON / sing-box JSON / share-ссылки +- [xray-stats-exporter](https://github.com/AlchemyLink/xray-stats-exporter) — Prometheus exporter для метрик трафика Xray по пользователям и inbound'ам - [Xray-core](https://github.com/XTLS/Xray-core) — ядро VPN - [sing-box](https://github.com/SagerNet/sing-box) — альтернативное ядро VPN (Hysteria2) diff --git a/roles/monitoring/defaults/main.yml b/roles/monitoring/defaults/main.yml new file mode 100644 index 0000000..a239fbb --- /dev/null +++ b/roles/monitoring/defaults/main.yml @@ -0,0 +1,71 @@ +--- + +# xray-exporter +xray_exporter_version: "latest" # "latest" or pinned e.g. "v0.2.0" +xray_exporter_github_repo: "compassvpn/xray-exporter" +xray_exporter_listen: "127.0.0.1:9550" +xray_exporter_metrics_path: "/scrape" +xray_exporter_xray_endpoint: "127.0.0.1:10085" +xray_exporter_log_path: "/var/log/Xray/access.log" +xray_exporter_log_time_window: 5 # minutes, sliding window for user activity metrics +xray_exporter_bin_dir: "/usr/local/bin" +xray_exporter_data_dir: "/var/lib/xray-exporter" # GeoLite2 *.mmdb download cwd (xrayuser must be writable) +xray_exporter_service_name: "xray-exporter" +xray_exporter_user: "xrayuser" # same user as Xray — needs read access to access.log +xray_exporter_group: "xrayuser" + +# VictoriaMetrics +victoriametrics_version: "latest" # "latest" or pinned e.g. "v1.101.0" +victoriametrics_github_repo: "VictoriaMetrics/VictoriaMetrics" +victoriametrics_listen: "127.0.0.1:8428" +victoriametrics_retention_months: 1 # 1 month retention +victoriametrics_data_dir: "/var/lib/victoriametrics" +victoriametrics_bin_dir: "/usr/local/bin" +victoriametrics_service_name: "victoriametrics" +victoriametrics_scrape_interval: "15s" +victoriametrics_scrape_timeout: "5s" +victoriametrics_scrape_config: "/etc/victoriametrics/scrape.yml" +victoriametrics_user: "victoriametrics" +victoriametrics_group: "victoriametrics" + +# WireGuard network (for ufw rules on RU) +wg_network: "10.10.0.0/24" + +# node_exporter (system metrics: CPU, RAM, disk, network) +node_exporter_version: "latest" +node_exporter_listen: "127.0.0.1:9100" # EU: localhost only +node_exporter_ru_listen: "10.10.0.2:9100" # RU: WireGuard interface only +node_exporter_ru_scrape: "10.10.0.2:9100" # EU scrapes RU via WireGuard +node_exporter_bin_dir: "/usr/local/bin" +node_exporter_service_name: "node_exporter" + +# xray-stats-exporter (per-user traffic via StatsService gRPC) +xray_stats_exporter_listen: "127.0.0.1:9551" +xray_stats_exporter_metrics_path: "/metrics" +xray_stats_exporter_xray_endpoint: "127.0.0.1:10085" +xray_stats_exporter_bin_dir: "/usr/local/bin" +xray_stats_exporter_service_name: "xray-stats-exporter" +xray_stats_exporter_user: "xrayuser" # needs read access to Xray gRPC API and access.log +xray_stats_exporter_group: "xrayuser" +xray_stats_exporter_log_path: "/var/log/Xray/access.log" +xray_stats_exporter_geo_city_db: "/var/lib/xray-exporter/GeoLite2-City.mmdb" +xray_stats_exporter_geo_asn_db: "/var/lib/xray-exporter/GeoLite2-ASN.mmdb" + +# raven-subscribe health probe (VictoriaMetrics scrapes /health → up metric) +raven_subscribe_scrape_addr: "127.0.0.1:8080" + +# Grafana +grafana_listen: "127.0.0.1:13000" +grafana_service_name: "grafana-server" +grafana_data_dir: "/var/lib/grafana" +grafana_log_dir: "/var/log/grafana" +grafana_provisioning_dir: "/etc/grafana/provisioning" +grafana_dashboard_id: 23181 # compassvpn dashboard on grafana.com (file: xray-compassvpn.json) +# Extra: xray-users-traffic.json — upload/download per user (xray-exporter + policy statsUser*) +# Dashboard 23181 embeds ${DS_GRAFANACLOUD-COMPASSVPN-PROM}; file provisioning often has no such variable — replaced in JSON after download +grafana_dashboard_datasource_placeholder: "${DS_GRAFANACLOUD-COMPASSVPN-PROM}" +grafana_prometheus_datasource_name: "grafanacloud-compassvpn-prom" +grafana_prometheus_datasource_uid: "grafanacloud-compassvpn-prom" +# grafana_admin_password: set in secrets.yml (ansible-vault) +# victoriametrics_auth_username: set in secrets.yml (ansible-vault) +# victoriametrics_auth_password: set in secrets.yml (ansible-vault) diff --git a/roles/monitoring/defaults/secrets.yml.example b/roles/monitoring/defaults/secrets.yml.example new file mode 100644 index 0000000..2b88d0b --- /dev/null +++ b/roles/monitoring/defaults/secrets.yml.example @@ -0,0 +1,7 @@ +--- +# Copy to secrets.yml and encrypt: ansible-vault encrypt secrets.yml --vault-password-file vault_password.txt +# Generate password: openssl rand -hex 16 + +grafana_admin_password: "CHANGE_ME" +victoriametrics_auth_username: "metrics" +victoriametrics_auth_password: "CHANGE_ME" diff --git a/roles/monitoring/handlers/main.yml b/roles/monitoring/handlers/main.yml new file mode 100644 index 0000000..5145e2b --- /dev/null +++ b/roles/monitoring/handlers/main.yml @@ -0,0 +1,35 @@ +--- +- name: Restart xray-exporter + ansible.builtin.systemd: + name: "{{ xray_exporter_service_name }}" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" + +- name: Restart node_exporter + ansible.builtin.systemd: + name: "{{ node_exporter_service_name }}" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" + +- name: Restart xray-stats-exporter + ansible.builtin.systemd: + name: "{{ xray_stats_exporter_service_name }}" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" + +- name: Restart victoriametrics + ansible.builtin.systemd: + name: "{{ victoriametrics_service_name }}" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" + +- name: Restart grafana + ansible.builtin.systemd: + name: "{{ grafana_service_name }}" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" diff --git a/roles/monitoring/tasks/grafana.yml b/roles/monitoring/tasks/grafana.yml new file mode 100644 index 0000000..a1a1d1f --- /dev/null +++ b/roles/monitoring/tasks/grafana.yml @@ -0,0 +1,137 @@ +--- +- name: Install Grafana apt dependencies + ansible.builtin.apt: + name: + - apt-transport-https + - software-properties-common + - wget + state: present + update_cache: true + +- name: Ensure /etc/apt/keyrings exists + ansible.builtin.file: + path: /etc/apt/keyrings + state: directory + owner: root + group: root + mode: "0755" + +- name: Remove stale Grafana GPG key if present + ansible.builtin.file: + path: /etc/apt/keyrings/grafana.gpg + state: absent + +- name: Download and dearmor Grafana GPG key + ansible.builtin.shell: + cmd: "curl -fsSL https://apt.grafana.com/gpg.key | gpg --dearmor -o /etc/apt/keyrings/grafana.gpg" + changed_when: true + +- name: Set permissions on Grafana GPG key + ansible.builtin.file: + path: /etc/apt/keyrings/grafana.gpg + owner: root + group: root + mode: "0644" + +- name: Add Grafana apt repository + ansible.builtin.apt_repository: + repo: "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" + filename: grafana + state: present + update_cache: true + +- name: Install Grafana + ansible.builtin.apt: + name: grafana + state: present + +- name: Ensure Grafana provisioning directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + owner: grafana + group: grafana + mode: "0755" + loop: + - "{{ grafana_provisioning_dir }}/datasources" + - "{{ grafana_provisioning_dir }}/dashboards" + - "{{ grafana_provisioning_dir }}/alerting" + - "{{ grafana_data_dir }}/dashboards" + +- name: Deploy grafana.ini + ansible.builtin.template: + src: grafana.ini.j2 + dest: /etc/grafana/grafana.ini + owner: root + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Deploy Grafana datasource (VictoriaMetrics) + ansible.builtin.template: + src: grafana-datasource.yml.j2 + dest: "{{ grafana_provisioning_dir }}/datasources/victoriametrics.yml" + owner: grafana + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Deploy Grafana dashboard provisioner config + ansible.builtin.template: + src: grafana-dashboards.yml.j2 + dest: "{{ grafana_provisioning_dir }}/dashboards/xray.yml" + owner: grafana + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Download compassvpn Grafana dashboard JSON + ansible.builtin.get_url: + url: "https://grafana.com/api/dashboards/{{ grafana_dashboard_id }}/revisions/latest/download" + dest: "{{ grafana_data_dir }}/dashboards/xray-compassvpn.json" + owner: grafana + group: grafana + mode: "0640" + retries: 3 + delay: 3 + +- name: Replace DS_* datasource placeholders with provisioned UID (file provisioning has no variable) + ansible.builtin.replace: + path: "{{ grafana_data_dir }}/dashboards/xray-compassvpn.json" + regexp: "{{ grafana_dashboard_datasource_placeholder | regex_escape }}" + replace: "{{ grafana_prometheus_datasource_uid }}" + notify: Restart grafana + +- name: Deploy Grafana Xray per-user traffic dashboard + ansible.builtin.template: + src: dashboards/xray-users-traffic.json.j2 + dest: "{{ grafana_data_dir }}/dashboards/xray-users-traffic.json" + owner: grafana + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Deploy Grafana server status dashboard + ansible.builtin.template: + src: dashboards/server-status.json.j2 + dest: "{{ grafana_data_dir }}/dashboards/server-status.json" + owner: grafana + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Deploy Grafana alerting rules + ansible.builtin.template: + src: grafana-alerting.yml.j2 + dest: "{{ grafana_provisioning_dir }}/alerting/raven-alerts.yml" + owner: grafana + group: grafana + mode: "0640" + notify: Restart grafana + +- name: Enable and start Grafana + ansible.builtin.systemd: + name: "{{ grafana_service_name }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/monitoring/tasks/main.yml b/roles/monitoring/tasks/main.yml new file mode 100644 index 0000000..72573c5 --- /dev/null +++ b/roles/monitoring/tasks/main.yml @@ -0,0 +1,29 @@ +--- +- name: Load monitoring secrets + ansible.builtin.include_vars: + file: "{{ role_path }}/defaults/secrets.yml" + tags: always + +- name: Deploy node_exporter + ansible.builtin.import_tasks: node_exporter.yml + tags: node_exporter + +- name: Deploy xray-exporter + ansible.builtin.import_tasks: xray_exporter.yml + when: inventory_hostname == 'vm_my_srv' + tags: xray_exporter + +- name: Deploy VictoriaMetrics + ansible.builtin.import_tasks: victoriametrics.yml + when: inventory_hostname == 'vm_my_srv' + tags: victoriametrics + +- name: Deploy xray-stats-exporter + ansible.builtin.import_tasks: xray_stats_exporter.yml + when: inventory_hostname == 'vm_my_srv' + tags: xray_stats_exporter + +- name: Deploy Grafana + ansible.builtin.import_tasks: grafana.yml + when: inventory_hostname == 'vm_my_srv' + tags: grafana diff --git a/roles/monitoring/tasks/node_exporter.yml b/roles/monitoring/tasks/node_exporter.yml new file mode 100644 index 0000000..a80ccf3 --- /dev/null +++ b/roles/monitoring/tasks/node_exporter.yml @@ -0,0 +1,101 @@ +--- +- name: Get latest node_exporter release version + ansible.builtin.uri: + url: "https://api.github.com/repos/prometheus/node_exporter/releases/latest" + method: GET + return_content: true + headers: + Accept: "application/vnd.github.v3+json" + status_code: 200 + register: node_exporter_release_info + run_once: true + retries: 3 + delay: 3 + until: node_exporter_release_info.status == 200 + when: node_exporter_version == "latest" + +- name: Set node_exporter version fact + ansible.builtin.set_fact: + node_exporter_resolved_version: >- + {{ + node_exporter_release_info.json.tag_name + if node_exporter_version == "latest" + else node_exporter_version + }} + +- name: Detect architecture for node_exporter + ansible.builtin.set_fact: + node_exporter_arch: >- + {{ + 'amd64' if ansible_architecture in ['x86_64', 'amd64'] + else 'arm64' if ansible_architecture in ['aarch64', 'arm64'] + else 'armv7' if ansible_architecture.startswith('arm') + else ansible_architecture + }} + +- name: Download node_exporter archive + ansible.builtin.get_url: + url: "https://github.com/prometheus/node_exporter/releases/download/{{ node_exporter_resolved_version }}/node_exporter-{{ node_exporter_resolved_version[1:] }}.linux-{{ node_exporter_arch }}.tar.gz" + dest: "/tmp/node_exporter-{{ node_exporter_resolved_version }}.tar.gz" + mode: "0644" + register: node_exporter_download + +- name: Extract node_exporter binary + ansible.builtin.unarchive: + src: "/tmp/node_exporter-{{ node_exporter_resolved_version }}.tar.gz" + dest: /tmp/ + remote_src: true + when: node_exporter_download.changed + +- name: Install node_exporter binary + ansible.builtin.copy: + src: "/tmp/node_exporter-{{ node_exporter_resolved_version[1:] }}.linux-{{ node_exporter_arch }}/node_exporter" + dest: "{{ node_exporter_bin_dir }}/node_exporter" + owner: root + group: root + mode: "0755" + remote_src: true + when: node_exporter_download.changed + notify: Restart node_exporter + +- name: Remove node_exporter temp files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "/tmp/node_exporter-{{ node_exporter_resolved_version }}.tar.gz" + - "/tmp/node_exporter-{{ node_exporter_resolved_version[1:] }}.linux-{{ node_exporter_arch }}" + +- name: Set node_exporter effective listen address + ansible.builtin.set_fact: + node_exporter_listen: >- + {{ + node_exporter_ru_listen + if inventory_hostname == 'vm_my_ru' + else node_exporter_listen + }} + +- name: Deploy node_exporter systemd unit + ansible.builtin.template: + src: node_exporter.service.j2 + dest: "/etc/systemd/system/node_exporter.service" + owner: root + group: root + mode: "0644" + notify: Restart node_exporter + +- name: Enable and start node_exporter + ansible.builtin.systemd: + name: "{{ node_exporter_service_name }}" + enabled: true + state: started + daemon_reload: true + +- name: Allow node_exporter from WireGuard subnet (RU ufw) + community.general.ufw: + rule: allow + src: "{{ wg_network }}" + port: "9100" + proto: tcp + comment: "node_exporter via WireGuard" + when: inventory_hostname == 'vm_my_ru' diff --git a/roles/monitoring/tasks/ssh_tunnel_ru.yml b/roles/monitoring/tasks/ssh_tunnel_ru.yml new file mode 100644 index 0000000..2e90311 --- /dev/null +++ b/roles/monitoring/tasks/ssh_tunnel_ru.yml @@ -0,0 +1,38 @@ +--- +# Sets up a persistent SSH tunnel on EU: 127.0.0.1:19100 → RU:9100 +# Requires: +# - /etc/ssh/tunnel_ru (private key, generated once manually or via keygen task) +# - /etc/ssh/tunnel_ru_known_hosts (RU host key, populated by ssh-keyscan) +# - 'tunnel' user on RU with authorized_keys: restrict,port-forwarding,permitopen="127.0.0.1:9100",from="EU_IP" + +- name: Scan RU host key into tunnel known_hosts + ansible.builtin.shell: + cmd: > + ssh-keyscan -p {{ hostvars['vm_my_ru']['ansible_port'] }} + -t ed25519 {{ hostvars['vm_my_ru']['ansible_host'] }} + > {{ ssh_tunnel_ru_known_hosts }} + creates: "{{ ssh_tunnel_ru_known_hosts }}" + changed_when: false + +- name: Set tunnel known_hosts permissions + ansible.builtin.file: + path: "{{ ssh_tunnel_ru_known_hosts }}" + owner: root + group: root + mode: "0644" + +- name: Deploy ssh-tunnel-ru systemd unit + ansible.builtin.template: + src: ssh-tunnel-ru.service.j2 + dest: "/etc/systemd/system/{{ ssh_tunnel_ru_service_name }}.service" + owner: root + group: root + mode: "0644" + notify: Restart ssh-tunnel-ru + +- name: Enable and start ssh-tunnel-ru + ansible.builtin.systemd: + name: "{{ ssh_tunnel_ru_service_name }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/monitoring/tasks/victoriametrics.yml b/roles/monitoring/tasks/victoriametrics.yml new file mode 100644 index 0000000..2fb6094 --- /dev/null +++ b/roles/monitoring/tasks/victoriametrics.yml @@ -0,0 +1,132 @@ +--- +- name: Create victoriametrics system group + ansible.builtin.group: + name: "{{ victoriametrics_group }}" + state: present + system: true + +- name: Create victoriametrics system user + ansible.builtin.user: + name: "{{ victoriametrics_user }}" + group: "{{ victoriametrics_group }}" + system: true + shell: /usr/sbin/nologin + create_home: false + +- name: Ensure VictoriaMetrics data directory exists + ansible.builtin.file: + path: "{{ victoriametrics_data_dir }}" + state: directory + owner: "{{ victoriametrics_user }}" + group: "{{ victoriametrics_group }}" + mode: "0750" + +- name: Ensure VictoriaMetrics config directory exists + ansible.builtin.file: + path: /etc/victoriametrics + state: directory + owner: root + group: root + mode: "0755" + +- name: Detect architecture for VictoriaMetrics + ansible.builtin.set_fact: + vm_arch: >- + {{ + 'amd64' if ansible_architecture in ['x86_64', 'amd64'] + else 'arm64' if ansible_architecture in ['aarch64', 'arm64'] + else ansible_architecture + }} + +- name: Get VictoriaMetrics releases list + ansible.builtin.uri: + url: "https://api.github.com/repos/{{ victoriametrics_github_repo }}/releases?per_page=10" + method: GET + return_content: true + headers: + Accept: "application/vnd.github.v3+json" + status_code: 200 + register: vm_release_list + run_once: true + retries: 3 + delay: 3 + until: vm_release_list.status == 200 + when: victoriametrics_version == "latest" + +- name: Set VictoriaMetrics version fact + ansible.builtin.set_fact: + vm_resolved_version: >- + {%- if victoriametrics_version != "latest" -%} + {{ victoriametrics_version }} + {%- else -%} + {%- set asset_pattern = 'victoria-metrics-linux-' + vm_arch + '-' -%} + {%- set ns = namespace(version='') -%} + {%- for release in vm_release_list.json -%} + {%- if ns.version == '' -%} + {%- for asset in release.assets -%} + {%- if asset.name.startswith(asset_pattern) and asset.name.endswith('.tar.gz') and 'enterprise' not in asset.name and 'cluster' not in asset.name -%} + {%- set ns.version = release.tag_name -%} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + {%- endfor -%} + {{ ns.version }} + {%- endif -%} + +- name: Download VictoriaMetrics binary + ansible.builtin.get_url: + url: "https://github.com/{{ victoriametrics_github_repo }}/releases/download/{{ vm_resolved_version }}/victoria-metrics-linux-{{ vm_arch }}-{{ vm_resolved_version }}.tar.gz" + dest: "/tmp/victoria-metrics-{{ vm_resolved_version }}.tar.gz" + mode: "0644" + register: vm_download + +- name: Extract VictoriaMetrics binary + ansible.builtin.unarchive: + src: "/tmp/victoria-metrics-{{ vm_resolved_version }}.tar.gz" + dest: /tmp/ + remote_src: true + when: vm_download.changed + +- name: Install VictoriaMetrics binary + ansible.builtin.copy: + src: /tmp/victoria-metrics-prod + dest: "{{ victoriametrics_bin_dir }}/victoria-metrics" + owner: root + group: root + mode: "0755" + remote_src: true + when: vm_download.changed + notify: Restart victoriametrics + +- name: Remove VictoriaMetrics temp files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "/tmp/victoria-metrics-{{ vm_resolved_version }}.tar.gz" + - "/tmp/victoria-metrics-prod" + +- name: Deploy scrape config + ansible.builtin.template: + src: scrape.yml.j2 + dest: "{{ victoriametrics_scrape_config }}" + owner: root + group: root + mode: "0644" + notify: Restart victoriametrics + +- name: Deploy VictoriaMetrics systemd unit + ansible.builtin.template: + src: victoriametrics.service.j2 + dest: "/etc/systemd/system/victoriametrics.service" + owner: root + group: root + mode: "0644" + notify: Restart victoriametrics + +- name: Enable and start VictoriaMetrics + ansible.builtin.systemd: + name: "{{ victoriametrics_service_name }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/monitoring/tasks/xray_exporter.yml b/roles/monitoring/tasks/xray_exporter.yml new file mode 100644 index 0000000..9beac2a --- /dev/null +++ b/roles/monitoring/tasks/xray_exporter.yml @@ -0,0 +1,67 @@ +--- +- name: Get latest xray-exporter release version + ansible.builtin.uri: + url: "https://api.github.com/repos/{{ xray_exporter_github_repo }}/releases/latest" + method: GET + return_content: true + headers: + Accept: "application/vnd.github.v3+json" + status_code: 200 + register: xray_exporter_release_info + run_once: true + retries: 3 + delay: 3 + until: xray_exporter_release_info.status == 200 + when: xray_exporter_version == "latest" + +- name: Set xray-exporter version fact + ansible.builtin.set_fact: + xray_exporter_resolved_version: >- + {{ + xray_exporter_release_info.json.tag_name + if xray_exporter_version == "latest" + else xray_exporter_version + }} + +- name: Detect architecture for xray-exporter + ansible.builtin.set_fact: + xray_exporter_arch: >- + {{ + 'amd64' if ansible_architecture in ['x86_64', 'amd64'] + else 'arm64' if ansible_architecture in ['aarch64', 'arm64'] + else 'arm' if ansible_architecture.startswith('arm') + else ansible_architecture + }} + +- name: Ensure xray-exporter data directory exists (GeoLite DBs) + ansible.builtin.file: + path: "{{ xray_exporter_data_dir }}" + state: directory + owner: "{{ xray_exporter_user }}" + group: "{{ xray_exporter_group }}" + mode: "0750" + +- name: Download xray-exporter binary + ansible.builtin.get_url: + url: "https://github.com/{{ xray_exporter_github_repo }}/releases/download/{{ xray_exporter_resolved_version }}/xray-exporter-linux-{{ xray_exporter_arch }}" + dest: "{{ xray_exporter_bin_dir }}/xray-exporter" + owner: root + group: root + mode: "0755" + notify: Restart xray-exporter + +- name: Deploy xray-exporter systemd unit + ansible.builtin.template: + src: xray-exporter.service.j2 + dest: "/etc/systemd/system/xray-exporter.service" + owner: root + group: root + mode: "0644" + notify: Restart xray-exporter + +- name: Enable and start xray-exporter + ansible.builtin.systemd: + name: "{{ xray_exporter_service_name }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/monitoring/tasks/xray_stats_exporter.yml b/roles/monitoring/tasks/xray_stats_exporter.yml new file mode 100644 index 0000000..50fd341 --- /dev/null +++ b/roles/monitoring/tasks/xray_stats_exporter.yml @@ -0,0 +1,26 @@ +--- +- name: Copy xray-stats-exporter binary + ansible.builtin.copy: + src: "{{ xray_stats_exporter_local_binary }}" + dest: "{{ xray_stats_exporter_bin_dir }}/xray-stats-exporter" + owner: root + group: root + mode: "0755" + when: xray_stats_exporter_local_binary is defined + notify: Restart xray-stats-exporter + +- name: Deploy xray-stats-exporter systemd unit + ansible.builtin.template: + src: xray-stats-exporter.service.j2 + dest: "/etc/systemd/system/xray-stats-exporter.service" + owner: root + group: root + mode: "0644" + notify: Restart xray-stats-exporter + +- name: Enable and start xray-stats-exporter + ansible.builtin.systemd: + name: "{{ xray_stats_exporter_service_name }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/monitoring/templates/dashboards/server-status.json.j2 b/roles/monitoring/templates/dashboards/server-status.json.j2 new file mode 100644 index 0000000..55ef41c --- /dev/null +++ b/roles/monitoring/templates/dashboards/server-status.json.j2 @@ -0,0 +1,763 @@ +{ + "annotations": {"list": []}, + "editable": true, + "graphTooltip": 1, + "links": [], + "panels": [ + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 0}, + "id": 100, + "title": "Доступность", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "mappings": [ + {"options": {"0": {"color": "red", "text": "DOWN"}, "1": {"color": "green", "text": "UP"}}, "type": "value"} + ], + "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, + "color": {"mode": "thresholds"} + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 0, "y": 1}, + "id": 1, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "orientation": "auto", "textMode": "auto", "colorMode": "background"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "up{job=\"node\", server=\"eu\"}", + "legendFormat": "EU", + "instant": true, + "refId": "A" + } + ], + "title": "EU (64.226.79.239)", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "mappings": [ + {"options": {"0": {"color": "red", "text": "DOWN"}, "1": {"color": "green", "text": "UP"}}, "type": "value"} + ], + "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, + "color": {"mode": "thresholds"} + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 4, "y": 1}, + "id": 2, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "orientation": "auto", "textMode": "auto", "colorMode": "background"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "up{job=\"node\", server=\"ru\"}", + "legendFormat": "RU", + "instant": true, + "refId": "A" + } + ], + "title": "RU (195.19.92.182)", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, + "color": {"mode": "thresholds"} + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 8, "y": 1}, + "id": 3, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "value", "textMode": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "time() - node_boot_time_seconds{server=\"eu\"}", + "legendFormat": "EU uptime", + "instant": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "time() - node_boot_time_seconds{server=\"ru\"}", + "legendFormat": "RU uptime", + "instant": true, + "refId": "B" + } + ], + "title": "Uptime", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "mappings": [ + {"options": {"0": {"color": "red", "text": "DOWN"}, "1": {"color": "green", "text": "UP"}}, "type": "value"} + ], + "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, + "color": {"mode": "thresholds"} + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 12, "y": 1}, + "id": 4, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "background", "textMode": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_up{job=\"xray\"}", + "legendFormat": "EU Xray", + "instant": true, + "refId": "A" + } + ], + "title": "EU Xray", + "type": "stat" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 5}, + "id": 101, + "title": "CPU", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "percentunit", + "min": 0, "max": 1 + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 6}, + "id": 10, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "1 - avg(rate(node_cpu_seconds_total{server=\"eu\", mode=\"idle\"}[$__rate_interval]))", + "legendFormat": "EU CPU usage", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "avg(rate(node_cpu_seconds_total{server=\"eu\", mode=\"iowait\"}[$__rate_interval]))", + "legendFormat": "EU iowait", + "range": true, + "refId": "B" + } + ], + "title": "EU CPU", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "percentunit", + "min": 0, "max": 1 + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 6}, + "id": 11, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "1 - avg(rate(node_cpu_seconds_total{server=\"ru\", mode=\"idle\"}[$__rate_interval]))", + "legendFormat": "RU CPU usage", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "avg(rate(node_cpu_seconds_total{server=\"ru\", mode=\"iowait\"}[$__rate_interval]))", + "legendFormat": "RU iowait", + "range": true, + "refId": "B" + } + ], + "title": "RU CPU", + "type": "timeseries" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 14}, + "id": 102, + "title": "Память", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 15, "showPoints": "never"}, + "unit": "bytes", + "min": 0 + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 15}, + "id": 20, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "node_memory_MemTotal_bytes{server=\"eu\"} - node_memory_MemAvailable_bytes{server=\"eu\"}", + "legendFormat": "EU used", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "node_memory_MemTotal_bytes{server=\"eu\"}", + "legendFormat": "EU total", + "range": true, + "refId": "B" + } + ], + "title": "EU RAM", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 15, "showPoints": "never"}, + "unit": "bytes", + "min": 0 + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 15}, + "id": 21, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "node_memory_MemTotal_bytes{server=\"ru\"} - node_memory_MemAvailable_bytes{server=\"ru\"}", + "legendFormat": "RU used", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "node_memory_MemTotal_bytes{server=\"ru\"}", + "legendFormat": "RU total", + "range": true, + "refId": "B" + } + ], + "title": "RU RAM", + "type": "timeseries" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 23}, + "id": 103, + "title": "Сеть", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "Bps" + }, + "overrides": [ + {"matcher": {"id": "byRegexp", "options": "rx"}, "properties": [{"id": "custom.transform", "value": "negative-Y"}]} + ] + }, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 24}, + "id": 30, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_network_transmit_bytes_total{server=\"eu\", device!~\"lo|docker.*|veth.*\"}[$__rate_interval]))", + "legendFormat": "EU tx", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_network_receive_bytes_total{server=\"eu\", device!~\"lo|docker.*|veth.*\"}[$__rate_interval]))", + "legendFormat": "EU rx", + "range": true, + "refId": "B" + } + ], + "title": "EU сеть (tx вверх / rx вниз)", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "Bps" + }, + "overrides": [ + {"matcher": {"id": "byRegexp", "options": "rx"}, "properties": [{"id": "custom.transform", "value": "negative-Y"}]} + ] + }, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 24}, + "id": 31, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_network_transmit_bytes_total{server=\"ru\", device!~\"lo|docker.*|veth.*\"}[$__rate_interval]))", + "legendFormat": "RU tx", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_network_receive_bytes_total{server=\"ru\", device!~\"lo|docker.*|veth.*\"}[$__rate_interval]))", + "legendFormat": "RU rx", + "range": true, + "refId": "B" + } + ], + "title": "RU сеть (tx вверх / rx вниз)", + "type": "timeseries" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 32}, + "id": 104, + "title": "Диск", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "thresholds": {"mode": "percentage", "steps": [ + {"color": "green", "value": null}, + {"color": "yellow", "value": 70}, + {"color": "red", "value": 85} + ]}, + "unit": "percentunit", "min": 0, "max": 1 + }, + "overrides": [] + }, + "gridPos": {"h": 6, "w": 6, "x": 0, "y": 33}, + "id": 40, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "orientation": "horizontal", "displayMode": "basic", "valueMode": "color"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "1 - node_filesystem_avail_bytes{server=\"eu\", mountpoint=\"/\"} / node_filesystem_size_bytes{server=\"eu\", mountpoint=\"/\"}", + "legendFormat": "EU /", + "instant": true, + "refId": "A" + } + ], + "title": "EU использование диска", + "type": "bargauge" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "thresholds": {"mode": "percentage", "steps": [ + {"color": "green", "value": null}, + {"color": "yellow", "value": 70}, + {"color": "red", "value": 85} + ]}, + "unit": "percentunit", "min": 0, "max": 1 + }, + "overrides": [] + }, + "gridPos": {"h": 6, "w": 6, "x": 6, "y": 33}, + "id": 41, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "orientation": "horizontal", "displayMode": "basic", "valueMode": "color"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "1 - node_filesystem_avail_bytes{server=\"ru\", mountpoint=\"/\"} / node_filesystem_size_bytes{server=\"ru\", mountpoint=\"/\"}", + "legendFormat": "RU /", + "instant": true, + "refId": "A" + } + ], + "title": "RU использование диска", + "type": "bargauge" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "Bps" + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 33}, + "id": 42, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_disk_read_bytes_total{server=\"eu\"}[$__rate_interval]))", + "legendFormat": "EU read", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_disk_written_bytes_total{server=\"eu\"}[$__rate_interval]))", + "legendFormat": "EU write", + "range": true, + "refId": "B" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_disk_read_bytes_total{server=\"ru\"}[$__rate_interval]))", + "legendFormat": "RU read", + "range": true, + "refId": "C" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "sum(rate(node_disk_written_bytes_total{server=\"ru\"}[$__rate_interval]))", + "legendFormat": "RU write", + "range": true, + "refId": "D" + } + ], + "title": "Disk I/O", + "type": "timeseries" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 41}, + "id": 105, + "title": "Xray (EU)", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "Bps" + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 42}, + "id": 50, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "rate(xray_traffic_downlink_bytes_total{dimension=\"inbound\"}[$__rate_interval])", + "legendFormat": "{% raw %}{{target}}{% endraw %} down", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "rate(xray_traffic_uplink_bytes_total{dimension=\"inbound\"}[$__rate_interval])", + "legendFormat": "{% raw %}{{target}}{% endraw %} up", + "range": true, + "refId": "B" + } + ], + "title": "Xray трафик по inbound", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": null}]}, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 12, "y": 42}, + "id": 51, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "value", "textMode": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_uptime_seconds", + "legendFormat": "Xray uptime", + "instant": true, + "refId": "A" + } + ], + "title": "Xray uptime", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 8, "x": 16, "y": 42}, + "id": 52, + "options": {"legend": {"displayMode": "list", "placement": "bottom"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_goroutines", + "legendFormat": "goroutines", + "range": true, + "refId": "A" + } + ], + "title": "Xray goroutines", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": null}]}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 12, "y": 46}, + "id": 53, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "value", "textMode": "auto", "orientation": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_unique_users", + "legendFormat": "unique users", + "instant": true, + "refId": "A" + } + ], + "title": "Активных пользователей", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": null}]}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 16, "y": 46}, + "id": 54, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "value", "textMode": "auto", "orientation": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_total_connections", + "legendFormat": "connections", + "instant": true, + "refId": "A" + } + ], + "title": "Соединений", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 50}, + "id": 55, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "rate(xray_outbound_requests_total{outbound=\"freedom\"}[$__rate_interval])", + "legendFormat": "freedom (пропущено)", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "rate(xray_outbound_requests_total{outbound=\"blocked\"}[$__rate_interval])", + "legendFormat": "blocked (заблокировано)", + "range": true, + "refId": "B" + } + ], + "title": "Xray routing: запросы/с", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 50}, + "id": 56, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_memstats_alloc_bytes", + "legendFormat": "heap alloc", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "xray_memstats_sys_bytes", + "legendFormat": "sys", + "range": true, + "refId": "B" + } + ], + "title": "Xray память (heap)", + "type": "timeseries" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 58}, + "id": 106, + "title": "Raven-subscribe", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "mappings": [ + {"options": {"0": {"color": "red", "text": "DOWN"}, "1": {"color": "green", "text": "UP"}}, "type": "value"} + ], + "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, + "color": {"mode": "thresholds"} + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 4, "x": 0, "y": 59}, + "id": 60, + "options": {"reduceOptions": {"calcs": ["lastNotNull"]}, "colorMode": "background", "textMode": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "up{job=\"raven-subscribe\"}", + "legendFormat": "Raven-subscribe", + "instant": true, + "refId": "A" + } + ], + "title": "Raven-subscribe", + "type": "stat" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"drawStyle": "line", "fillOpacity": 10, "showPoints": "never"}, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 8, "x": 4, "y": 59}, + "id": 61, + "options": {"legend": {"displayMode": "list", "placement": "bottom"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "expr": "scrape_duration_seconds{job=\"raven-subscribe\"}", + "legendFormat": "scrape latency", + "range": true, + "refId": "A" + } + ], + "title": "Raven-subscribe latency (/health)", + "type": "timeseries" + } + + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": ["server", "node", "raven"], + "templating": {"list": []}, + "time": {"from": "now-3h", "to": "now"}, + "timepicker": {}, + "timezone": "browser", + "title": "Серверы EU / RU — состояние", + "uid": "raven-server-status", + "version": 1, + "weekStart": "" +} diff --git a/roles/monitoring/templates/dashboards/xray-users-traffic.json.j2 b/roles/monitoring/templates/dashboards/xray-users-traffic.json.j2 new file mode 100644 index 0000000..c8bd2c7 --- /dev/null +++ b/roles/monitoring/templates/dashboards/xray-users-traffic.json.j2 @@ -0,0 +1,241 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": {"type": "grafana", "uid": "-- Grafana --"}, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "links": [], + "liveNow": false, + "panels": [ + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 0}, + "id": 10, + "title": "Пользователи — трафик", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": { + "axisBorderShow": false, + "drawStyle": "line", + "fillOpacity": 10, + "showPoints": "never" + }, + "unit": "Bps" + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 24, "x": 0, "y": 1}, + "id": 1, + "options": { + "legend": {"displayMode": "table", "placement": "right", "showLegend": true, "calcs": ["lastNotNull", "max"]} + }, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "rate(xray_user_uplink_bytes_total[$__rate_interval])", + "legendFormat": "{% raw %}{{email}}{% endraw %}", + "range": true, + "refId": "A" + } + ], + "title": "Upload (client → server), bytes/s by user", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": { + "axisBorderShow": false, + "drawStyle": "line", + "fillOpacity": 10, + "showPoints": "never" + }, + "unit": "Bps" + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 24, "x": 0, "y": 11}, + "id": 2, + "options": { + "legend": {"displayMode": "table", "placement": "right", "showLegend": true, "calcs": ["lastNotNull", "max"]} + }, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "rate(xray_user_downlink_bytes_total[$__rate_interval])", + "legendFormat": "{% raw %}{{email}}{% endraw %}", + "range": true, + "refId": "A" + } + ], + "title": "Download (server → client), bytes/s by user", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "unit": "bytes", + "min": 0 + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 12, "x": 0, "y": 21}, + "id": 3, + "options": {"orientation": "horizontal", "displayMode": "gradient", "reduceOptions": {"calcs": ["lastNotNull"]}, "valueMode": "color", "showUnfilled": true}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "sort_desc(increase(xray_user_downlink_bytes_total[$__range]))", + "instant": true, + "legendFormat": "{% raw %}{{email}}{% endraw %}", + "refId": "A" + } + ], + "title": "Download за период (топ пользователей)", + "type": "bargauge" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "unit": "bytes", + "min": 0 + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 12, "x": 12, "y": 21}, + "id": 4, + "options": {"orientation": "horizontal", "displayMode": "gradient", "reduceOptions": {"calcs": ["lastNotNull"]}, "valueMode": "color", "showUnfilled": true}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "sort_desc(increase(xray_user_uplink_bytes_total[$__range]))", + "instant": true, + "legendFormat": "{% raw %}{{email}}{% endraw %}", + "refId": "A" + } + ], + "title": "Upload за период (топ пользователей)", + "type": "bargauge" + }, + + { + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 31}, + "id": 20, + "title": "Inbound — трафик по протоколам", + "type": "row" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": { + "axisBorderShow": false, + "drawStyle": "line", + "fillOpacity": 10, + "showPoints": "never" + }, + "unit": "Bps" + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 12, "x": 0, "y": 32}, + "id": 21, + "options": { + "legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["lastNotNull", "max"]} + }, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "rate(xray_inbound_downlink_bytes_total[$__rate_interval])", + "legendFormat": "{% raw %}{{inbound}}{% endraw %} down", + "range": true, + "refId": "A" + }, + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "rate(xray_inbound_uplink_bytes_total[$__rate_interval])", + "legendFormat": "{% raw %}{{inbound}}{% endraw %} up", + "range": true, + "refId": "B" + } + ], + "title": "Трафик по inbound (bytes/s)", + "type": "timeseries" + }, + + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "unit": "bytes", + "min": 0 + }, + "overrides": [] + }, + "gridPos": {"h": 10, "w": 12, "x": 12, "y": 32}, + "id": 22, + "options": {"orientation": "horizontal", "displayMode": "gradient", "reduceOptions": {"calcs": ["lastNotNull"]}, "valueMode": "color", "showUnfilled": true}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "{{ grafana_prometheus_datasource_uid }}"}, + "editorMode": "code", + "expr": "sort_desc(increase(xray_inbound_downlink_bytes_total[$__range]))", + "instant": true, + "legendFormat": "{% raw %}{{inbound}}{% endraw %}", + "refId": "A" + } + ], + "title": "Inbound download за период", + "type": "bargauge" + } + + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": ["xray", "raven", "users"], + "templating": {"list": []}, + "time": {"from": "now-6h", "to": "now"}, + "timepicker": {}, + "timezone": "browser", + "title": "Xray — per-user upload / download", + "uid": "raven-xray-users-traffic", + "version": 2, + "weekStart": "" +} diff --git a/roles/monitoring/templates/grafana-alerting.yml.j2 b/roles/monitoring/templates/grafana-alerting.yml.j2 new file mode 100644 index 0000000..4e1f298 --- /dev/null +++ b/roles/monitoring/templates/grafana-alerting.yml.j2 @@ -0,0 +1,193 @@ +apiVersion: 1 + +groups: + - orgId: 1 + name: raven-critical + folder: Raven + interval: 1m + rules: + + - uid: xray-down + title: "Xray API недоступен" + condition: C + data: + - refId: A + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: "{{ grafana_prometheus_datasource_uid }}" + model: + expr: "xray_up{job=\"xray-stats\"}" + instant: true + refId: A + - refId: C + datasourceUid: __expr__ + model: + type: threshold + refId: C + conditions: + - evaluator: + params: [1] + type: lt + operator: + type: and + query: + params: [A] + reducer: + type: last + noDataState: Alerting + execErrState: Alerting + for: 2m + annotations: + summary: "Xray API не отвечает на gRPC запросы" + description: "xray_up == 0 уже более 2 минут. Сервис Xray может быть упавшим." + labels: + severity: critical + + - uid: raven-subscribe-down + title: "Raven-subscribe недоступен" + condition: C + data: + - refId: A + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: "{{ grafana_prometheus_datasource_uid }}" + model: + expr: "up{job=\"raven-subscribe\"}" + instant: true + refId: A + - refId: C + datasourceUid: __expr__ + model: + type: threshold + refId: C + conditions: + - evaluator: + params: [1] + type: lt + operator: + type: and + query: + params: [A] + reducer: + type: last + noDataState: Alerting + execErrState: Alerting + for: 2m + annotations: + summary: "Raven-subscribe не отвечает на /health" + description: "Сервис raven-subscribe недоступен более 2 минут. Подписки не работают." + labels: + severity: critical + + - uid: eu-server-down + title: "EU сервер недоступен" + condition: C + data: + - refId: A + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: "{{ grafana_prometheus_datasource_uid }}" + model: + expr: "up{job=\"node\", server=\"eu\"}" + instant: true + refId: A + - refId: C + datasourceUid: __expr__ + model: + type: threshold + refId: C + conditions: + - evaluator: + params: [1] + type: lt + operator: + type: and + query: + params: [A] + reducer: + type: last + noDataState: Alerting + execErrState: Alerting + for: 3m + annotations: + summary: "EU сервер (64.226.79.239) недоступен" + description: "node_exporter на EU сервере не отвечает более 3 минут." + labels: + severity: critical + + - uid: ru-server-down + title: "RU сервер недоступен" + condition: C + data: + - refId: A + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: "{{ grafana_prometheus_datasource_uid }}" + model: + expr: "up{job=\"node\", server=\"ru\"}" + instant: true + refId: A + - refId: C + datasourceUid: __expr__ + model: + type: threshold + refId: C + conditions: + - evaluator: + params: [1] + type: lt + operator: + type: and + query: + params: [A] + reducer: + type: last + noDataState: Alerting + execErrState: Alerting + for: 3m + annotations: + summary: "RU сервер (195.19.92.182) недоступен" + description: "node_exporter на RU сервере не отвечает более 3 минут." + labels: + severity: warning + + - uid: eu-disk-high + title: "EU диск заполнен >85%" + condition: C + data: + - refId: A + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: "{{ grafana_prometheus_datasource_uid }}" + model: + expr: "1 - node_filesystem_avail_bytes{server=\"eu\", mountpoint=\"/\"} / node_filesystem_size_bytes{server=\"eu\", mountpoint=\"/\"}" + instant: true + refId: A + - refId: C + datasourceUid: __expr__ + model: + type: threshold + refId: C + conditions: + - evaluator: + params: [0.85] + type: gt + operator: + type: and + query: + params: [A] + reducer: + type: last + noDataState: NoData + execErrState: Error + for: 5m + annotations: + summary: "EU диск заполнен более чем на 85%" + description: "Дисковое пространство на EU сервере критически мало. Могут перестать писаться логи." + labels: + severity: warning diff --git a/roles/monitoring/templates/grafana-dashboards.yml.j2 b/roles/monitoring/templates/grafana-dashboards.yml.j2 new file mode 100644 index 0000000..6c2a6a2 --- /dev/null +++ b/roles/monitoring/templates/grafana-dashboards.yml.j2 @@ -0,0 +1,9 @@ +apiVersion: 1 + +providers: + - name: xray + type: file + disableDeletion: true + updateIntervalSeconds: 60 + options: + path: {{ grafana_data_dir }}/dashboards diff --git a/roles/monitoring/templates/grafana-datasource.yml.j2 b/roles/monitoring/templates/grafana-datasource.yml.j2 new file mode 100644 index 0000000..7d169d9 --- /dev/null +++ b/roles/monitoring/templates/grafana-datasource.yml.j2 @@ -0,0 +1,14 @@ +apiVersion: 1 + +datasources: + - name: "{{ grafana_prometheus_datasource_name }}" + uid: "{{ grafana_prometheus_datasource_uid }}" + type: prometheus + access: proxy + url: http://{{ victoriametrics_listen }} + isDefault: true + editable: false + basicAuth: true + basicAuthUser: {{ victoriametrics_auth_username }} + secureJsonData: + basicAuthPassword: {{ victoriametrics_auth_password }} diff --git a/roles/monitoring/templates/grafana.ini.j2 b/roles/monitoring/templates/grafana.ini.j2 new file mode 100644 index 0000000..b9c1750 --- /dev/null +++ b/roles/monitoring/templates/grafana.ini.j2 @@ -0,0 +1,35 @@ +[server] +http_addr = {{ grafana_listen.split(':')[0] }} +http_port = {{ grafana_listen.split(':')[1] }} +# SSH tunnel: ssh -L 13000:127.0.0.1:13000 ... then open http://localhost:13000 + +[security] +admin_user = admin +admin_password = {{ grafana_admin_password }} +disable_initial_admin_creation = false +cookie_secure = false # SSH tunnel is localhost, no HTTPS needed here +cookie_samesite = lax +secret_key = {{ grafana_admin_password | hash('sha256') }} + +[auth.anonymous] +enabled = false + +[auth] +disable_login_form = false + +[users] +allow_sign_up = false +allow_org_create = false + +[log] +mode = file +level = warn + +[paths] +data = {{ grafana_data_dir }} +logs = {{ grafana_log_dir }} +provisioning = {{ grafana_provisioning_dir }} + +[analytics] +reporting_enabled = false +check_for_updates = false diff --git a/roles/monitoring/templates/node_exporter.service.j2 b/roles/monitoring/templates/node_exporter.service.j2 new file mode 100644 index 0000000..a94d23e --- /dev/null +++ b/roles/monitoring/templates/node_exporter.service.j2 @@ -0,0 +1,27 @@ +[Unit] +Description=Prometheus Node Exporter +After=network.target + +[Service] +User=nobody +Group=nogroup +NoNewPrivileges=true +ExecStart={{ node_exporter_bin_dir }}/node_exporter \ + --web.listen-address={{ node_exporter_listen }} \ + --collector.disable-defaults \ + --collector.cpu \ + --collector.meminfo \ + --collector.filesystem \ + --collector.netdev \ + --collector.loadavg \ + --collector.time \ + --collector.uname \ + --collector.stat \ + --collector.diskstats \ + --collector.conntrack +Restart=on-failure +RestartSec=5s +TimeoutStopSec=10s + +[Install] +WantedBy=multi-user.target diff --git a/roles/monitoring/templates/scrape.yml.j2 b/roles/monitoring/templates/scrape.yml.j2 new file mode 100644 index 0000000..e0fac4c --- /dev/null +++ b/roles/monitoring/templates/scrape.yml.j2 @@ -0,0 +1,38 @@ +global: + scrape_interval: {{ victoriametrics_scrape_interval }} + scrape_timeout: {{ victoriametrics_scrape_timeout }} + +scrape_configs: + - job_name: xray + metrics_path: {{ xray_exporter_metrics_path }} + static_configs: + - targets: + - {{ xray_exporter_listen }} + + - job_name: xray-stats + metrics_path: {{ xray_stats_exporter_metrics_path }} + static_configs: + - targets: + - {{ xray_stats_exporter_listen }} + + - job_name: node + static_configs: + - targets: + - {{ node_exporter_listen }} + labels: + server: eu + instance: "{{ hostvars['vm_my_srv']['ansible_host'] }}" + - targets: + - {{ node_exporter_ru_scrape }} # WireGuard → RU:9100 + labels: + server: ru + instance: "{{ hostvars['vm_my_ru']['ansible_host'] }}" + + # Raven-subscribe health probe — up=1 if service responds to /health + # VictoriaMetrics records the scrape as up=0 when the endpoint is unreachable, + # regardless of whether the response body is valid Prometheus format. + - job_name: raven-subscribe + metrics_path: /health + static_configs: + - targets: + - {{ raven_subscribe_scrape_addr }} diff --git a/roles/monitoring/templates/ssh-tunnel-ru.service.j2 b/roles/monitoring/templates/ssh-tunnel-ru.service.j2 new file mode 100644 index 0000000..862e5aa --- /dev/null +++ b/roles/monitoring/templates/ssh-tunnel-ru.service.j2 @@ -0,0 +1,24 @@ +[Unit] +Description=SSH tunnel to RU node_exporter ({{ node_exporter_tunnel_listen }} -> RU:9100) +After=network.target +Wants=network-online.target + +[Service] +User=root +ExecStart=/usr/bin/ssh \ + -i {{ ssh_tunnel_ru_key }} \ + -o StrictHostKeyChecking=yes \ + -o UserKnownHostsFile={{ ssh_tunnel_ru_known_hosts }} \ + -o ServerAliveInterval=30 \ + -o ServerAliveCountMax=3 \ + -o ExitOnForwardFailure=yes \ + -o BatchMode=yes \ + -p {{ hostvars['vm_my_ru']['ansible_port'] }} \ + {{ ssh_tunnel_ru_user }}@{{ hostvars['vm_my_ru']['ansible_host'] }} \ + -L {{ node_exporter_tunnel_listen }}:127.0.0.1:9100 -N +Restart=on-failure +RestartSec=10s +TimeoutStopSec=5s + +[Install] +WantedBy=multi-user.target diff --git a/roles/monitoring/templates/victoriametrics.service.j2 b/roles/monitoring/templates/victoriametrics.service.j2 new file mode 100644 index 0000000..634f357 --- /dev/null +++ b/roles/monitoring/templates/victoriametrics.service.j2 @@ -0,0 +1,23 @@ +[Unit] +Description=VictoriaMetrics time series database +Documentation=https://docs.victoriametrics.com +After=network.target + +[Service] +User={{ victoriametrics_user }} +Group={{ victoriametrics_group }} +NoNewPrivileges=true +ExecStart={{ victoriametrics_bin_dir }}/victoria-metrics \ + -httpListenAddr={{ victoriametrics_listen }} \ + -storageDataPath={{ victoriametrics_data_dir }} \ + -retentionPeriod={{ victoriametrics_retention_months }}M \ + -promscrape.config={{ victoriametrics_scrape_config }} \ + -httpAuth.username={{ victoriametrics_auth_username }} \ + -httpAuth.password={{ victoriametrics_auth_password }} +Restart=on-failure +RestartSec=5s +TimeoutStopSec=30s +LimitNOFILE=262144 + +[Install] +WantedBy=multi-user.target diff --git a/roles/monitoring/templates/xray-exporter.service.j2 b/roles/monitoring/templates/xray-exporter.service.j2 new file mode 100644 index 0000000..354eeb2 --- /dev/null +++ b/roles/monitoring/templates/xray-exporter.service.j2 @@ -0,0 +1,22 @@ +[Unit] +Description=Xray Prometheus Exporter +Documentation=https://github.com/compassvpn/xray-exporter +After=network.target xray.service + +[Service] +User={{ xray_exporter_user }} +Group={{ xray_exporter_group }} +WorkingDirectory={{ xray_exporter_data_dir }} +NoNewPrivileges=true +ExecStart={{ xray_exporter_bin_dir }}/xray-exporter \ + --listen={{ xray_exporter_listen }} \ + --metrics-path={{ xray_exporter_metrics_path }} \ + --xray-endpoint={{ xray_exporter_xray_endpoint }} \ + --log-path={{ xray_exporter_log_path }} \ + --log-time-window={{ xray_exporter_log_time_window }} +Restart=on-failure +RestartSec=5s +TimeoutStopSec=10s + +[Install] +WantedBy=multi-user.target diff --git a/roles/monitoring/templates/xray-stats-exporter.service.j2 b/roles/monitoring/templates/xray-stats-exporter.service.j2 new file mode 100644 index 0000000..df9d86a --- /dev/null +++ b/roles/monitoring/templates/xray-stats-exporter.service.j2 @@ -0,0 +1,25 @@ +[Unit] +Description=Xray Per-User Stats Prometheus Exporter +After=network.target xray.service + +[Service] +User={{ xray_stats_exporter_user }} +Group={{ xray_stats_exporter_group }} +NoNewPrivileges=true +ProtectSystem=strict +PrivateTmp=yes +ProtectHome=yes +ReadOnlyPaths={{ xray_stats_exporter_log_path }} /var/lib/xray-exporter +ExecStart={{ xray_stats_exporter_bin_dir }}/xray-stats-exporter \ + --listen={{ xray_stats_exporter_listen }} \ + --metrics-path={{ xray_stats_exporter_metrics_path }} \ + --xray-endpoint={{ xray_stats_exporter_xray_endpoint }} \ + --log-path={{ xray_stats_exporter_log_path }} \ + --geo-city-db={{ xray_stats_exporter_geo_city_db }} \ + --geo-asn-db={{ xray_stats_exporter_geo_asn_db }} +Restart=on-failure +RestartSec=5s +TimeoutStopSec=10s + +[Install] +WantedBy=multi-user.target diff --git a/roles/nginx_frontend/defaults/main.yml b/roles/nginx_frontend/defaults/main.yml index dccf8ee..839d384 100644 --- a/roles/nginx_frontend/defaults/main.yml +++ b/roles/nginx_frontend/defaults/main.yml @@ -1,10 +1,11 @@ --- -# nginx_frontend role — TLS frontend for EU VPS +# nginx_frontend role — TLS frontend + SNI stream routing for EU VPS # # Responsibilities: # - Install nginx + certbot # - Obtain Let's Encrypt certificate for nginx_frontend_domain -# - Proxy Xray XHTTP (nginx_frontend_xhttp_path) → 127.0.0.1:nginx_frontend_xhttp_port +# - SNI routing on :443 → Xray XHTTP, Xray Reality, or nginx HTTPS +# - Proxy Raven-subscribe (/sub/, /c/, /api/, /health) on :8443 # ── Domain ──────────────────────────────────────────────────────────────────── nginx_frontend_domain: "media.example.com" # Set to your EU VPS domain @@ -12,21 +13,20 @@ nginx_frontend_domain: "media.example.com" # Set to your EU VPS domain # ── Certbot ─────────────────────────────────────────────────────────────────── nginx_frontend_certbot_email: "" # Set in secrets.yml -# ── nginx listen port ───────────────────────────────────────────────────────── -# IMPORTANT: Xray VLESS Reality already binds to 443 (TCP). -# nginx_frontend must listen on a different port (e.g., 8443, 9443). -# The relay role will proxy to this port over HTTPS with SNI. -nginx_frontend_listen_port: 8443 # Must NOT conflict with xray_vless_port (443) +# ── nginx HTTPS listen port ────────────────────────────────────────────────── +# Serves Raven-subscribe (subscriptions + admin API) and acts as Reality dest +# for XHTTP probe responses. NOT the main entry point — :443 stream is. +nginx_frontend_listen_port: 8443 # ── Raven-subscribe upstream ────────────────────────────────────────────────── nginx_frontend_raven_port: 8080 # Must match raven_subscribe_listen_addr port -# ── Xray XHTTP upstream ─────────────────────────────────────────────────────── -nginx_frontend_xhttp_port: 2053 # Must match xray_xhttp.port -nginx_frontend_xhttp_path: "/api/v3/data-sync" # Must match xray_xhttp.xhttpSettings.path - -# ── TCP stream relay for Xray VLESS Reality ─────────────────────────────────── -# Stream proxy: nginx_frontend_reality_port → 127.0.0.1:443 (Xray) -# Allows clients to reach Reality via media.example.com instead of direct EU IP. -nginx_frontend_reality_stream_enabled: true -nginx_frontend_reality_port: 8445 # External TCP port for Reality stream +# ── SNI stream routing on :443 ─────────────────────────────────────────────── +# nginx stream with ssl_preread reads SNI from TLS ClientHello and routes: +# SNI = xhttp_sni (e.g. www.adobe.com) → Xray XHTTP+Reality (xhttp_port) +# SNI = nginx_frontend_domain → nginx HTTPS (listen_port) for Raven +# default (e.g. askubuntu.com) → Xray Reality TCP (reality_port) +nginx_frontend_stream_enabled: true +nginx_frontend_stream_xhttp_sni: "www.adobe.com" # Must match xray_xhttp.reality.server_names[0] +nginx_frontend_stream_xhttp_port: 2053 # Must match xray_xhttp.port +nginx_frontend_stream_reality_port: 4443 # Must match xray_vless_port diff --git a/roles/nginx_frontend/inventory.ini b/roles/nginx_frontend/inventory.ini index 763a413..4891780 100644 --- a/roles/nginx_frontend/inventory.ini +++ b/roles/nginx_frontend/inventory.ini @@ -1,2 +1,2 @@ [eu] -vpn ansible_host=EU_VPS_IP ansible_user=deploy +vpn ansible_host=64.226.79.239 ansible_user=konkov ansible_port=2200 ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_ssh_host_key_checking=false diff --git a/roles/nginx_frontend/tasks/nginx_ssl.yml b/roles/nginx_frontend/tasks/nginx_ssl.yml index cedb92e..066c5b3 100644 --- a/roles/nginx_frontend/tasks/nginx_ssl.yml +++ b/roles/nginx_frontend/tasks/nginx_ssl.yml @@ -2,7 +2,7 @@ - name: Nginx frontend | Deploy HTTPS config ansible.builtin.template: src: nginx/https.conf.j2 - dest: "/etc/nginx/conf.d/{{ nginx_frontend_domain }}.conf" + dest: "/etc/nginx/sites-enabled/{{ nginx_frontend_domain }}.conf" owner: root group: root mode: "0644" diff --git a/roles/nginx_frontend/tasks/stream.yml b/roles/nginx_frontend/tasks/stream.yml index 7ea512c..567cc0b 100644 --- a/roles/nginx_frontend/tasks/stream.yml +++ b/roles/nginx_frontend/tasks/stream.yml @@ -18,7 +18,7 @@ insertafter: EOF notify: Reload nginx -- name: Nginx frontend | Deploy Reality stream config +- name: Nginx frontend | Deploy SNI stream routing config ansible.builtin.template: src: nginx/stream.conf.j2 dest: "/etc/nginx/stream.d/{{ nginx_frontend_domain }}.conf" @@ -26,11 +26,11 @@ group: root mode: "0644" notify: Reload nginx - when: nginx_frontend_reality_stream_enabled + when: nginx_frontend_stream_enabled -- name: Nginx frontend | Remove Reality stream config (disabled) +- name: Nginx frontend | Remove SNI stream config (disabled) ansible.builtin.file: path: "/etc/nginx/stream.d/{{ nginx_frontend_domain }}.conf" state: absent notify: Reload nginx - when: not nginx_frontend_reality_stream_enabled + when: not nginx_frontend_stream_enabled diff --git a/roles/nginx_frontend/templates/nginx/https.conf.j2 b/roles/nginx_frontend/templates/nginx/https.conf.j2 index bed7ef5..9fddad0 100644 --- a/roles/nginx_frontend/templates/nginx/https.conf.j2 +++ b/roles/nginx_frontend/templates/nginx/https.conf.j2 @@ -1,25 +1,27 @@ # {{ nginx_frontend_domain }} — HTTPS frontend for EU server # Managed by Ansible nginx_frontend role # +# Serves Raven-subscribe and acts as fallback dest for XHTTP Reality probes. +# XHTTP VPN traffic goes through nginx stream SNI routing on :443 directly. +# # Routes: # /sub/, /c/ → Raven-subscribe (127.0.0.1:{{ nginx_frontend_raven_port }}) # /api/ → Raven-subscribe admin API # /health → Raven-subscribe health check -# {{ nginx_frontend_xhttp_path }} → Xray XHTTP (127.0.0.1:{{ nginx_frontend_xhttp_port }}) - -# ── Redirect HTTP → HTTPS ──────────────────────────────────────────────────── -server { - listen 80; - server_name {{ nginx_frontend_domain }}; - return 301 https://$host$request_uri; -} +# / → Default page (landing / 404) # ── HTTPS ───────────────────────────────────────────────────────────────────── server { - listen {{ nginx_frontend_listen_port }} ssl; + # proxy_protocol: accepts PROXY protocol v2 header sent by nginx stream on :443 + # This is required because stream block sends proxy_protocol on to all upstreams. + listen {{ nginx_frontend_listen_port }} ssl proxy_protocol; http2 on; server_name {{ nginx_frontend_domain }}; + # Use real client IP from PROXY protocol header (set by nginx stream) + real_ip_header proxy_protocol; + set_real_ip_from 127.0.0.1; + ssl_certificate /etc/letsencrypt/live/{{ nginx_frontend_domain }}/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/{{ nginx_frontend_domain }}/privkey.pem; include /etc/letsencrypt/options-ssl-nginx.conf; @@ -29,8 +31,8 @@ server { location ~ ^/(sub|c)/ { proxy_pass http://127.0.0.1:{{ nginx_frontend_raven_port }}; proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $proxy_protocol_addr; + proxy_set_header X-Forwarded-For $proxy_protocol_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 30s; proxy_connect_timeout 5s; @@ -40,8 +42,8 @@ server { location /api/ { proxy_pass http://127.0.0.1:{{ nginx_frontend_raven_port }}; proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $proxy_protocol_addr; + proxy_set_header X-Forwarded-For $proxy_protocol_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 30s; proxy_connect_timeout 5s; @@ -55,20 +57,6 @@ server { proxy_connect_timeout 5s; } - # ── Xray XHTTP ─────────────────────────────────────────────────────────── - location {{ nginx_frontend_xhttp_path }} { - proxy_pass http://127.0.0.1:{{ nginx_frontend_xhttp_port }}; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection ""; - proxy_read_timeout 3600s; - proxy_connect_timeout 5s; - proxy_buffering off; - } - # ── Everything else → 404 ──────────────────────────────────────────────── location / { return 404; diff --git a/roles/nginx_frontend/templates/nginx/stream.conf.j2 b/roles/nginx_frontend/templates/nginx/stream.conf.j2 index c7de9a6..e9496c2 100644 --- a/roles/nginx_frontend/templates/nginx/stream.conf.j2 +++ b/roles/nginx_frontend/templates/nginx/stream.conf.j2 @@ -1,15 +1,40 @@ -# {{ nginx_frontend_domain }} — nginx stream TCP relay for Xray VLESS Reality +# {{ nginx_frontend_domain }} — nginx stream SNI routing # Managed by Ansible nginx_frontend role -# Proxies: external:{{ nginx_frontend_reality_port }} → 127.0.0.1:443 (Xray) +# +# Reads SNI from TLS ClientHello (ssl_preread) and routes TCP: +# {{ nginx_frontend_stream_xhttp_sni }} → Xray XHTTP+Reality (127.0.0.1:{{ nginx_frontend_stream_xhttp_port }}) +# {{ nginx_frontend_domain }} → nginx HTTPS (127.0.0.1:{{ nginx_frontend_listen_port }}) +# default (other SNI) → Xray VLESS Reality (127.0.0.1:{{ nginx_frontend_stream_reality_port }}) +# # This file is included inside stream {} block in nginx.conf -upstream xray_reality_local { - server 127.0.0.1:443; +map $ssl_preread_server_name $sni_backend { + {{ nginx_frontend_stream_xhttp_sni }} xhttp_reality; + {{ nginx_frontend_domain }} nginx_https; + default vless_reality; } +upstream xhttp_reality { + server 127.0.0.1:{{ nginx_frontend_stream_xhttp_port }}; +} + +upstream nginx_https { + server 127.0.0.1:{{ nginx_frontend_listen_port }}; +} + +upstream vless_reality { + server 127.0.0.1:{{ nginx_frontend_stream_reality_port }}; +} + +# Routes all SNI backends through one server block. +# PROXY protocol is sent to ALL upstreams so each backend knows the real client IP: +# - Xray inbounds: accept via "xver": 2 in realitySettings +# - nginx HTTPS: accept via "proxy_protocol on" in http server block server { - listen {{ nginx_frontend_reality_port }}; - proxy_pass xray_reality_local; + listen 443 reuseport; + proxy_pass $sni_backend; + ssl_preread on; + proxy_protocol on; proxy_connect_timeout 10s; proxy_timeout 600s; } diff --git a/roles/raven_subscribe/defaults/main.yml b/roles/raven_subscribe/defaults/main.yml index aae8e04..93056ef 100644 --- a/roles/raven_subscribe/defaults/main.yml +++ b/roles/raven_subscribe/defaults/main.yml @@ -6,7 +6,7 @@ raven_subscribe_config_dir: "/etc/xray-subscription" raven_subscribe_db_dir: "/var/lib/xray-subscription" raven_subscribe_service_name: "xray-subscription" -raven_subscribe_listen_addr: ":8080" +raven_subscribe_listen_addr: "127.0.0.1:8080" raven_subscribe_sync_interval_seconds: 60 raven_subscribe_rate_limit_sub_per_min: 60 raven_subscribe_rate_limit_admin_per_min: 30 @@ -27,11 +27,13 @@ raven_subscribe_xray_api_addr: "127.0.0.1:10085" raven_subscribe_xray_config_dir: "/etc/xray/config.d" # Per-inbound host overrides. Falls back to raven_subscribe_server_host when tag not listed. -# Set in secrets.yml. Empty = all inbounds use server_host. +# Set in secrets.yml. With SNI routing both protocols are on :443 behind nginx stream. +# Example: {"vless-reality-in": "zirgate.com", "vless-xhttp-in": "zirgate.com"} raven_subscribe_inbound_hosts: {} # Per-inbound port overrides. Falls back to inbound's own port when tag not listed. -# Set in secrets.yml. Example: {"vless-reality-in": 8444} for TCP relay on RU. +# Set in secrets.yml. With SNI routing both protocols share port 443. +# Example: {"vless-reality-in": 443, "vless-xhttp-in": 443} raven_subscribe_inbound_ports: {} # Path to sing-box config file. Leave empty to disable sing-box sync. diff --git a/roles/relay/defaults/main.yml b/roles/relay/defaults/main.yml index 3a8d131..44cbdac 100644 --- a/roles/relay/defaults/main.yml +++ b/roles/relay/defaults/main.yml @@ -1,11 +1,14 @@ --- -# Relay role — nginx reverse proxy on RU VPS -# Domain layout example: -# example.com A → RU VPS IP (static stub site) -# my.example.com A → RU VPS IP (relay → Raven subscriptions + API) +# Relay role — nginx SNI routing + reverse proxy on RU VPS # -# EU server (managed by nginx_frontend role, not this role): -# media.example.com A → EU VPS IP (nginx_frontend → Xray XHTTP) +# Port 443 is handled by nginx stream with ssl_preread (SNI routing): +# my.example.com / example.com → local nginx HTTPS (:8443) for subscriptions + stub +# everything else (VPN SNIs) → EU server :443 (SNI routing on EU side) +# +# Domain layout: +# example.com A → RU VPS IP (stub site, via SNI → local HTTPS) +# my.example.com A → RU VPS IP (Raven relay, via SNI → local HTTPS) +# media.example.com A → EU VPS IP (nginx_frontend, not this role) # ── Domain ─────────────────────────────────────────────────────────────────── relay_domain: "example.com" # Set to your RU VPS domain @@ -15,16 +18,19 @@ relay_sub_my: "my.{{ relay_domain }}" # Raven-subscribe relay (RU VPS) # Set in secrets.yml # relay_upstream_host: "1.2.3.4" # EU server IP address -# Port where nginx_frontend listens on EU server (Raven-subscribe is behind it) -# Must match nginx_frontend_listen_port (default: 8443, NOT 443 which is taken by Xray) +# Port where nginx_frontend SNI stream listens on EU server (both protocols) +relay_upstream_port: 443 + +# Port where nginx_frontend HTTPS listens on EU server (Raven-subscribe behind it) relay_upstream_raven_port: 8443 -# ── TCP stream relay (VLESS Reality) ───────────────────────────────────────── -# Proxies raw TCP on relay_stream_port → EU server:relay_upstream_xray_port -# Clients connect to example.com:relay_stream_port instead of EU IP directly. + +# ── SNI stream routing on :443 ─────────────────────────────────────────────── +# nginx stream with ssl_preread reads SNI from TLS ClientHello and routes: +# relay_domain / relay_sub_my → local nginx HTTPS (relay_https_port) +# everything else → EU server (relay_upstream_port) relay_stream_enabled: true -relay_stream_port: 8444 # Listening port on RU server (must be free) -relay_upstream_xray_port: 8445 # nginx_frontend Reality stream port on EU server +relay_https_port: 8443 # Local HTTPS port (behind stream, stub + Raven relay) # ── nginx ───────────────────────────────────────────────────────────────────── relay_nginx_user: "www-data" diff --git a/roles/relay/inventory.ini b/roles/relay/inventory.ini index af0c8cf..cdaf016 100644 --- a/roles/relay/inventory.ini +++ b/roles/relay/inventory.ini @@ -1,2 +1,2 @@ [relay] -relay ansible_host=RU_VPS_IP ansible_user=deploy +relay ansible_host=195.19.92.182 ansible_user=deploy ansible_port=2200 ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_ssh_host_key_checking=false diff --git a/roles/relay/templates/nginx/https.conf.j2 b/roles/relay/templates/nginx/https.conf.j2 index 2905c55..9932da9 100644 --- a/roles/relay/templates/nginx/https.conf.j2 +++ b/roles/relay/templates/nginx/https.conf.j2 @@ -1,16 +1,12 @@ -# {{ relay_domain }} — HTTPS relay config +# {{ relay_domain }} — HTTPS config (behind stream SNI routing) # Managed by Ansible relay role - -# ── Redirect HTTP → HTTPS ──────────────────────────────────────────────────── -server { - listen 80; - server_name {{ relay_domain }} {{ relay_sub_my }}; - return 301 https://$host$request_uri; -} +# +# Listens on {{ relay_https_port }} (NOT 443 — port 443 is used by nginx stream SNI routing). +# Stream routes {{ relay_domain }} and {{ relay_sub_my }} SNI here. # ── {{ relay_domain }} — stub site ────────────────────────────────────────────────── server { - listen 443 ssl; + listen {{ relay_https_port }} ssl; http2 on; server_name {{ relay_domain }}; @@ -29,7 +25,7 @@ server { # ── {{ relay_sub_my }} — Raven-subscribe relay ─────────────────────────────────── server { - listen 443 ssl; + listen {{ relay_https_port }} ssl; http2 on; server_name {{ relay_sub_my }}; @@ -49,4 +45,3 @@ server { proxy_connect_timeout 10s; } } - diff --git a/roles/relay/templates/nginx/stream.conf.j2 b/roles/relay/templates/nginx/stream.conf.j2 index 4e7ba87..08d8fae 100644 --- a/roles/relay/templates/nginx/stream.conf.j2 +++ b/roles/relay/templates/nginx/stream.conf.j2 @@ -1,15 +1,33 @@ -# {{ relay_domain }} — nginx stream TCP relay +# {{ relay_domain }} — nginx stream SNI routing # Managed by Ansible relay role -# Proxies VLESS Reality TCP traffic: RU:{{ relay_stream_port }} → EU:{{ relay_upstream_xray_port }} +# +# Reads SNI from TLS ClientHello (ssl_preread) and routes TCP: +# {{ relay_domain }} → local nginx HTTPS (127.0.0.1:{{ relay_https_port }}) for stub site +# {{ relay_sub_my }} → local nginx HTTPS (127.0.0.1:{{ relay_https_port }}) for Raven relay +# everything else (VPN) → EU server ({{ relay_upstream_host }}:{{ relay_upstream_port }}) +# # This file is included inside stream {} block in nginx.conf -upstream xray_reality { - server {{ relay_upstream_host }}:{{ relay_upstream_xray_port }}; +{% if relay_stream_enabled %} +map $ssl_preread_server_name $relay_backend { + {{ relay_domain }} local_https; + {{ relay_sub_my }} local_https; + default eu_vpn; +} + +upstream local_https { + server 127.0.0.1:{{ relay_https_port }}; +} + +upstream eu_vpn { + server {{ relay_upstream_host }}:{{ relay_upstream_port }}; } server { - listen {{ relay_stream_port }}; - proxy_pass xray_reality; + listen 443 reuseport; + proxy_pass $relay_backend; + ssl_preread on; proxy_connect_timeout 10s; proxy_timeout 600s; } +{% endif %} diff --git a/roles/role_monitoring.yml b/roles/role_monitoring.yml new file mode 100644 index 0000000..8eba6b9 --- /dev/null +++ b/roles/role_monitoring.yml @@ -0,0 +1,15 @@ +--- +# Deploy monitoring stack +# EU (vm_my_srv): node_exporter + xray-exporter + xray-stats-exporter + VictoriaMetrics + Grafana +# RU (vm_my_ru): node_exporter only (scraped by VictoriaMetrics via WireGuard 10.10.0.2:9100) +# +# Access Grafana: ssh -L 13000:127.0.0.1:13000 user@eu-server +# then open http://localhost:13000 + +- name: Deploy monitoring stack + hosts: + - vm_my_srv + - vm_my_ru + become: true + roles: + - role: monitoring diff --git a/roles/role_wireguard.yml b/roles/role_wireguard.yml new file mode 100644 index 0000000..6f43e2d --- /dev/null +++ b/roles/role_wireguard.yml @@ -0,0 +1,21 @@ +--- +# Deploy WireGuard mesh between EU and RU servers +# Usage: +# ansible-playbook roles/role_wireguard.yml -i roles/hosts.yml --vault-password-file vault_password.txt +# +# Tags: +# wireguard_eu — configure wg0 on EU (install, config) +# wireguard_ru — configure wg0 on RU (install, config) + +- name: Deploy WireGuard + hosts: + - vm_my_srv + - vm_my_ru + become: true + + vars_files: + - wireguard/defaults/main.yml + - wireguard/defaults/secrets.yml + + roles: + - role: wireguard diff --git a/roles/wireguard/defaults/main.yml b/roles/wireguard/defaults/main.yml new file mode 100644 index 0000000..ab0ff1a --- /dev/null +++ b/roles/wireguard/defaults/main.yml @@ -0,0 +1,13 @@ +--- +wg_interface: wg0 +wg_port: 51820 # UDP port on EU +wg_eu_ip: "10.10.0.1" # EU WireGuard IP +wg_ru_ip: "10.10.0.2" # RU WireGuard IP +wg_network: "10.10.0.0/24" +wg_config_dir: "/etc/wireguard" +wg_config_path: "/etc/wireguard/{{ wg_interface }}.conf" + +# wg_eu_private_key: set in secrets.yml (ansible-vault) +# wg_eu_public_key: set in secrets.yml (ansible-vault) +# wg_ru_private_key: set in secrets.yml (ansible-vault) +# wg_ru_public_key: set in secrets.yml (ansible-vault) diff --git a/roles/wireguard/defaults/secrets.yml.example b/roles/wireguard/defaults/secrets.yml.example new file mode 100644 index 0000000..630c783 --- /dev/null +++ b/roles/wireguard/defaults/secrets.yml.example @@ -0,0 +1,14 @@ +--- +# Copy to secrets.yml and encrypt: +# ansible-vault encrypt secrets.yml --vault-password-file vault_password.txt +# +# Generate key pair (run twice — for EU and RU): +# wg genkey | tee /tmp/wg_private.key | wg pubkey > /tmp/wg_public.key +# cat /tmp/wg_private.key # → wg_XX_private_key +# cat /tmp/wg_public.key # → wg_XX_public_key +# rm /tmp/wg_private.key /tmp/wg_public.key + +wg_eu_private_key: "CHANGE_ME" +wg_eu_public_key: "CHANGE_ME" +wg_ru_private_key: "CHANGE_ME" +wg_ru_public_key: "CHANGE_ME" diff --git a/roles/wireguard/handlers/main.yml b/roles/wireguard/handlers/main.yml new file mode 100644 index 0000000..770adc1 --- /dev/null +++ b/roles/wireguard/handlers/main.yml @@ -0,0 +1,7 @@ +--- +- name: Restart wg0 + ansible.builtin.systemd: + name: "wg-quick@wg0" + state: restarted + daemon_reload: true + when: ansible_facts['service_mgr'] == "systemd" diff --git a/roles/wireguard/tasks/eu.yml b/roles/wireguard/tasks/eu.yml new file mode 100644 index 0000000..0391974 --- /dev/null +++ b/roles/wireguard/tasks/eu.yml @@ -0,0 +1,32 @@ +--- +- name: Install wireguard-tools + ansible.builtin.apt: + name: wireguard-tools + state: present + update_cache: false + +- name: Ensure wireguard config directory exists + ansible.builtin.file: + path: "{{ wg_config_dir }}" + state: directory + owner: root + group: root + mode: "0700" + +- name: Deploy wg0 config on EU + ansible.builtin.template: + src: wg0-eu.conf.j2 + dest: "{{ wg_config_path }}" + owner: root + group: root + mode: "0600" + notify: Restart wg0 + +- name: Enable and start wg-quick@wg0 on EU + ansible.builtin.systemd: + name: "wg-quick@{{ wg_interface }}" + enabled: true + state: started + daemon_reload: true + +# No firewall configured on EU (iptables policy ACCEPT) — UDP {{ wg_port }} is open by default diff --git a/roles/wireguard/tasks/main.yml b/roles/wireguard/tasks/main.yml new file mode 100644 index 0000000..d296633 --- /dev/null +++ b/roles/wireguard/tasks/main.yml @@ -0,0 +1,15 @@ +--- +- name: Load wireguard secrets + ansible.builtin.include_vars: + file: "{{ role_path }}/defaults/secrets.yml" + tags: always + +- name: Configure WireGuard on EU + ansible.builtin.import_tasks: eu.yml + when: inventory_hostname == 'vm_my_srv' + tags: wireguard_eu + +- name: Configure WireGuard on RU + ansible.builtin.import_tasks: ru.yml + when: inventory_hostname == 'vm_my_ru' + tags: wireguard_ru diff --git a/roles/wireguard/tasks/ru.yml b/roles/wireguard/tasks/ru.yml new file mode 100644 index 0000000..900577e --- /dev/null +++ b/roles/wireguard/tasks/ru.yml @@ -0,0 +1,30 @@ +--- +- name: Install wireguard-tools + ansible.builtin.apt: + name: wireguard-tools + state: present + update_cache: false + +- name: Ensure wireguard config directory exists + ansible.builtin.file: + path: "{{ wg_config_dir }}" + state: directory + owner: root + group: root + mode: "0700" + +- name: Deploy wg0 config on RU + ansible.builtin.template: + src: wg0-ru.conf.j2 + dest: "{{ wg_config_path }}" + owner: root + group: root + mode: "0600" + notify: Restart wg0 + +- name: Enable and start wg-quick@wg0 on RU + ansible.builtin.systemd: + name: "wg-quick@{{ wg_interface }}" + enabled: true + state: started + daemon_reload: true diff --git a/roles/wireguard/templates/wg0-eu.conf.j2 b/roles/wireguard/templates/wg0-eu.conf.j2 new file mode 100644 index 0000000..3c31fac --- /dev/null +++ b/roles/wireguard/templates/wg0-eu.conf.j2 @@ -0,0 +1,10 @@ +[Interface] +Address = {{ wg_eu_ip }}/24 +PrivateKey = {{ wg_eu_private_key }} +ListenPort = {{ wg_port }} + +[Peer] +# RU server +PublicKey = {{ wg_ru_public_key }} +AllowedIPs = {{ wg_ru_ip }}/32 +PersistentKeepalive = 25 diff --git a/roles/wireguard/templates/wg0-ru.conf.j2 b/roles/wireguard/templates/wg0-ru.conf.j2 new file mode 100644 index 0000000..85e439e --- /dev/null +++ b/roles/wireguard/templates/wg0-ru.conf.j2 @@ -0,0 +1,10 @@ +[Interface] +Address = {{ wg_ru_ip }}/24 +PrivateKey = {{ wg_ru_private_key }} + +[Peer] +# EU server +PublicKey = {{ wg_eu_public_key }} +Endpoint = {{ hostvars['vm_my_srv']['ansible_host'] }}:{{ wg_port }} +AllowedIPs = {{ wg_eu_ip }}/32 +PersistentKeepalive = 25 diff --git a/roles/xray/defaults/main.yml b/roles/xray/defaults/main.yml index 976396d..d104a65 100644 --- a/roles/xray/defaults/main.yml +++ b/roles/xray/defaults/main.yml @@ -17,12 +17,14 @@ xray_log_dir: "/var/log/Xray" # Directory for Xray logs xray_bin_dir: "/usr/local/bin" # Directory for Xray binaries xray_nologin_shell: "/usr/sbin/nologin" # Shell for the Xray user (to prevent login) -# VLESS + XTLS-Reality Configuration +# VLESS + XTLS-Reality Configuration (TCP) +# Listens on localhost — nginx stream SNI routing on :443 forwards traffic here. xray_vless_tag: vless-reality-in # Tag for VLESS inbound -xray_vless_port: 443 # Port for incoming VLESS connections -xray_reality_dest: "askubuntu.com:443" # Destination for XTLS-Reality (usually a TLS website) -xray_reality_server_names: # List of domain names for SNI/ServerName (used in Reality) - - "askubuntu.com" # Replace with your actual domain or leave empty for random +xray_vless_port: 4443 # Local port (behind nginx stream SNI routing on :443) +xray_vless_listen: "127.0.0.1" # Bind to localhost only (nginx stream fronts :443) +xray_reality_dest: "askubuntu.com:443" # Destination for XTLS-Reality (camouflage domain) +xray_reality_server_names: # SNI names for Reality TCP (must match dest) + - "askubuntu.com" # VLESS Encryption — server inbound decryption string (Xray-core >= 25.x, PR #5067). # "none" — standard VLESS, no extra encryption layer, compatible with all clients. @@ -65,8 +67,6 @@ xray_blocked_domains: # List of domains to block - "geosite:category-ads-all" - "geosite:category-ads" - "geosite:category-public-tracker" - - "geosite:google-ads" - - "geosite:spotify-ads" # Xray Reality api configuration xray_api: @@ -82,15 +82,22 @@ xray_api: protocol: "dokodemo-door" # Protocol for API (dokodemo-door)" tag: "api-inbound" -# Xray XHTTP configuration +# Xray XHTTP + Reality configuration +# Listens on localhost — nginx stream SNI routing on :443 forwards traffic here. +# Uses separate Reality dest/serverNames from TCP Reality (different SNI for SNI-based routing). xray_xhttp: tag: "vless-xhttp-in" # Tag for XHTTP inbound - port: 2053 # Port for XHTTP inbound - network: "xhttp" # Network type for XHTTP (http or ws) + port: 2053 # Local port (behind nginx stream SNI routing on :443) + listen: "127.0.0.1" # Bind to localhost only (nginx stream fronts :443) + network: "xhttp" # Network type xhttpSettings: mode: "auto" path: "/api/v3/data-sync" - scMaxPacketSize: 50000 + scMaxPacketSize: 1048576 + reality: # XHTTP-specific Reality settings (separate from TCP Reality) + dest: "www.adobe.com:443" # Camouflage domain (TLS 1.3, H2, OCSP, Akamai CDN) + server_names: + - "www.adobe.com" ##### !!!!! This is a secret, do not share it publicly !!!!!#### @@ -108,7 +115,7 @@ xray_xhttp: # xray_users: # - id: "UUID_1" # Replace with your actual UUID # flow: "xtls-rprx-vision" -# email: "someEmailForIdentyfy" +# email: "someEmailForIdentyfy" # Non-empty recommended; if omitted, id is used (needed for per-user traffic stats) # - id: "UUID_2" # flow: "xtls-rprx-vision" # email: "someEmailForIdentyfy" diff --git a/roles/xray/templates/conf/inbounds/200-in-vless-reality.json.j2 b/roles/xray/templates/conf/inbounds/200-in-vless-reality.json.j2 index 1ec0d8d..44c6438 100644 --- a/roles/xray/templates/conf/inbounds/200-in-vless-reality.json.j2 +++ b/roles/xray/templates/conf/inbounds/200-in-vless-reality.json.j2 @@ -1,6 +1,7 @@ { "inbounds": [ { + "listen": "{{ xray_vless_listen | default('0.0.0.0') }}", "port": {{ xray_vless_port }}, "protocol": "vless", "tag": "{{ xray_vless_tag }}", @@ -11,7 +12,7 @@ { "id": "{{ user.id }}", "flow": "{{ 'xtls-rprx-vision' if _pq else user.flow }}", - "email": "{{ user.email | default('') }}", + "email": "{{ (user.email | default('') | trim) or user.id }}", "level": 0 }{{ "," if not loop.last }} {% endfor %} @@ -44,7 +45,8 @@ {% for short_id in xray_reality.short_id %} "{{ short_id }}"{{ "," if not loop.last }} {% endfor %} - ] + ], + "xver": 2 } }, "sniffing": { diff --git a/roles/xray/templates/conf/inbounds/210-in-xhttp.json.j2 b/roles/xray/templates/conf/inbounds/210-in-xhttp.json.j2 index 76941f2..01f6a24 100644 --- a/roles/xray/templates/conf/inbounds/210-in-xhttp.json.j2 +++ b/roles/xray/templates/conf/inbounds/210-in-xhttp.json.j2 @@ -1,9 +1,10 @@ { "inbounds": [ { + "listen": "{{ xray_xhttp.listen | default('0.0.0.0') }}", "port": {{ xray_xhttp.port }}, "protocol": "vless", - "tag": "{{ xray_common.inbound_tag.vless_xhttp_in }}", + "tag": "{{ xray_common.inbound_tag.vless_xhttp_in }}", "settings": { "clients": [ {% set _pq = xray_vless_decryption | default('none') != 'none' %} @@ -12,7 +13,7 @@ { "id": "{{ user.id }}", "flow": "{{ 'xtls-rprx-vision' if _pq else _flow }}", - "email": "{{ user.email | default('') }}", + "email": "{{ (user.email | default('') | trim) or user.id }}", "level": 0 }{{ "," if not loop.last }} {% endfor %} @@ -31,7 +32,7 @@ "streamSettings": { "network": "{{ xray_xhttp.network }}", "realitySettings": { - "dest": "{{ xray_reality_dest }}", + "dest": "{{ xray_xhttp.reality.dest | default(xray_reality_dest) }}", "privateKey": "{{ xray_reality.private_key }}", "spiderX": "{{ xray_reality.spiderX }}", {% set _seed = xray_reality.mldsa65_seed | default('') %} @@ -43,7 +44,8 @@ {% endif %} {% endif %} "serverNames": [ - {% for name in xray_reality_server_names %} + {% set _xhttp_sni = xray_xhttp.reality.server_names | default(xray_reality_server_names) %} + {% for name in _xhttp_sni %} "{{ name }}"{{ "," if not loop.last }} {% endfor %} ], @@ -53,15 +55,18 @@ {% endfor %} ], "show": false, - "xver": 0 + "xver": 2 }, "security": "reality", "xhttpSettings": { "mode": "{{ xray_xhttp.xhttpSettings.mode }}", "path": "{{ xray_xhttp.xhttpSettings.path }}", - "scMaxPacketSize": {{ xray_xhttp.xhttpSettings.scMaxPacketSize }} + "scMaxPacketSize": {{ xray_xhttp.xhttpSettings.scMaxPacketSize }}, + "extra": { + "xPaddingBytes": "100-1000" + } } } } ] -} \ No newline at end of file +} diff --git a/roles/xray/templates/conf/routing/400-routing.json.j2 b/roles/xray/templates/conf/routing/400-routing.json.j2 index 774f09f..cada260 100644 --- a/roles/xray/templates/conf/routing/400-routing.json.j2 +++ b/roles/xray/templates/conf/routing/400-routing.json.j2 @@ -1,6 +1,6 @@ { "routing": { - "domainStrategy": "IPIfNonMatch", + "domainStrategy": "AsIs", "rules": [ {% if xray_blocked_domains | length > 0 %} { diff --git a/roles/xray/templates/conf/users/230-in-xhttp-users.json.j2 b/roles/xray/templates/conf/users/230-in-xhttp-users.json.j2 index aad54ec..0379cb3 100644 --- a/roles/xray/templates/conf/users/230-in-xhttp-users.json.j2 +++ b/roles/xray/templates/conf/users/230-in-xhttp-users.json.j2 @@ -10,7 +10,7 @@ { "id": "{{ user.id }}", "flow": "{{ 'xtls-rprx-vision' if _pq else _flow }}", - "email": "{{ user.email | default('') }}", + "email": "{{ (user.email | default('') | trim) or user.id }}", "level": 0 }{{ "," if not loop.last }} {% endfor %} diff --git a/roles/xray/templates/conf/users/240-in-vless-users.json.j2 b/roles/xray/templates/conf/users/240-in-vless-users.json.j2 index e327dd2..6f72c6d 100644 --- a/roles/xray/templates/conf/users/240-in-vless-users.json.j2 +++ b/roles/xray/templates/conf/users/240-in-vless-users.json.j2 @@ -10,7 +10,7 @@ { "id": "{{ user.id }}", "flow": "{{ 'xtls-rprx-vision' if _pq else _flow }}", - "email": "{{ user.email | default('') }}", + "email": "{{ (user.email | default('') | trim) or user.id }}", "level": 0 }{{ "," if not loop.last }} {% endfor %} diff --git a/roles/xray/templates/xray.service.j2 b/roles/xray/templates/xray.service.j2 index 65f8409..18028c9 100644 --- a/roles/xray/templates/xray.service.j2 +++ b/roles/xray/templates/xray.service.j2 @@ -11,8 +11,10 @@ AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE NoNewPrivileges=true ExecStart={{ xray_install_dir }}/xray run -confdir {{ xray_config_dir }}config.d Restart=on-failure +RestartSec=3s RestartPreventExitStatus=23 -LimitNPROC=10000 +TimeoutStopSec=30s +LimitNPROC=2000 LimitNOFILE=1000000 [Install]