diff --git a/migration/mongosync_insights/PACKAGING.md b/migration/mongosync_insights/PACKAGING.md new file mode 100644 index 00000000..51bc5836 --- /dev/null +++ b/migration/mongosync_insights/PACKAGING.md @@ -0,0 +1,167 @@ +# Packaging Mongosync Insights as an RPM + +This guide explains how to build a **self-contained RPM** that can be installed on air-gapped (no internet) RHEL/CentOS machines. The RPM includes an embedded Python runtime and all dependencies — the target machine needs **nothing** pre-installed. + +## Build Prerequisites + +The **build machine** (not the target) needs: + +| Requirement | Install | +|---|---| +| Python 3.11+ | `yum install python3.11` or build from source | +| pip | Included with Python 3.11+ | +| Ruby + gem | `yum install ruby rubygems` | +| fpm | `gem install fpm` | +| rpmbuild | `yum install rpm-build` | + +> **Important:** Build on the **same or older** RHEL version as the target machine. PyInstaller binaries are glibc-version-specific — a binary built on RHEL 9 will not run on RHEL 8. When in doubt, build on the oldest target you need to support. + +## Building the RPM + +```bash +cd migration/mongosync_insights +./build_rpm.sh +``` + +The script will: + +1. Read the app version from `app_config.py` +2. Create a temporary Python virtual environment +3. Install all dependencies from `requirements.txt` +4. Run PyInstaller to produce a standalone directory bundle +5. Package everything into an RPM using `fpm` +6. Clean up the temporary venv and staging area + +Output: + +``` +dist/mongosync-insights--1.x86_64.rpm +``` + +## Installing on Target RHEL + +Copy the RPM to the target machine (USB, SCP, etc.) and install: + +```bash +sudo rpm -i mongosync-insights-0.8.0.18-1.x86_64.rpm +``` + +No internet access is required. No additional packages need to be installed. + +**Installed files:** + +| Path | Description | +|---|---| +| `/opt/mongosync-insights/` | Application directory (binary + bundled Python + deps) | +| `/usr/local/bin/mongosync-insights` | Wrapper script (adds the tool to `$PATH`) | +| `/usr/lib/systemd/system/mongosync-insights.service` | Systemd unit file | +| `/etc/mongosync-insights/env` | Configuration environment file | + +## Configuration + +Edit `/etc/mongosync-insights/env` to configure the application: + +```bash +# Listen on all interfaces (default: 127.0.0.1) +MI_HOST=0.0.0.0 + +# Change port (default: 3030) +MI_PORT=8080 + +# Pre-configure MongoDB connection string for live monitoring +MI_CONNECTION_STRING=mongodb+srv://user:pass@cluster.mongodb.net/ + +# Pre-configure mongosync progress endpoint +MI_PROGRESS_ENDPOINT_URL=host:port/api/v1/progress + +# Dashboard auto-refresh interval in seconds (default: 10) +MI_REFRESH_TIME=5 + +# Enable HTTPS +MI_SSL_ENABLED=true +MI_SSL_CERT=/etc/mongosync-insights/cert.pem +MI_SSL_KEY=/etc/mongosync-insights/key.pem + +# Log level (default: INFO) +LOG_LEVEL=DEBUG +``` + +See [CONFIGURATION.md](CONFIGURATION.md) for the full reference. + +## Running + +### Start the service + +```bash +sudo systemctl start mongosync-insights +``` + +### Enable on boot + +```bash +sudo systemctl enable mongosync-insights +``` + +### Check status + +```bash +sudo systemctl status mongosync-insights +``` + +### View logs + +```bash +journalctl -u mongosync-insights -f +``` + +### Run manually (without systemd) + +```bash +/opt/mongosync-insights/mongosync-insights +``` + +Or with environment variables: + +```bash +MI_HOST=0.0.0.0 MI_PORT=8080 /opt/mongosync-insights/mongosync-insights +``` + +## Uninstalling + +```bash +sudo rpm -e mongosync-insights +``` + +This stops the service, removes all installed files, and reloads systemd. + +## Upgrading + +```bash +sudo rpm -U mongosync-insights--1.x86_64.rpm +``` + +The configuration file at `/etc/mongosync-insights/env` is preserved during upgrades. + +## Architecture Notes + +- The RPM is **architecture-specific** (`x86_64` or `aarch64`). Build on the same architecture as the target. +- The RPM is **glibc-version-specific**. Build on the same or older RHEL version as the target. +- The bundled `certifi` CA certificates are frozen at build time. Rebuild the RPM to update them. +- No `libmagic` / `file-libs` system library is required — file type detection uses pure-Python magic-byte inspection. + +## Troubleshooting + +### "GLIBC_x.xx not found" on the target machine + +The RPM was built on a newer OS than the target. Rebuild on the same or older RHEL version. + +### Service fails to start + +Check the journal for details: +```bash +journalctl -u mongosync-insights --no-pager -n 50 +``` + +Common causes: +- Port already in use — change `MI_PORT` in `/etc/mongosync-insights/env` +- SSL certificate not found — verify paths in the env file diff --git a/migration/mongosync_insights/README.md b/migration/mongosync_insights/README.md index f6da1ac0..8d628bd5 100644 --- a/migration/mongosync_insights/README.md +++ b/migration/mongosync_insights/README.md @@ -21,35 +21,24 @@ Mongosync Insights provides four main capabilities: ## Installation -### 1. Download the Tool +### Option A: RPM Installation (Air-Gapped / Offline Environments) -Download or clone the Mongosync Insights folder from this repository. - -### 2. Install System Dependencies - -Before installing Python packages, ensure **libmagic** is installed on your system (required for file type detection): +For machines without internet access, a self-contained RPM is available that bundles the Python runtime and all dependencies. No additional packages need to be installed on the target. -**macOS:** ```bash -brew install libmagic +sudo rpm -i mongosync-insights-.x86_64.rpm +sudo systemctl start mongosync-insights ``` -**Ubuntu/Debian:** -```bash -sudo apt-get update -sudo apt-get install libmagic1 -``` +See **[PACKAGING.md](PACKAGING.md)** for how to build the RPM, configure, and run the service. -**Red Hat/CentOS/Fedora:** -```bash -sudo yum install file-libs -``` +### Option B: Manual Installation (Development / Connected Environments) -**Windows:** -- Download and install from [https://github.com/nscaife/file-windows](https://github.com/nscaife/file-windows) -- Or use: `pip install python-magic-bin` (includes precompiled libmagic) +#### 1. Download the Tool + +Download or clone the Mongosync Insights folder from this repository. -### 3. Install Python Dependencies +#### 2. Install Python Dependencies Navigate to the directory containing the Python script and the `requirements.txt` file: @@ -243,6 +232,7 @@ python3 mongosync_insights.py For detailed guides, see: +- **[PACKAGING.md](PACKAGING.md)** - Build a self-contained RPM for offline/air-gapped deployment - **[CONFIGURATION.md](CONFIGURATION.md)** - Complete environment variables reference, configuration options, and MongoDB connection pooling - **[HTTPS_SETUP.md](HTTPS_SETUP.md)** - Enable HTTPS/SSL for secure deployments - **[VALIDATION.md](VALIDATION.md)** - Connection string validation, sanitization, and error handling diff --git a/migration/mongosync_insights/app_config.py b/migration/mongosync_insights/app_config.py index 8d977c20..5c73f799 100644 --- a/migration/mongosync_insights/app_config.py +++ b/migration/mongosync_insights/app_config.py @@ -28,6 +28,8 @@ MAX_FILE_SIZE = int(os.getenv('MI_MAX_FILE_SIZE', str(10 * 1024 * 1024 * 1024))) # 10GB default ALLOWED_EXTENSIONS = {'.log', '.json', '.out', '.gz', '.zip', '.bz2', '.tar.gz', '.tgz', '.tar.bz2'} ALLOWED_MIME_TYPES = [ + 'text/plain', + 'application/json', 'application/x-ndjson', 'application/gzip', 'application/x-gzip', 'application/zip', 'application/x-zip-compressed', diff --git a/migration/mongosync_insights/build_rpm.sh b/migration/mongosync_insights/build_rpm.sh new file mode 100755 index 00000000..93ac8e8b --- /dev/null +++ b/migration/mongosync_insights/build_rpm.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +# ============================================================================= +# build_rpm.sh — Build a self-contained RPM for Mongosync Insights +# +# The RPM bundles the Python interpreter, all dependencies, templates, images, +# and JSON configs via PyInstaller so the target machine needs nothing extra. +# +# Prerequisites (build machine only): +# - Python 3.11+ +# - pip +# - ruby + gem (for fpm) +# - fpm: gem install fpm +# - rpmbuild: yum install rpm-build (fpm uses it under the hood) +# +# Usage: +# cd migration/mongosync_insights +# chmod +x build_rpm.sh +# ./build_rpm.sh +# +# Output: +# ./dist/mongosync-insights--1.x86_64.rpm +# ============================================================================= +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +# --------------------------------------------------------------------------- +# 1. Read version from app_config.py +# --------------------------------------------------------------------------- +APP_VERSION=$(python3 -c " +import re, pathlib +m = re.search(r'APP_VERSION\s*=\s*\"([^\"]+)\"', pathlib.Path('app_config.py').read_text()) +print(m.group(1)) +") +echo "==> Building Mongosync Insights v${APP_VERSION}" + +# --------------------------------------------------------------------------- +# 2. Check build prerequisites +# --------------------------------------------------------------------------- +for cmd in python3 pip3 fpm rpmbuild; do + if ! command -v "$cmd" &>/dev/null; then + echo "ERROR: '$cmd' is required but not found in PATH." >&2 + if [[ "$cmd" == "fpm" ]]; then + echo " Install it with: gem install fpm" >&2 + fi + exit 1 + fi +done + +PYTHON_VERSION=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') +if python3 -c "import sys; sys.exit(0 if sys.version_info >= (3,11) else 1)" 2>/dev/null; then + echo "==> Python ${PYTHON_VERSION} detected — OK" +else + echo "ERROR: Python 3.11+ is required (found ${PYTHON_VERSION})." >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# 3. Create a clean virtual environment +# --------------------------------------------------------------------------- +VENV_DIR="$SCRIPT_DIR/.build_venv" +echo "==> Creating virtual environment at ${VENV_DIR}" +rm -rf "$VENV_DIR" +python3 -m venv "$VENV_DIR" +# shellcheck disable=SC1091 +source "$VENV_DIR/bin/activate" + +pip install --upgrade pip setuptools wheel +pip install -r requirements.txt +pip install pyinstaller + +# --------------------------------------------------------------------------- +# 4. Run PyInstaller +# --------------------------------------------------------------------------- +echo "==> Running PyInstaller" +pyinstaller --clean --noconfirm mongosync_insights.spec + +DIST_DIR="$SCRIPT_DIR/dist/mongosync-insights" +if [[ ! -d "$DIST_DIR" ]]; then + echo "ERROR: PyInstaller output directory not found at ${DIST_DIR}" >&2 + exit 1 +fi +echo "==> PyInstaller bundle created at ${DIST_DIR}" + +# --------------------------------------------------------------------------- +# 5. Prepare RPM staging area +# --------------------------------------------------------------------------- +STAGING="$SCRIPT_DIR/dist/rpm-staging" +rm -rf "$STAGING" + +INSTALL_PREFIX="/opt/mongosync-insights" + +# Application files +mkdir -p "$STAGING/$INSTALL_PREFIX" +cp -a "$DIST_DIR"/. "$STAGING/$INSTALL_PREFIX/" + +# Wrapper script +mkdir -p "$STAGING/usr/local/bin" +cat > "$STAGING/usr/local/bin/mongosync-insights" <<'WRAPPER' +#!/usr/bin/env bash +exec /opt/mongosync-insights/mongosync-insights "$@" +WRAPPER +chmod 755 "$STAGING/usr/local/bin/mongosync-insights" + +# Systemd service file +mkdir -p "$STAGING/usr/lib/systemd/system" +cp "$SCRIPT_DIR/mongosync-insights.service" "$STAGING/usr/lib/systemd/system/" + +# Default environment file +mkdir -p "$STAGING/etc/mongosync-insights" +cat > "$STAGING/etc/mongosync-insights/env" <<'ENVFILE' +# Mongosync Insights configuration +# Uncomment and edit the variables you need. +# See CONFIGURATION.md for the full reference. + +# MI_HOST=0.0.0.0 +# MI_PORT=3030 +# MI_CONNECTION_STRING=mongodb+srv://user:pass@cluster.mongodb.net/ +# MI_PROGRESS_ENDPOINT_URL=host:port/api/v1/progress +# MI_REFRESH_TIME=10 +# MI_SSL_ENABLED=false +# MI_SSL_CERT=/etc/mongosync-insights/cert.pem +# MI_SSL_KEY=/etc/mongosync-insights/key.pem +# LOG_LEVEL=INFO +ENVFILE + +# --------------------------------------------------------------------------- +# 6. Build the RPM with fpm +# --------------------------------------------------------------------------- +echo "==> Building RPM with fpm" + +# Post-install script: reload systemd +POST_INSTALL=$(mktemp) +cat > "$POST_INSTALL" <<'SCRIPT' +#!/bin/bash +systemctl daemon-reload 2>/dev/null || true +echo "" +echo "Mongosync Insights installed to /opt/mongosync-insights/" +echo "" +echo " Configure: /etc/mongosync-insights/env" +echo " Start: systemctl start mongosync-insights" +echo " Status: systemctl status mongosync-insights" +echo " Logs: journalctl -u mongosync-insights -f" +echo "" +SCRIPT + +# Pre-uninstall script: stop the service +PRE_UNINSTALL=$(mktemp) +cat > "$PRE_UNINSTALL" <<'SCRIPT' +#!/bin/bash +systemctl stop mongosync-insights 2>/dev/null || true +systemctl disable mongosync-insights 2>/dev/null || true +SCRIPT + +# Post-uninstall script: reload systemd +POST_UNINSTALL=$(mktemp) +cat > "$POST_UNINSTALL" <<'SCRIPT' +#!/bin/bash +systemctl daemon-reload 2>/dev/null || true +SCRIPT + +RPM_OUTPUT="$SCRIPT_DIR/dist" + +# Remove .build-id symlinks that PyInstaller copies from system libraries. +# These conflict with RHEL system packages (ncurses-libs, zlib, openssl-libs, etc.) +find "$STAGING" -path '*/.build-id' -type d -exec rm -rf {} + 2>/dev/null || true + +fpm \ + -s dir \ + -t rpm \ + -n mongosync-insights \ + -v "$APP_VERSION" \ + --iteration 1 \ + --license "Apache-2.0" \ + --vendor "MongoDB Support" \ + --description "Mongosync Insights — MongoDB migration monitoring dashboard" \ + --url "https://github.com/mongodb/support-tools" \ + --architecture native \ + --rpm-auto-add-directories \ + --rpm-rpmbuild-define '_build_id_links none' \ + --exclude '**/.build-id' \ + --after-install "$POST_INSTALL" \ + --before-remove "$PRE_UNINSTALL" \ + --after-remove "$POST_UNINSTALL" \ + --config-files /etc/mongosync-insights/env \ + --package "$RPM_OUTPUT" \ + -C "$STAGING" \ + . + +rm -f "$POST_INSTALL" "$PRE_UNINSTALL" "$POST_UNINSTALL" + +# --------------------------------------------------------------------------- +# 7. Clean up +# --------------------------------------------------------------------------- +deactivate 2>/dev/null || true +rm -rf "$VENV_DIR" "$STAGING" + +RPM_FILE=$(ls -1 "$RPM_OUTPUT"/mongosync-insights-*.rpm 2>/dev/null | head -1) +if [[ -n "$RPM_FILE" ]]; then + echo "" + echo "==> RPM built successfully:" + echo " ${RPM_FILE}" + echo "" + echo " Install on target: sudo rpm -i $(basename "$RPM_FILE")" + echo "" +else + echo "ERROR: RPM file not found in ${RPM_OUTPUT}" >&2 + exit 1 +fi diff --git a/migration/mongosync_insights/mongosync-insights.service b/migration/mongosync_insights/mongosync-insights.service new file mode 100644 index 00000000..863dd441 --- /dev/null +++ b/migration/mongosync_insights/mongosync-insights.service @@ -0,0 +1,24 @@ +[Unit] +Description=Mongosync Insights - MongoDB Migration Monitoring Dashboard +After=network.target + +[Service] +Type=simple +ExecStart=/opt/mongosync-insights/mongosync-insights +WorkingDirectory=/opt/mongosync-insights +EnvironmentFile=-/etc/mongosync-insights/env +Restart=on-failure +RestartSec=5 + +# Security hardening +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/opt/mongosync-insights + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=mongosync-insights + +[Install] +WantedBy=multi-user.target diff --git a/migration/mongosync_insights/mongosync_insights.py b/migration/mongosync_insights/mongosync_insights.py index b81c6ee9..7494cc45 100644 --- a/migration/mongosync_insights/mongosync_insights.py +++ b/migration/mongosync_insights/mongosync_insights.py @@ -1,4 +1,6 @@ import logging +import sys +import os from flask import Flask, render_template, request, make_response from mongosync_plot_logs import upload_file from mongosync_plot_metadata import plotMetrics, gatherMetrics, gatherPartitionsMetrics, gatherEndpointMetrics @@ -32,8 +34,17 @@ def _store_session_data(new_data): # Setup logging logger = setup_logging() +# Resolve base path for templates & static assets (supports PyInstaller bundles) +if getattr(sys, 'frozen', False): + _base_path = sys._MEIPASS +else: + _base_path = os.path.dirname(os.path.abspath(__file__)) + # Create a Flask app -app = Flask(__name__, static_folder='images', static_url_path='/images') +app = Flask(__name__, + template_folder=os.path.join(_base_path, 'templates'), + static_folder=os.path.join(_base_path, 'images'), + static_url_path='/images') # Configure Flask for file uploads app.config['MAX_CONTENT_LENGTH'] = MAX_FILE_SIZE diff --git a/migration/mongosync_insights/mongosync_insights.spec b/migration/mongosync_insights/mongosync_insights.spec new file mode 100644 index 00000000..f94fb870 --- /dev/null +++ b/migration/mongosync_insights/mongosync_insights.spec @@ -0,0 +1,81 @@ +# -*- mode: python ; coding: utf-8 -*- +""" +PyInstaller spec file for Mongosync Insights. +Produces a one-directory bundle that includes the Python interpreter, +all dependencies, templates, images, and JSON configuration files. + +Usage (from the mongosync_insights directory): + pyinstaller mongosync_insights.spec +""" +import os +import importlib + +block_cipher = None + +# Resolve certifi CA bundle so TLS connections work in the frozen build +certifi_path = os.path.join(os.path.dirname(importlib.import_module('certifi').__file__), 'cacert.pem') + +a = Analysis( + ['mongosync_insights.py'], + pathex=[], + binaries=[], + datas=[ + # Flask HTML templates + ('templates', 'templates'), + # Static images served by Flask + ('images', 'images'), + # Runtime JSON configuration files + ('error_patterns.json', '.'), + ('mongosync_metrics.json', '.'), + # certifi CA bundle for pymongo TLS + (certifi_path, 'certifi'), + ], + hiddenimports=[ + 'mongosync_plot_logs', + 'mongosync_plot_metadata', + 'mongosync_plot_prometheus_metrics', + 'mongosync_plot_utils', + 'migration_verifier', + 'file_decompressor', + 'app_config', + 'connection_validator', + 'plotly', + 'engineio.async_drivers.threading', + 'dns.resolver', + 'dns.rdatatype', + ], + hookspath=[], + hooksconfig={}, + runtime_hooks=[], + excludes=[], + win_no_prefer_redirects=False, + win_private_assemblies=False, + cipher=block_cipher, + noarchive=False, +) + +pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher) + +exe = EXE( + pyz, + a.scripts, + [], + exclude_binaries=True, + name='mongosync-insights', + debug=False, + bootloader_ignore_signals=False, + strip=False, + upx=True, + console=True, +) + +coll = COLLECT( + exe, + a.binaries, + a.zipfiles, + a.datas, + strip=False, + upx=True, + upx_exclude=[], + name='mongosync-insights', +) diff --git a/migration/mongosync_insights/mongosync_plot_logs.py b/migration/mongosync_insights/mongosync_plot_logs.py index 09414b3c..42d97c04 100644 --- a/migration/mongosync_insights/mongosync_plot_logs.py +++ b/migration/mongosync_insights/mongosync_plot_logs.py @@ -9,13 +9,42 @@ import re import logging import os -import magic +import mimetypes from werkzeug.utils import secure_filename from mongosync_plot_utils import format_byte_size, convert_bytes from app_config import MAX_FILE_SIZE, ALLOWED_EXTENSIONS, ALLOWED_MIME_TYPES, load_error_patterns, classify_file_type from file_decompressor import decompress_file_classified, is_compressed_mime_type from mongosync_plot_prometheus_metrics import MetricsCollector, create_metrics_plots + +def detect_mime_type(file_sample: bytes, filename: str) -> str: + """ + Detect MIME type using magic bytes and file extension. + Pure-Python replacement for python-magic — no system libmagic needed. + """ + # Check magic bytes from the file header + if file_sample[:2] == b'\x1f\x8b': + return 'application/gzip' + if file_sample[:4] == b'PK\x03\x04': + return 'application/zip' + if file_sample[:3] == b'BZh': + return 'application/x-bzip2' + if len(file_sample) > 262 and file_sample[257:262] == b'ustar': + return 'application/x-tar' + + # Fall back to extension-based detection + mime_type, _ = mimetypes.guess_type(filename) + if mime_type: + return mime_type + + # If content looks like text, report it as text/plain + try: + file_sample.decode('utf-8') + return 'text/plain' + except UnicodeDecodeError: + return 'application/octet-stream' + + def upload_file(): # Use the centralized logging configuration logger = logging.getLogger(__name__) @@ -67,27 +96,19 @@ def upload_file(): error_title="File Too Large", error_message=f"File size ({actual_size_mb:.1f} MB) exceeds maximum allowed size ({max_size_mb:.1f} MB).") - # Check MIME type using python-magic - try: - mime = magic.Magic(mime=True) - file.seek(0) - # Read first 2KB for MIME detection (sufficient for most file types) - file_sample = file.read(2048) - file_mime_type = mime.from_buffer(file_sample) - file.seek(0) # Reset to beginning - - logger.info(f"Detected MIME type: {file_mime_type}") - - if file_mime_type not in ALLOWED_MIME_TYPES: - logger.error(f"Invalid MIME type: {file_mime_type}. Allowed: {ALLOWED_MIME_TYPES}") - return render_template('error.html', - error_title="Invalid File Type", - error_message=f"File MIME type '{file_mime_type}' is not allowed. Only JSON/text files are accepted. Detected type: {file_mime_type}") - except Exception as e: - logger.error(f"Error detecting MIME type: {e}") + # Detect MIME type using magic bytes and file extension (no libmagic needed) + file.seek(0) + file_sample = file.read(2048) + file_mime_type = detect_mime_type(file_sample, filename) + file.seek(0) + + logger.info(f"Detected MIME type: {file_mime_type}") + + if file_mime_type not in ALLOWED_MIME_TYPES: + logger.error(f"Invalid MIME type: {file_mime_type}. Allowed: {ALLOWED_MIME_TYPES}") return render_template('error.html', - error_title="File Validation Error", - error_message=f"Unable to validate file type: {str(e)}") + error_title="Invalid File Type", + error_message=f"File MIME type '{file_mime_type}' is not allowed. Only JSON/text files are accepted. Detected type: {file_mime_type}") logger.info(f"File validation passed: {filename} ({file_size} bytes, {file_ext}, MIME: {file_mime_type})") # Optimized single-pass log parsing with streaming approach diff --git a/migration/mongosync_insights/requirements.txt b/migration/mongosync_insights/requirements.txt index 644d84fe..a65e9cb2 100644 --- a/migration/mongosync_insights/requirements.txt +++ b/migration/mongosync_insights/requirements.txt @@ -21,9 +21,6 @@ dnspython==2.6.1 python-dateutil==2.8.2 six==1.16.0 -# File Processing -python-magic==0.4.27 - # HTTP Requests requests==2.32.3