My automated workflow - running on a headless Debian system

[5559175]

Through a lot of trial and error I think I have got quite a robust capture and analysis method for my Reolink CX810 IP camera (over RTSP). This works well in very low light and still produces colour video without needing to turn on IR lighting. I know there are other projects out there doing similar sorts of things but I couldn't find anything that did exactly what I wanted - most are geared up for "all sky" cameras it seems.

I am capturing first to a .ts file then when either the value of "$SECONDS" has elapsed or it receives a SIGTERM or SIGINT, the script "finalizes" the video into an mp4 container. This seems to help should there be any interruption of signal, jittery framerate from RTSP, or otherwise to avoid any sort of corruption/weirdness in the resulting video.

All of this is wrapped up in a very basic Flask webapp which Gemini wrote for me along with the scripts that I've tweaked a bit (I'm not a developer at all). It runs on my headless Debian NAS. More details of how I got MetDetPy to work nicely on my specific hardware are in this bug report

I can either just use the webapp to start/stop recording just the 4K video or alternatively I can run the auto-detect meteors "pipeline" which does in sequence: the stream recording, timelapse creation, 720p transcode, MetDetPy analysis and finally ClipToolkit clip creation. There is also an extra step inserted into the pipeline before running ClipToolkit which uses "sed" in the detections.json to swap the path of the 720p video used for analysis to the full 4K one to get better quality clips generated.

I have edited things since this original message and added some new functions to:

View a live stream of the camera via HTTP-FLV and a static image for easy viewing...the URLs are probably specific to the Reolink I would guess - I have had Gemini do this all for me and it suggested using a proxy inside Flask to workaround a CORS issue which the camera has which worked.

I can also now view the meteor clip folders (and clips) and there's a function to concatenate all clips together into one single MP4 file.

I have separate "STOP" (which acts just on the capture recording) and "STOP ALL" which stops the rest of the pipeline if I chose that in case I want to interrupt it manually for any reason. Finally, there is a basic log that is recorded which also times how long each post-process step in the pipeline takes.

Hopefully this might be useful to someone else...

"app.py" is the webapp itself - it runs on TCP port 5050.
"templates/index.html" is the webapp interface.
"skycamcapture.sh" creates the original 4K stream recording from my camera.
"timelapse.sh" creates the timelapse at 20x speed - it does it in a very fast way without re-encoding the video by changing the presentation timestamps.
"transcode_720p.sh" creates the 720p version from the original 4K video accelerated for an NVIDIA GPU.
"pipeline.sh" is the optional sequence which can be called to do everything.

app.py:

import os
import subprocess
import glob
import shutil
import requests
from flask import Flask, render_template, request, jsonify, send_from_directory, Response

app = Flask(__name__)
STORAGE_PATH = "/export/media/skycam"
METEOR_PATH = "/export/media/meteors"
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))

def is_file_being_written(filename):
    target = os.path.realpath(filename)
    for fd_path in glob.glob('/proc/*/fd/*'):
        try:
            if os.path.realpath(fd_path) == target:
                parts = fd_path.split('/')
                pid, fd_num = parts[2], parts[4]
                info_path = f"/proc/{pid}/fdinfo/{fd_num}"
                with open(info_path, 'r') as f:
                    for line in f:
                        if line.startswith('flags:'):
                            flags_oct = int(line.split()[1], 8)
                            if (flags_oct & 3) in [1, 2]:
                                return True
        except (OSError, FileNotFoundError, ValueError):
            continue
    return False

@app.route('/')
def index():
    return render_template('index.html')

# --- PROXIES (To bypass CORS security) ---

@app.route('/stream_proxy')
def stream_proxy():
    url = "http://ip.of.camera/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=password"
    def generate():
        try:
            r = requests.get(url, stream=True, timeout=10)
            for chunk in r.iter_content(chunk_size=1024 * 32):
                if chunk:
                    yield chunk
        except Exception as e:
            print(f"Stream Proxy Error: {e}")
    return Response(generate(), mimetype='video/x-flv')

@app.route('/snap_proxy')
def snap_proxy():
    url = "http://ip.of.camera/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=1&user=user&password=password&width=3840&height=2160"
    try:
        r = requests.get(url, timeout=10)
        return Response(r.content, mimetype='image/jpeg')
    except Exception as e:
        return jsonify({"status": f"Snap Error: {str(e)}"}), 500

# --- CAPTURE CONTROLS ---

@app.route('/start', methods=['POST'])
def start():
    hours = request.json.get('hours', 1)
    subprocess.Popen(['/bin/bash', os.path.join(SCRIPT_DIR, 'skycamcapture.sh'), str(hours)])
    return jsonify({"status": "Recording started"})

@app.route('/autodetect', methods=['POST'])
def autodetect():
    hours = request.json.get('hours', 1)
    subprocess.Popen(['/bin/bash', os.path.join(SCRIPT_DIR, 'skycamcapture.sh'), str(hours), "auto-detect-meteors"])
    return jsonify({"status": "AUTO-DETECT-METEORS: Recording started"})

@app.route('/stop', methods=['POST'])
def stop():
    subprocess.run(['pkill', '-f', 'skycamcapture.sh'])
    return jsonify({"status": "Stopped"})

@app.route('/stop_all', methods=['POST'])
def stop_all():
    for proc in ['skycamcapture.sh', 'pipeline.sh', 'MetDetPy.py', 'ClipToolkit.py']:
        subprocess.run(['pkill', '-f', proc])
    return jsonify({"status": "ALL CAPTURE AND PIPELINES TERMINATED"})

# --- LOGS & PROCESSING ---

@app.route('/log')
def get_log():
    log_path = os.path.join(STORAGE_PATH, 'pipeline.log')
    if os.path.exists(log_path):
        with open(log_path, 'r') as f: return f.read()
    return "Log file not found."

@app.route('/log', methods=['DELETE'])
def delete_log():
    log_path = os.path.join(STORAGE_PATH, 'pipeline.log')
    if os.path.exists(log_path):
        os.remove(log_path)
        return jsonify({"status": "Log deleted"})
    return jsonify({"status": "Log not found"}), 404

@app.route('/timelapse/<path:filename>', methods=['POST'])
def trigger_timelapse(filename):
    parts = filename.split('/')
    full_path = os.path.join(STORAGE_PATH, "/".join(parts[1:]))
    subprocess.Popen(['/bin/bash', os.path.join(SCRIPT_DIR, 'timelapse.sh'), full_path])
    return jsonify({"status": "Timelapse started"})

@app.route('/transcode/<path:filename>', methods=['POST'])
def trigger_transcode(filename):
    parts = filename.split('/')
    full_path = os.path.join(STORAGE_PATH, "/".join(parts[1:]))
    subprocess.Popen(['/bin/bash', os.path.join(SCRIPT_DIR, 'transcode_720p.sh'), full_path])
    return jsonify({"status": "720p Transcode started"})

@app.route('/concat/<folder>', methods=['POST'])
def concat_meteors(folder):
    folder_path = os.path.join(METEOR_PATH, folder)
    if not os.path.exists(folder_path): return jsonify({"status": "Folder not found"}), 404
    files = sorted([f for f in os.listdir(folder_path) if f.endswith('.mp4') and '_meteors.mp4' not in f])
    if not files: return jsonify({"status": "No clips found"}), 400
    output_path = os.path.join(folder_path, f"{folder}_meteors.mp4")
    list_file = os.path.join(folder_path, "files.txt")
    try:
        with open(list_file, 'w') as f:
            for file in files: f.write(f"file '{file}'\n")
        subprocess.run(['ffmpeg', '-y', '-f', 'concat', '-safe', '0', '-i', list_file, '-c', 'copy', output_path], check=True)
        os.remove(list_file)
        subprocess.run(['chown', '1000:1000', output_path])
        return jsonify({"status": "Concatenation complete"})
    except Exception as e:
        return jsonify({"status": f"Error: {str(e)}"}), 500

# --- NAVIGATION & DELETION ---

@app.route('/folders')
def list_folders():
    sc = [{"name": f, "type": "skycam"} for f in os.listdir(STORAGE_PATH) if os.path.isdir(os.path.join(STORAGE_PATH, f))]
    met = [{"name": f, "type": "meteors"} for f in os.listdir(METEOR_PATH) if os.path.isdir(os.path.join(METEOR_PATH, f))]
    return jsonify(sorted(sc + met, key=lambda x: x['name'], reverse=True))

@app.route('/files/<type>/<folder>')
def list_files(type, folder):
    base = STORAGE_PATH if type == "skycam" else METEOR_PATH
    folder_path = os.path.join(base, folder)
    files = []
    if os.path.exists(folder_path):
        for f in sorted(os.listdir(folder_path), reverse=True):
            if f.lower().endswith(('.mp4', '.avi')):
                full_path = os.path.join(folder_path, f)
                is_busy = is_file_being_written(full_path)
                status = "IDLE"
                if is_busy:
                    if "_tl.mp4" in f: status = "TIMELAPSE"
                    elif "_720p.mp4" in f: status = "720P"
                    else: status = "RECORDING"
                files.append({"name": f, "path": f"{type}/{folder}/{f}", "status": status, "type": type})
    return jsonify(files)

@app.route('/video/<path:filename>')
def serve_video(filename):
    parts = filename.split('/')
    base = STORAGE_PATH if parts[0] == "skycam" else METEOR_PATH
    return send_from_directory(base, "/".join(parts[1:]))

@app.route('/delete/<path:filename>', methods=['DELETE'])
def delete_file(filename):
    parts = filename.split('/')
    base = STORAGE_PATH if parts[0] == "skycam" else METEOR_PATH
    file_path = os.path.join(base, "/".join(parts[1:]))
    if os.path.exists(file_path):
        os.remove(file_path)
        return jsonify({"status": "Deleted"})
    return jsonify({"status": "File not found"}), 404

@app.route('/delete_folder/<type>/<folder>', methods=['DELETE'])
def delete_folder(type, folder):
    base = STORAGE_PATH if type == "skycam" else METEOR_PATH
    folder_path = os.path.join(base, folder)
    if os.path.exists(folder_path):
        shutil.rmtree(folder_path)
        return jsonify({"status": "Folder deleted"})
    return jsonify({"status": "Not found"}), 404

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5050)

templates/index.html:

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>SKYCAM</title>
    <script src="https://cdn.tailwindcss.com"></script>
    <script src="https://cdn.jsdelivr.net/npm/mpegts.js@1.7.3/dist/mpegts.min.js"></script>
    <style>
        @keyframes pulse-border { 0%, 100% { border-color: rgba(59, 130, 246, 0.4); } 50% { border-color: rgba(59, 130, 246, 1); } }
        @keyframes pulse-red { 0%, 100% { border-color: rgba(239, 68, 68, 0.4); } 50% { border-color: rgba(239, 68, 68, 1); } }
        .is-processing { animation: pulse-border 2s infinite; border-width: 2px; }
        .is-recording { animation: pulse-red 2s infinite; border-width: 2px; }
        pre::-webkit-scrollbar { width: 8px; }
        pre::-webkit-scrollbar-thumb { background: #334155; border-radius: 4px; }
    </style>
</head>
<body class="bg-slate-900 text-white p-6 font-sans">

    <div class="max-w-4xl mx-auto flex flex-col md:flex-row justify-between items-center mb-6 bg-slate-800 p-4 rounded-lg shadow-lg gap-4 border-b-2 border-slate-700">
        <h1 class="text-2xl font-bold text-blue-400 uppercase italic tracking-tighter text-shadow">SKYCAM</h1>
        <div class="flex gap-2 flex-wrap justify-center">
            <input type="number" id="hrs" value="1" step="0.5" class="w-16 bg-slate-700 p-2 rounded text-white border border-slate-600 outline-none">
            <button onclick="cmd('start')" class="bg-blue-600 hover:bg-blue-500 px-3 py-2 rounded font-bold text-xs uppercase">Start</button>
            <button onclick="cmd('autodetect')" class="bg-green-600 hover:bg-green-500 px-3 py-2 rounded font-bold text-xs text-center leading-tight uppercase">Auto-Detect<br>Meteors</button>
            <button onclick="toggleLive()" id="live-btn" class="bg-purple-600 hover:bg-purple-500 px-3 py-2 rounded font-bold text-xs uppercase">Live Feed</button>
            <button onclick="showSnap()" class="bg-indigo-600 hover:bg-indigo-500 px-3 py-2 rounded font-bold text-xs uppercase">Live Image</button>
            <button onclick="cmd('stop')" class="bg-orange-600 hover:bg-orange-500 px-3 py-2 rounded font-bold text-xs uppercase">Stop</button>
            <button onclick="cmd('stop_all')" class="bg-red-700 hover:bg-red-600 px-3 py-2 rounded font-bold text-xs uppercase">Stop All</button>
        </div>
    </div>

    <div id="live-container" class="max-w-4xl mx-auto mb-8 hidden">
        <div class="bg-black rounded-xl overflow-hidden border-2 border-purple-500 shadow-lg">
            <div class="p-2 bg-slate-800 flex justify-between items-center">
                <span class="text-[10px] font-black tracking-widest text-purple-400 ml-2 animate-pulse uppercase italic">● Live Camera Feed</span>
                <button onclick="toggleLive()" class="text-xs font-bold px-2 text-slate-400 hover:text-white">CLOSE ✕</button>
            </div>
            <video id="live-video" controls autoplay muted class="w-full aspect-video bg-black"></video>
        </div>
    </div>

    <div id="snap-container" class="max-w-4xl mx-auto mb-8 hidden">
        <div class="bg-black rounded-xl overflow-hidden border-2 border-indigo-500 shadow-lg">
            <div class="p-2 bg-slate-800 flex justify-between items-center">
                <span class="text-[10px] font-black tracking-widest text-indigo-400 ml-2 uppercase italic">📸 4K Static Snapshot</span>
                <button onclick="document.getElementById('snap-container').classList.add('hidden')" class="text-xs font-bold px-2 text-slate-400">CLOSE ✕</button>
            </div>
            <img id="snap-img" src="" class="w-full h-auto bg-black" alt="Live Camera Snapshot">
        </div>
    </div>

    <div class="max-w-4xl mx-auto mb-4 flex gap-2">
        <button onclick="viewLog()" class="text-[10px] bg-slate-800 border border-slate-700 px-4 py-2 rounded font-bold tracking-widest uppercase">View Pipeline Log</button>
        <button onclick="delLog()" class="text-[10px] text-red-400 bg-slate-800 border border-slate-700 px-4 py-2 rounded font-bold tracking-widest uppercase">Clear Log</button>
    </div>
    <pre id="log-viewer" class="max-w-4xl mx-auto mb-8 p-4 bg-black text-green-500 text-[10px] hidden overflow-auto max-h-64 rounded border border-slate-800 font-mono"></pre>

    <div class="max-w-4xl mx-auto mb-6 flex items-center gap-4">
        <button onclick="loadFolders()" class="bg-slate-700 hover:bg-slate-600 border border-slate-500 px-6 py-2 rounded-full font-bold text-blue-300 active:scale-95 transition-all text-xs uppercase">🔄 Refresh</button>
        <div id="path-display" class="text-slate-500 font-mono text-xs uppercase tracking-widest truncate italic">/ Root</div>
    </div>

    <div id="main-view" class="max-w-4xl mx-auto grid grid-cols-1 md:grid-cols-2 gap-6 pb-20"></div>

    <script>
        const pathDisplay = document.getElementById('path-display');
        const view = document.getElementById('main-view');
        let player = null;

        // Browser History Support
        window.onpopstate = (e) => { e.state?.view === 'files' ? loadFiles(e.state.type, e.state.folder, false) : loadFolders(false); };

        async function cmd(action) {
            const h = document.getElementById('hrs').value;
            const res = await fetch('/' + action, { method: 'POST', headers: {'Content-Type': 'application/json'}, body: JSON.stringify({hours: h}) });
            const data = await res.json(); alert(data.status); loadFolders();
        }

        async function viewLog() {
            const el = document.getElementById('log-viewer');
            if (el.classList.contains('hidden')) {
                const res = await fetch('/log'); el.innerText = await res.text(); el.classList.remove('hidden'); el.scrollTop = el.scrollHeight;
            } else { el.classList.add('hidden'); }
        }

        async function delLog() { if(confirm('Clear Log?')) { await fetch('/log', {method:'DELETE'}); viewLog(); } }

        function showSnap() {
            const container = document.getElementById('snap-container');
            const img = document.getElementById('snap-img');
            img.src = '/snap_proxy?t=' + new Date().getTime();
            container.classList.remove('hidden');
            container.scrollIntoView({ behavior: 'smooth' });
        }

        async function toggleLive() {
            const container = document.getElementById('live-container');
            const videoElement = document.getElementById('live-video');
            const btn = document.getElementById('live-btn');
            if (container.classList.contains('hidden')) {
                if (mpegts.isSupported()) {
                    player = mpegts.createPlayer({type: 'flv', isLive: true, url: '/stream_proxy'}, {enableStashBuffer: false, liveBufferLatencyChasing: true, autoCleanupSourceBuffer: true});
                    player.attachMediaElement(videoElement); player.load(); player.play();
                    container.classList.remove('hidden'); btn.classList.replace('bg-purple-600', 'bg-red-600'); btn.innerText = "Close Live";
                    container.scrollIntoView({ behavior: 'smooth' });
                }
            } else {
                if (player) { player.pause(); player.unload(); player.detachMediaElement(); player.destroy(); player = null; }
                container.classList.add('hidden'); btn.classList.replace('bg-red-600', 'bg-purple-600'); btn.innerText = "Live Feed";
            }
        }

        async function loadFolders(push = true) {
            if (push) history.pushState({view: 'folders'}, "", "#root");
            pathDisplay.innerText = "/ ROOT";
            const response = await fetch('/folders');
            const folders = await response.json();
            view.innerHTML = '';
            folders.forEach(f => {
                const card = document.createElement('div');
                card.className = "bg-slate-800 p-8 rounded-xl border border-slate-700 text-center cursor-pointer hover:border-blue-500 transition-all shadow-lg group relative";
                const isMet = f.type === 'meteors';
                const tag = `<span class="absolute top-2 right-3 text-[9px] ${isMet?'text-green-500':'text-blue-500'} font-black tracking-tighter uppercase">${isMet?'Meteors':'Recordings'}</span>`;
                const delBtn = `<button onclick="event.stopPropagation(); delFolder('${f.type}', '${f.name}')" class="absolute top-2 left-3 text-[9px] text-red-600 font-black opacity-0 group-hover:opacity-100 uppercase italic">Del Folder</button>`;
                card.innerHTML = `${tag}${delBtn}<span class="text-4xl block">${isMet?'☄️':'📁'}</span><b class="mt-4 block font-mono text-lg">${f.name}</b>`;
                card.onclick = () => loadFiles(f.type, f.name);
                view.appendChild(card);
            });
        }

        async function loadFiles(type, folder, push = true) {
            if (push) history.pushState({view: 'files', type, folder}, "", `#${type}-${folder}`);
            pathDisplay.innerHTML = `<span class="text-blue-500">/ ${type.toUpperCase()} /</span> ${folder}`;
            view.innerHTML = '';
            if (type === 'meteors') {
                const b = document.createElement('div'); b.className = "col-span-full mb-4";
                b.innerHTML = `<button onclick="runConcat('${folder}')" class="w-full py-3 bg-green-700 hover:bg-green-600 rounded-lg font-black text-xs tracking-widest shadow-lg border border-green-500/30 uppercase">⚡ Concatenate All (.MP4)</button>`;
                view.appendChild(b);
            }
            const response = await fetch(`/files/${type}/${folder}`);
            const files = await response.json();
            files.forEach(f => {
                const item = document.createElement('div');
                const isBusy = f.status !== "IDLE";
                const isRec = f.status === "RECORDING";
                item.className = `bg-slate-800 rounded-lg overflow-hidden border shadow-xl relative transition-all ${isRec ? 'is-recording border-red-500' : (isBusy ? 'is-processing' : 'border-slate-700')}`;
                const overlay = isBusy ? `<div class="absolute inset-0 bg-slate-900/90 z-10 flex flex-col items-center justify-center"><div class="w-8 h-8 border-2 ${isRec ? 'border-red-500' : 'border-blue-500'} border-t-transparent rounded-full animate-spin"></div><span class="mt-3 text-[10px] font-black ${isRec ? 'text-red-500' : 'text-blue-400'} tracking-widest uppercase italic text-center">${isRec ? 'RECORDING...' : f.status + '...'}</span></div>` : '';
                const manualButtons = (type === 'skycam' && !isBusy) ? `<button onclick="process('${f.path}', 'timelapse')" class="text-blue-400 hover:text-white uppercase">TL</button><button onclick="process('${f.path}', 'transcode')" class="text-blue-400 hover:text-white uppercase">720</button>` : '';
                item.innerHTML = `${overlay}<div class="p-3 bg-slate-900 flex justify-between items-center text-[10px] font-black uppercase"><span class="truncate pr-4 ${isBusy ? (isRec ? 'text-red-500' : 'text-blue-400') : 'text-slate-500'} font-mono">${f.name}</span><div class="flex gap-4">${manualButtons}<button onclick="del('${f.path}', '${type}', '${folder}')" class="text-red-600 hover:text-white ${isBusy ? 'opacity-20 pointer-events-none' : ''} uppercase">Del</button></div></div><video controls playsinline class="w-full bg-black aspect-video ${isBusy ? 'opacity-0' : ''}"><source src="/video/${f.path}" type="video/mp4"></video>`;
                view.appendChild(item);
            });
        }

        async function process(path, type) { const res = await fetch(`/${type}/${path}`, { method: 'POST' }); alert((await res.json()).status); const p = path.split('/'); loadFiles(p[0], p[1], false); }
        async function delFolder(t, f) { if(confirm(`Delete folder ${f}?`)) { await fetch(`/delete_folder/${t}/${f}`, {method:'DELETE'}); loadFolders(false); } }
        async function runConcat(f) { if(!confirm('Join clips?')) return; pathDisplay.innerText = "CONCATENATING..."; const r = await fetch(`/concat/${f}`, {method:'POST'}); alert((await r.json()).status); loadFiles('meteors', f, false); }
        async function del(p, t, f) { if(confirm('Delete file?')) { await fetch('/delete/' + p, {method:'DELETE'}); loadFiles(t, f, false); } }

        loadFolders(false);
    </script>
</body>
</html>

skycamcapture.sh:

#!/bin/bash
# $1 = Duration in decimal hours
# $2 = Mode (optional, "auto-detect-meteors" triggers pipeline)

MODE="$2"
SECONDS=$(echo "$1 * 3600 / 1" | bc)

DATE_VAL=$(date +%Y%m%d)
TIME_VAL=$(date +%H%M%S)
FINAL_DIR="/export/media/skycam/$DATE_VAL"
TEMP_FILE="$FINAL_DIR/capture_${TIME_VAL}.ts"
OUTPUT_FILE="$FINAL_DIR/capture_${TIME_VAL}.mp4"

mkdir -p "$FINAL_DIR"

# Web App Fix: Create symlink
ln -s "$TEMP_FILE" "$OUTPUT_FILE"

finalize_video() {
    if [ ! -z "$FFMPEG_PID" ]; then
        kill -TERM "$FFMPEG_PID" 2>/dev/null
        wait "$FFMPEG_PID" 2>/dev/null
    fi

    if [ -f "$TEMP_FILE" ]; then
        rm -f "$OUTPUT_FILE"
        ffmpeg -hide_banner -y -loglevel error \
        -i "$TEMP_FILE" -c copy \
        -video_track_timescale 25 \
        -fflags +genpts+igndts \
        -avoid_negative_ts make_zero \
        -movflags +faststart \
        "$OUTPUT_FILE" && rm "$TEMP_FILE"
    fi

    # Conditional Pipeline Trigger
    if [ "$MODE" == "auto-detect-meteors" ]; then
        /bin/bash /app/pipeline.sh "$OUTPUT_FILE" "$DATE_VAL" &
    fi

    exit 0
}

trap finalize_video SIGTERM SIGINT

ffmpeg -hide_banner -y -loglevel error \
-rtsp_transport tcp \
-use_wallclock_as_timestamps 1 \
-i "rtsp://user:pass@ip.of.camera:554/h265Preview_01_main" \
-t "$SECONDS" \
-vcodec copy -an \
-f mpegts \
"$TEMP_FILE" &

FFMPEG_PID=$!
wait $FFMPEG_PID
finalize_video

timelapse.sh:

#!/bin/bash
# $1 = Full path to the input file

INPUT_FILE="$1"
OUTPUT_FILE="${INPUT_FILE%.mp4}_tl.mp4"

# Fast bitstream copy using only keyframes
ffmpeg -hide_banner -loglevel error -discard nokey -i "$INPUT_FILE" \
-c:v copy -bsf:v "setts=ts=N/20/TB" -an "$OUTPUT_FILE"

transcode_720p.sh:

#!/bin/bash
# $1 = Full path to the input file

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

INPUT_FILE="$1"
OUTPUT_FILE="${INPUT_FILE%.mp4}_720p.mp4"

# Your specific CUDA hardware-accelerated command
ffmpeg -y -nostdin -hide_banner \
-hwaccel cuda \
-hwaccel_output_format cuda \
-fflags +genpts \
-c:v hevc_cuvid -resize 1280x720 -i "$INPUT_FILE" \
-c:v hevc_nvenc \
-preset p1 \
-rc constqp -qp 30 \
-spatial-aq 0 \
-temporal-aq 0 \
-nonref_p 1 \
-an \
-loglevel warning \
"$OUTPUT_FILE"

pipeline.sh

#!/bin/bash
# $1 = Original MP4 Path
# $2 = Date Value (YYYYMMDD)

ORIGINAL_MP4="$1"
DATE_VAL="$2"
TRANSCODE_720P="${ORIGINAL_MP4%.mp4}_720p.mp4"
DETECTIONS_JSON="/root/MetDetPy/detections/detections.json"
LOG_FILE="/export/media/skycam/pipeline.log"
CLIP_DIR="/export/media/meteors/$DATE_VAL"

log_msg() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Function to calculate and log duration
log_duration() {
    local start=$1
    local end=$(date +%s)
    local diff=$((end - start))
    echo " (Took ${diff}s)" >> "$LOG_FILE"
}

log_msg "START: AUTO-DETECT-METEORS for $ORIGINAL_MP4"
TOTAL_START=$(date +%s)

# 1. Timelapse and Transcode
echo -n "$(date '+%Y-%m-%d %H:%M:%S') - STEP 1: Timelapse and 720p Transcode..." >> "$LOG_FILE"
STEP_START=$(date +%s)
# Run timelapse in background, transcode in foreground
/bin/bash /app/timelapse.sh "$ORIGINAL_MP4" > /dev/null 2>&1 &
/bin/bash /app/transcode_720p.sh "$ORIGINAL_MP4" > /dev/null 2>&1
wait
log_duration $STEP_START

# 2. MetDetPy Analysis
echo -n "$(date '+%Y-%m-%d %H:%M:%S') - STEP 2: MetDetPy analysis..." >> "$LOG_FILE"
STEP_START=$(date +%s)
/bin/bash /root/MetDetPy/run_p600.sh "$TRANSCODE_720P" > /dev/null 2>&1
if [ $? -eq 0 ]; then
    log_duration $STEP_START
else
    echo " FAILED" >> "$LOG_FILE"
fi

# 3. Path Swap in JSON
# This ensures ClipToolkit cuts from the 4K original, not the 720p version
echo -n "$(date '+%Y-%m-%d %H:%M:%S') - STEP 3: Swapping JSON paths..." >> "$LOG_FILE"
STEP_START=$(date +%s)
if [ -f "$DETECTIONS_JSON" ]; then
    sed -i "s|$TRANSCODE_720P|$ORIGINAL_MP4|g" "$DETECTIONS_JSON"
    log_duration $STEP_START
else
    echo " FAILED (File not found)" >> "$LOG_FILE"
fi

# 4. Clip Generation and MP4 Conversion
echo -n "$(date '+%Y-%m-%d %H:%M:%S') - STEP 4: ClipToolkit generation & MP4 conversion..." >> "$LOG_FILE"
STEP_START=$(date +%s)
mkdir -p "$CLIP_DIR"
cd /root/MetDetPy || exit

# Run the python toolkit
/root/MetDetPy/.venv/bin/python ClipToolkit.py --mode video --enable-filter-rules \
--save-path "$CLIP_DIR" \
"$DETECTIONS_JSON" --padding-before 2 --padding-after 2 > /dev/null 2>&1

if [ $? -eq 0 ]; then
    # REMUX: Convert AVI to MP4 for web compatibility without re-encoding
    for avi_file in "$CLIP_DIR"/*.avi; do
        if [ -f "$avi_file" ]; then
            ffmpeg -y -hide_banner -loglevel error -i "$avi_file" -c copy "${avi_file%.avi}.mp4" && rm "$avi_file"
        fi
    done
    log_duration $STEP_START
else
    echo " FAILED" >> "$LOG_FILE"
fi

# 5. Permissions
echo -n "$(date '+%Y-%m-%d %H:%M:%S') - STEP 5: Finalizing permissions..." >> "$LOG_FILE"
STEP_START=$(date +%s)
# Ensure both storage paths and the log are owned by UID 1000
chown -R 1000:1000 /export/media/meteors
chown -R 1000:1000 /export/media/skycam
chown 1000:1000 "$LOG_FILE"
log_duration $STEP_START

FINAL_END=$(date +%s)
TOTAL_DIFF=$((FINAL_END - TOTAL_START))
log_msg "FINISH: AUTO-DETECT-METEORS complete for $ORIGINAL_MP4 (Total Time: ${TOTAL_DIFF}s)"
echo "----------------------------------------------------------------" >> "$LOG_FILE"