ECSAM: Edge-Cloud Segment Anything Model System (keep updating)

Overview

ECSAM is a research and prototype system for edge-cloud video analytics based on Meta's Segment Anything Model (SAM). It enables real-time, prompt-driven video segmentation and analytics on resource-constrained edge devices, with dynamic offloading to the cloud. ECSAM is designed for applications such as visual tour guides, smart surveillance, and interactive media analytics.

Motivation

• Foundation models like SAM enable flexible, prompt-based segmentation but are resource-intensive.
• Edge devices (e.g., Nvidia Jetson) have limited compute and bandwidth.
• ECSAM bridges the gap with:
  • Visual prompt transformation
  • Adaptive model partitioning
  • Real-time resource and network monitoring
  • Efficient edge-cloud communication (QUIC, compression)

Architecture

ECSAM extends Meta SAM with:

• Edge modules:
  • Context/session management (ECSAM_Context_Mgmt)
  • Visual prompt transformation (ECSAM_VP_Trans)
  • Image encoder with partitioning (ECSAM_ImgEnc)
  • Embedding decoder (ECSAM_Dec)
  • TensorRT inference, GStreamer capture, resource monitoring
• Cloud modules:
  • Cloud registration (ECSAM_Cloud_Reg)
  • Cloud offloading/inference (ECSAM_Cloud_Offloading)
  • Meta SAM base modules (AppContext, ImgtoEmb, EmbDec)
• Network simulation:
  • QUIC streaming with compression
  • Realistic bandwidth traces (4G, 5G, WiFi, wired)
• Datasets:
  • KITTI, LIV360 loaders and batchers

Directory Structure

edge/         # Edge-side modules (inference, capture, partition, resource)
cloud/        # Cloud-side modules (registration, offloading, SAM base)
network/      # Network simulation and streaming (QUIC, traces)
dataset/      # Dataset loaders (KITTI, LIV360)
algorithm/    # Core algorithms (mutual information, partitioning)

requirements.txt
README.md
ecsam_system_main.py   # Main entry point
evaluate_ecsam.py      # Evaluation script
ecsam.txt              # Paper/description

Installation

1. Clone the repo:

   git clone https://github.com/yourusername/ecsam.git
   cd ecsam

2. Install dependencies:

   pip install -r requirements.txt

3. (Optional) Install Meta SAM and TensorRT for full functionality.

Example Usage

Run the ECSAM pipeline

python ecsam_system_main.py

Evaluate ECSAM output

python evaluate_ecsam.py

Load and batch a dataset

from dataset.dataset_loader import DatasetLoader
loader = DatasetLoader()
loader.load_kitti('path/to/kitti/images', 'path/to/kitti/masks')
imgs, masks = loader.get_batch(4)

Simulate a network trace

from network.network_simulator import NetworkSimulator
sim = NetworkSimulator(network_type='4g_lte', duration=120, interval=1)
sim.simulate_trace(seed=42)
sim.save_trace('4g_lte_trace.json')

Features

• Edge-cloud partitioning with dynamic control loop
• Visual prompt transformation for efficient prompt handling
• TensorRT inference and GStreamer capture on Jetson
• QUIC streaming with fp16/int8/ROI compression
• Resource monitoring (CPU, memory, GPU, network)
• Network simulation for realistic experiments
• Dataset support for KITTI and LIV360

Citation

If you use ECSAM in your research, please cite:

@article{lu2024samedge,
  title={Samedge: An edge-cloud video analytics architecture for the segment anything model},
  author={Lu, Rui and Shi, Siping and Liu, Yanting and Wang, Dan},
  journal={arXiv preprint arXiv:2409.14784},
  year={2024}
}

Acknowledgments

• Meta AI for the Segment Anything Model (SAM)
• KITTI and LIV360 dataset providers
• Nvidia for Jetson and TensorRT