Skip to content

open_source_code_readme.md

Latest commit

 

History

History
197 lines (183 loc) · 10.2 KB

open_source_code_readme.md

File metadata and controls

197 lines (183 loc) · 10.2 KB

ros2_openvino_toolkit

1. Introduction

The OpenVINO™ toolkit quickly deploys applications and solutions that emulate human vision. Based on Convolutional Neural Networks (CNN), the Toolkit extends computer vision (CV) workloads across Intel® hardware, maximizing performance.

This project is a ROS2 wrapper for CV API of OpenVINO™, providing the following features:

  • Support CPU and GPU platforms
  • Support standard USB camera and Intel® RealSense™ camera
  • Support Video or Image file as detection source
  • Face detection
  • Emotion recognition
  • Age and gender recognition
  • Head pose recognition
  • Demo application to show above detection and recognitions

2. Prerequisite

  • An x86_64 computer running Ubuntu 16.04 and Ubuntu 18.04. Below processors are supported:

    • 6th-8th Generation Intel® Core™
    • Intel® Xeon® v5 family
    • Intel® Xeon® v6 family

  • ROS2 Bouncy

  • OpenVINO™ Toolkit Open Source

    • The Deep Learning Deployment Toolkit that helps to enable fast, heterogeneous deep learning inferencing for Intel® processors (CPU and GPU/Intel® Processor Graphics), and supports more than 100 public and custom models.
    • Open Model Zoo includes 20+ pre-trained deep learning models to expedite development and improve deep learning inference on Intel® processors (CPU, Intel Processor Graphics, FPGA, VPU), along with many samples to easily get started.

  • RGB Camera, e.g. RealSense D400 Series or standard USB camera or Video/Image File

  • Graphics are required only if you use a GPU. The official system requirements for GPU are:

    • 6th to 8th generation Intel® Core™ processors with Iris® Pro graphics and Intel® HD Graphics
    • 6th to 8th generation Intel® Xeon® processors with Iris Pro graphics and Intel HD Graphics (excluding the e5 product family, which does not have graphics)
    • Intel® Pentium® processors N4200/5, N3350/5, N3450/5 with Intel HD Graphics

  • Use one of the following methods to determine the GPU on your hardware:

    • [lspci] command: GPU info may lie in the [VGA compatible controller] line.
    • Ubuntu system: Menu [System Settings] --> [Details] may help you find the graphics information.
    • Openvino: Download the install package, install_GUI.sh inside will check the GPU information before installation.

3. Environment Setup

Note:You can choose to build the environment using ./environment_setup.sh script.

./environment_setup.sh hostname password

Note:You can also choose to follow the steps below to build the environment step by step.

  • Install ROS2 Bouncy (guide)

  • Install OpenVINO™ Toolkit Open Source

    • Install OpenCV 3.3 or later(guide) 
       [compiler] sudo apt-get install build-essential
 [required] sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
 [optional] sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev 
 cd ~/code
 git clone https://github.com/opencv/opencv.git
 git clone https://github.com/opencv/opencv_contrib.git
 cd opencv && git checkout 3.3.1 && cd ..
 cd opencv_contrib && git checkout 3.3.1 && cd ..
 cd opencv
 mkdir build && cd build
 cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local -D OPENCV_EXTRA_MODULES_PATH=/home/<hostname>/code/opencv_contrib/modules/ ..
 make -j8
 sudo make install
      • Additional steps are required on ubuntu 18.04 
         sudo add-apt-repository "deb http://security.ubuntu.com/ubuntu xenial-security main"
 sudo apt update
 sudo apt install libjasper1 libjasper-dev
    • Install OpenCL Driver for GPU(guide)
       cd ~/Downloads
 wget http://registrationcenter-download.intel.com/akdlm/irc_nas/11396/SRB5.0_linux64.zip
 unzip SRB5.0_linux64.zip -d SRB5.0_linux64
 cd SRB5.0_linux64
 sudo apt-get install xz-utils
 mkdir intel-opencl
 tar -C intel-opencl -Jxf intel-opencl-r5.0-63503.x86_64.tar.xz
 tar -C intel-opencl -Jxf intel-opencl-devel-r5.0-63503.x86_64.tar.xz
 tar -C intel-opencl -Jxf intel-opencl-cpu-r5.0-63503.x86_64.tar.xz
 sudo cp -R intel-opencl/* /
 sudo ldconfig
    • Install Deep Learning Deployment Toolkit(guide)
       mkdir ~/code && cd ~/code
 git clone https://github.com/opencv/dldt.git
 cd dldt/inference-engine/
 ./install_dependencies.sh
 mkdir build && cd build
 cmake -DCMAKE_BUILD_TYPE=Release ..
 make -j8
      Note:On Ubuntu 18.04,replace libpng12-dev with libpng-dev in the install_dependencies.sh file.
    • Install Open Model Zoo(guide)
       cd ~/code
 git clone https://github.com/opencv/open_model_zoo.git
 cd open_model_zoo/demos/
 mkdir build && cd build
 cmake -DCMAKE_BUILD_TYPE=Release <path_to_inference_engine_demos_directory>
 make -j8
      Note:<path_to_inference_engine_demos_directory> is /home/<hostname>/code/dldt/inference-engine,After that you can find binaries for all demos applications in the intel64/Release subfolder.

  • Install Intel® RealSense™ SDK 2.0 (tag v2.14.1)

  • Other Dependencies 
     # numpy
 pip3 install numpy
    • Ubuntu 16.04 
       # libboost
 sudo apt-get install -y --no-install-recommends libboost-all-dev
 cd /usr/lib/x86_64-linux-gnu
 sudo ln -s libboost_python-py35.so libboost_python3.so
    • Ubuntu 18.04 
       #libboost
 sudo apt-get install -y --no-install-recommends libboost-all-dev
 sudo apt install libboost-python1.62.0

4. Building and Installation

  • set ENV InferenceEngine_DIR, CPU_EXTENSION_LIB and GFLAGS_LIB

     export InferenceEngine_DIR=/home/<hostname>/code/dldt/inference-engine/build/
 export CPU_EXTENSION_LIB=/home/<hostname>/code/dldt/inference-engine/bin/intel64/Release/lib/libcpu_extension.so
 export GFLAGS_LIB=/home/<hostname>/code/dldt/inference-engine/bin/intel64/Release/lib/libgflags_nothreads.a

  • Install ROS2_OpenVINO packages

     mkdir -p ~/ros2_overlay_ws/src
 cd ~/ros2_overlay_ws/src
 git clone https://github.com/intel/ros2_openvino_toolkit
 git clone https://github.com/intel/ros2_object_msgs
 git clone https://github.com/ros-perception/vision_opencv -b ros2

  • Build package

     source ~/ros2_ws/install/local_setup.bash
 cd ~/ros2_overlay_ws
 colcon build --symlink-install

5. Running the Demo

  • Preparation
    • download model file (excute once)
       cd ~/code/open_model_zoo/model_downloader
 python3 downloader.py --name face-detection-adas-0001
 python3 downloader.py --name age-gender-recognition-retail-0013
 python3 downloader.py --name emotions-recognition-retail-0003
 python3 downloader.py --name head-pose-estimation-adas-0001
 python3 downloader.py --name person-vehicle-bike-detection-crossroad-0078
    • copy label files (excute once)
       sudo cp ~/ros2_overlay_ws/src/ros2_openvino_toolkit/data/labels/emotions-recognition/FP32/emotions-recognition-retail-0003.labels /home/<hostname>/code/open_model_zoo/model_downloader/Retail/object_attributes/emotions_recognition/0003/dldt
    • set ENV LD_LIBRARY_PATH
       export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<hostname>/code/dldt/inference-engine/bin/intel64/Release/lib
  • run sample code with parameters extracted from yaml. 
     ros2 run dynamic_vino_sample pipeline_with_params -config /home/<hostname>/ros2_overlay_ws/src/ros2_openvino_toolkit/sample/param/pipeline_people_oss.yaml
  • run object detection sample code with paramters extracted from yaml. 
     ros2 run dynamic_vino_sample object_detection_with_params -config /home/<hostname>/ros2_overlay_ws/src/ros2_openvino_toolkit/sample/param/pipeline_object_oss.yaml
    Note: Need to manually modify the path of the model file in the .yaml configuration file. 
     model: /home/<hostname>/code/open_model_zoo/model_downloader/Transportation/object_detection/face/pruned_mobilenet_reduced_ssd_shared_weights/dldt/face-detection-adas-0001.xml

6. Interfaces

6.1 Topic

7. Known Issues

  • Parameters "-m_ag, -m_hp, -m_em" should be optional, but samples throw exception without them.
  • Parameters "-n_ag, -n_hp, -n_em" doesn't work. The maximum number of face/age/headpose/emotion is always 16.
  • Standard USB camera can be unexpected launched with input parameter "-i RealSenseCamera".
Any security issue should be reported using process at https://01.org/security