4Kp30 Multi-Sensor Camera with AI Inference Solution System Example Design for Agilex™ 5 Devices - Repository

Overview

This repository contains the necessary files and collateral to create and build the Camera with AI Inference Solution System Example Design.

An example of AI Detect	An example of AI Pose

The products of this repository are generated using Software flow, Hardware flow, and YOLO v8 Nano detection and pose graph compilation are:

Product	Type	Description
.sof	SRAM Object File	FPGA bitstream to be loadeded over JTAG
.rbf	Raw Binary File	FPGA bitstream file to be loaded from the microSD Card
.jic	JTAG Indirect Configuration File	QSPI Flash programming file for booting from microSD Card
.wiz.gz	Image File	microSD Card Image file containing the Linux embedded system and Camera Software Application
.txt	Categories File	list of categories equating to each channel of the YOLO v8 graph result
.bin	Compiled Graph File	compiled YOLO v8 graph

License Requirements

The System Example Design supports the OpenCore Plus (OCP) evaluation license. The resulting .sof can be tested on Hardware using both the time limited and JTAG tethered features of the license.

Alternatively, full licenses for the VVP IP Suite, VVP Tone Mapping Operator (TMO) IP, and 3D LUT IP are required to produce a .jic and .rbf for a Hardware, SD Card turnkey solution.

The Altera® FPGA AI Suite license operates independently to the rest of the IP. A full license gives you unlimited inferences whereas the evaluation license for this System Example Design gives you 100k free inferences before it times out (just under 1 hour at 30 FPS). Once expired, the AI Overlay will display a permanent "license limit reached" banner.

The following table summarizes the solution and operation based on the license combinations:

<style> r { color: Red; font-weight: bold;} g { color: Green; font-weight: bold;} </style>

Combination	Video Full	OCP Tethered	OCP Untethered	AI Full	AI Eval	SOF	JIC/RBF	Operation
1	✓	✗	✗	✓	✗	✓	✓	Unlimited
2	✓	✗	✗	✗	✓	✓	✓	1 Hour Inference Limited
3	✗	✓	✗	✓	✗	✓	✗	Tether Limited
4	✗	✓	✗	✗	✓	✓	✗	1 Hour Inference Limited
5	✗	✗	✓	✓	✗	✓	✗	1 Hour Untethered Limited
6	✗	✗	✓	✗	✓	✓	✗	1 Hour Untethered and Inference Limited

Note that for all cases, free licenses for MIPI D-Phy IP, MIPI CSI-2 IP, and Nios® V Processor must be downloaded and installed.

Additional licensing information can be found here:

• VVP IP Suite
• Tone Mapping Operator
• 3D LUT
• MIPI IP
• Nios® V Processor
• Altera® FPGA AI Suite

Software Flow

The Software flow generates the appropriate microSD Card Image based on your license and solution requirements.

YOLO v8 Nano detection and pose graph compilation

The YOLO v8 Nano detection and pose graph compilation generates the compiled graphs to be copied to microSD card.

Hardware Flow

The Hardware flow uses the Modular Design Toolkit (MDT) to create and build the Quartus® project for the Camera Solution System Example Design.

Software Requirements

The MDT requires the following Linux versions of software tools:

• Altera® Quartus® Prime Pro Edition version 25.1 Linux
• NiosV Processor for Altera® FPGA

Creating and compiling the Camera Solution System Example Design

Create the design using the Modular Design Toolkit (MDT)

Follow the next steps to create the Quartus® and Platform Designer Project for the Camera Solution System Example Design:

• Create your workspace and clone the repository using --recurse-submodules:

cd <workspace>
git clone -b <TAG> --recurse-submodules https://github.com/altera-fpga/agilex-ed-camera-ai.git agilex-ed-camera-ai

• Define a ./<project> location of your choice, creating directory structure where necessary.

• Navigate to the agilex-ed-camera-ai directory containing the cloned repository and create your project, selecting the XML variant based on your license and solution requirements:

  (Note that the LICENSED variable within the .xml must match your FPGA AI Suite license (full or evaluation) otherwise you will get a default FPGA AI Suite 10k inference limited design (just over 5 minutes at 30 FPS), even if you have a full license)

# SOF MDT Flow (assumes license combination 4 or 6)
# (For license combination 3 or 5, modify the LICENSED variable in the .xml to 1. You must have a full license otherwise you will get a 10k inference limited design)
cd agilex-ed-camera-ai/AGX_5E_Altera_Modular_Dk_ISP_designs
quartus_sh -t ./modular-design-toolkit/scripts/create/create_shell.tcl -proj_path <project> -proj_name agilex5_modkit_vvpisp -xml_path ./AGX_5E_Altera_Modular_Dk_ISP_designs/AGX_5E_Modular_Devkit_ISP_AI_FF_RD.xml

# RBF MDT Flow (assumes license combination 1)
# (For license combination 2, modify the LICENSED variable in the .xml to 0, otherwise you will get a 10k inference limited design)
cd agilex-ed-camera-ai/AGX_5E_Altera_Modular_Dk_ISP_designs
quartus_sh -t ./modular-design-toolkit/scripts/create/create_shell.tcl -proj_path <project> -proj_name agilex5_modkit_vvpisp -xml_path ./AGX_5E_Altera_Modular_Dk_ISP_designs/AGX_5E_Modular_Devkit_ISP_AI_RD.xml

This will create your Quartus® Prime and Platform Designer Project in ./<project>. The folder structure is consistent with the MDT methodology.

Building the design using the Modular Design Toolkit (MDT)

Follow the next steps to build the Camera Solution System Example Design:

• Navigate to the ./<project>/scripts directory and build your project, selecting the post processing step option for your chosen MDT flow:

cd ./<project>/scripts 
# SOF MDT Flow
quartus_sh -t build_shell.tcl -update_ocs -full_compile -ff_post_agx5e

cd ./<project>/scripts 
# RBF MDT Flow (not supported with OCP evaluation license)
quartus_sh -t build_shell.tcl -update_ocs -full_compile -hps_post_agx5e

The MDT build options (all of which are needed for a working build):

• -update_ocs is used to generate the automatic Offset Capability Structure (OCS) ROM, which will be built into the project during compilation.
• -full_compile performs not just the full Quartus compilation, but also compiles any Nios® V software into .hex ROM files built into the project during compilation.
• -ff_post_agx5e option post processes the FPGA First .sof with a first stage bootloader from a U-Boot secondary program loader binary file u-boot-spl-dtb_ff.hex.
• -hps_post_agx5e option post processes the HPS first .sof with a U-Boot secondary program loader binary file u-boot-spl-dtb.hex.

The FPGA programming file/s are located in the ./<project>/quartus/output_files directory and will differ depending on the MDT flow used:

• For SOF MDT Flow:
  • fsbl_agilex5_modkit_vvpisp_time_limited.sof
• For RBF MDT Flow:
  • agilex5_modkit_vvpisp.hps_first.hps.jic and agilex5_modkit_vvpisp.hps_first.core.rbf.

Running the Camera Solution System Example Design

Hardware Requirements

The following equipment is needed to test the Camera Solution System Example Design on hardware:

• Agilex™ 5 FPGA E-Series 065B Modular Development Kit.
• 1 or 2 Framos FSM:GO IMX678C Camera Modules, with:
  • Wide 110deg HFOV Lens, or
  • Medium 100deg HFOV Lens, or
  • Narrow 54deg HFOV Lens.
• Mount/Tripod
  • Framos Tripod Mount Adapter.
  • Tripod.
• A Framos cable for PixelMate MIPI-CSI-2 for each Camera Module.
  • 150mm flex-cable, or
  • 300mm micro-coax cable.
• (Optional) Framos GMSL3 5m.
• Minimum 8GB U3 micro SD Card.
• DP Cable or HDMI Cable with Converter Dongle (recommend 4Kp60 capable).
• USB Micro B JTAG Cable (for JTAG programming).
• USB Micro B Serial Cable (for HPS terminal connection).
• RJ45 Ethernet Cable (HPS network connection).
• Monitor/TV (recommended 4Kp60 capable).

In addition, the following Software tools might be useful:

• Altera® Quartus® Prime Pro Edition version 25.1 Programmer and Tools Standalone.
• SD Card Formatter.
• SD Card Imager such as Win32DiskImager.
• TeraTerm or PuTTY for serial connection.
• Web browser (i.e. Chrome) for Webserver connection.

Hardware Setup

Ensure you Modular Development Kit has these default switch settings (note that the SOM is the mezzanine board):

Switch	Board	Position
S4	SOM	OFF-OFF
S1	SOM	ON-ON
S7	Carrier	OFF-OFF-OFF-ON
S13	Carrier	OFF-OFF-ON-OFF
S1	Carrier	OFF-OFF-OFF-OFF
S5	Carrier	OFF-OFF-OFF-OFF
S2	Carrier	OFF-OFF-OFF-OFF
S11	Carrier (underside)	OFF-OFF-OFF-OFF
S6	Carrier (underside)	OFF-OFF-OFF-OFF
S4	Carrier	OFF

Make the following connections to the Modular Development Kit:

• Micro USB cable (carrier board J35) to Host PC
• Micro USB cable (SOM board J2 - HSP_UART) to Host PC
• RJ45 cable (ethernet port on the SOM board J6 - ETH 1G HPS) to Host PC
• Framos cable to Framos Camera Modules (ensure pin 1 aligns to pin 1)
• Display Port Tx to Monitor (using suitable cable and converter dongle if using HDMI)

Burn your generated microSD Card image and insert it into the microSD Card slot located on the SOM board. Ensure you are using the correct image for your license and solution requirements and that it matches the Hardware Flow used.

If you are using the .rbf with full license flow, then you must program the QSPI Flash which should only need to be done once: (skip this step if you are using the .sof with OCP license flow)

• Power down the board.
• Set S4 on the SOM to OFF-OFF.
• Power up the board.
• Use the following command:

quartus_pgm -c 1 -m jtag -o "pvi;agilex5_modkit_vvpisp.hps_first.hps.jic"

• Alternatively use the Quartus® Programmer GUI
  • Configure the JTAG Hardware by selecting the boards byteBlaster hardware. The Hardware Frequency should be 24MHz.
  • Auto Detect and select the A5EC065BB32AR0 device.
  • Change File and select agilex5_modkit_vvpisp.hps_first.hps.jic. The MT25QU02G device should be shown.
  • Check the Program/Configure box and press Start and wait until it completes (note that it can take several minutes).
• Power down the board.
• Set S4 on the SOM to ON-ON (enables booting from the microSD Card).

Power up the board and select the correct COMx port. Set up the serial terminal emulator using 115200 baud rate, 8 Data bits, 1 Stop bit, CRC and Hardware flow control disabled.

If you are using the .sof with OCP license flow, then you must program the .sof: (skip this step is you are using the .rbf with full license flow)

• If S4 on the SOM is NOT set to OFF-OFF, follow these additional steps:
  • Power down the board
  • Set S4 on the SOM to OFF-OFF. (This prevents the starting of any bootloader and FPGA configuration after power up and until the SOF is programmed over JTAG).
  • Power up the board.
• Use the following command:

quartus_pgm -c 1 -m jtag -o "p;fsbl_agilex5_modkit_vvpisp_time_limited.sof"

• Alternatively use the Quartus® Programmer GUI
  • Configure the JTAG Hardware by selecting the boards byteBlaster hardware. The Hardware Frequency should be 24MHz.
  • Auto Detect and select the A5EC065BB32AR0 device.
  • Change File and select fsbl_agilex5_modkit_vvpisp_time_limited.sof.
  • Check the Program/Configure box and press Start and wait until it completes.

The Linux OS will boot and the Camera Solution System Example Design Application should run automatically. A few seconds after Linux boots, the application will detect the attached Monitor and the output will be displayed using the best supported format. Take note of the board's IP address during boot. Connect your web browser to the boards IP address so you can interact with the Camera Solution System Example Design using the GUI.

Note that the boards IP address can also be found by using the terminal, logging in as root (no password required), and querying the Ethernet controller:

root
ip a

eth0 provides the IPv4 or IPv6 address to connect your web browser to. Examples of web browser URLs are http://192.168.0.1 for IPv4, and http://[fe80::a8bb:ccff:fe55:6688] for IPv6 (note the square brackets).

Resources

Full documentation for this Camera Solution System Example Design can be found on the Altera® FPGA Developer Site.