Difference between revisions of "Edge AI SDK/AI Framework/Nvidia x86 64"

From ESS-WIKI
Jump to: navigation, search
(Created page with " = RTX-A5000 AI Suite rc = <span style="font-size:larger;">[https://hailo.ai/products/hailo-software/hailo-ai-software-suite/#sw-overview Hailo AI Software Suite] offers brea...")
 
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  
−
= RTX-A5000 AI Suite rc =
+
= DeepStream =
  
−
<span style="font-size:larger;">[https://hailo.ai/products/hailo-software/hailo-ai-software-suite/#sw-overview Hailo AI Software Suite] offers breakthrough AI accelerators and Vision processors uniquely designed to accelerate embedded deep learning applications on edge devices.</span>
+
DeepStream&nbsp;is a complete streaming analytics toolkit based on GStreamer for AI-based multi-sensor processing, video, audio, and image understanding. It’s ideal for vision AI developers, software partners, startups, and OEMs building IVA apps and services. Developers can now create stream processing pipelines that incorporate neural networks and other complex processing tasks such as tracking, video encoding/decoding, and video rendering. DeepStream pipelines enable real-time analytics on video, image, and sensor data.
  
−
<span style="font-size:larger;">Hailo devices are accompanied by a comprehensive AI SDK that enables the compilation of deep learning models and the implementation of AI applications in production environments. The model build environment seamlessly integrates with common ML frameworks to allow smooth and easy integration in existing development ecosystems. The runtime environment enables integration and deployment in host processors, such as x86 and ARM based products, when utilizing Hailo-8, and in Hailo-15 vision processor.</span>
+
DeepStream’s multi-platform support gives you a faster, easier way to develop vision AI applications and services. You can even deploy them on-premises, on the edge, and in the cloud with the click of a button.&nbsp;
  
−
&nbsp;
+
More Info refer to&nbsp;[https://developer.nvidia.com/deepstream-sdk https://developer.nvidia.com/deepstream-sdk]
  
−
&nbsp;
+
=== DeepStream 6.4 Prerequisites: ===
  
−
= Applications =
+
You must install the following components:
  
−
<span style="font-size:larger;">[https://hailo.ai/products/hailo-software/hailo-ai-software-suite/#sw-tappas TAPPAS] is a solution designed to streamline the development and deployment of edge applications demanding high AI performance. This reference application software package empowers users to expedite their time-to-market by minimizing the development workload. TAPPAS encompasses a user-friendly set of fully operational application examples based on GStreamer, featuring pipeline elements and pre-trained AI tasks. These examples leverage advanced Deep Neural Networks, highlighting Hailo's AI processors' top-notch throughput and power efficiency. Furthermore, TAPPAS serves as a demonstration of Hailo's system integration capabilities, showcasing specific use cases on predefined software and hardware platforms. Utilizing TAPPAS simplifies integration with Hailo's runtime software stack and offers a starting point for users to fine-tune their applications. By demonstrating Hailo's system integration scenarios on both predefined software and hardware platforms, it can be used for evaluations, reference code, and demos. This approach effectively accelerates time to market, streamlines integration with Hailo's runtime software stack, and provides customers with a foundation to fine-tune their applications.</span>
+
*Ubuntu 22.04
 +
*GStreamer 1.20.3
 +
*Nvidia Driver R535.104.12
 +
*CUDA 12.2
 +
*TensorRT 8.6.1.6
  
−
<span style="font-size:larger;">Refer to [https://github.com/hailo-ai/tappas github-TAPPAS]</span>
+
= TensorRT =
  
−
&nbsp;
+
[https://developer.nvidia.com/tensorrt TensorRT]&nbsp;is a high performance deep learning inference runtime for image classification, segmentation, and object detection neural networks. TensorRT is built on CUDA, NVIDIA’s parallel programming model, and enables you to optimize inference for all deep learning frameworks. It includes a deep learning inference optimizer and runtime that delivers low latency and high-throughput for deep learning inference applications.
  
 
&nbsp;
 
&nbsp;
  
−
== <span style="font-size:larger;">Edge AI SDK&nbsp;/ Application</span> ==
+
= <span style="font-size:larger;">Edge AI SDK&nbsp;/ Application</span> =
  
−
<span style="font-size:larger;">Quick Start / Application / Video or WebCam / VPU</span>
+
<span style="font-size:larger;">Quick Start (Vision) / Application / Video or WebCam / dGPU</span>
  
−
<span style="font-size:larger;">[[File:Objdet.png|800x450px|ObjectDetection]]</span>
+
[[File:EdgeAISDK rtxa5000.png|800x450px|EdgeAISDK rtxa5000.png]]
  
 
<span style="font-size:larger;">&nbsp;</span>
 
<span style="font-size:larger;">&nbsp;</span>
Line 34: Line 38:
 
|-
 
|-
 
| style="width: 154px;" | <span style="font-size:larger;">Object Detection</span>
 
| style="width: 154px;" | <span style="font-size:larger;">Object Detection</span>
−
| style="width: 179px;" | <span style="font-size:larger;">yolov5m_wo_spp_60p.hef</span>
+
| style="width: 179px;" | yolov3.weights
 
|-
 
|-
 
| style="width: 154px;" | <span style="font-size:larger;">Person Detection</span>
 
| style="width: 154px;" | <span style="font-size:larger;">Person Detection</span>
−
| style="width: 179px;" | <span style="font-size:larger;">yolov5s_personface_reid.hef / repvgg_a0_person_reid_2048.hef</span>
+
| style="width: 179px;" | sample_ssd_relu6.uff
 
|-
 
|-
 
| style="width: 154px;" | <span style="font-size:larger;">Face Detection</span>
 
| style="width: 154px;" | <span style="font-size:larger;">Face Detection</span>
−
| style="width: 179px;" | <span style="font-size:larger;">arcface_mobilefacenet_v1.hef / scrfd_10g.hef</span>
+
| style="width: 179px;" | facenet.etlt
 
|-
 
|-
 
| style="width: 154px;" | <span style="font-size:larger;">Pose Estimation</span>
 
| style="width: 154px;" | <span style="font-size:larger;">Pose Estimation</span>
−
| style="width: 179px;" | <span style="font-size:larger;">centerpose_regnetx_1.6gf_fpn.hef</span>
+
| style="width: 179px;" | model.etlt
 
|}
 
|}
  
 
= <span style="font-size:larger;">Benchmark</span> =
 
= <span style="font-size:larger;">Benchmark</span> =
  
−
<span style="font-size:larger;"><span style="font-size:larger;">In order to measure FPS, power and latency of the Hailo Model Zoo networks you can use the HailoRT command line interface. For more information please refer to the HailoRT documentation in [https://github.com/hailo-ai/hailo_model_zoo/blob/master/docs/BENCHMARKS.rst link].</span></span>
+
<span style="font-size:larger;"><span style="font-size:larger;">In order to measure FPS, power and latency of the RTX-A5000 you can use&nbsp;the&nbsp;command "trtexec" . For more information please refer to the ''trtexec''&nbsp;documentation in [https://github.com/NVIDIA/TensorRT/tree/main/samples/trtexec link].</span></span>
  
 
<span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span>
 
<span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span>
Line 56: Line 60:
 
&nbsp;
 
&nbsp;
  
−
== <span style="font-size:larger;"><span style="font-size:larger;">Hailo-8 Benchmark</span></span> ==
+
== <span style="font-size:larger;"><span style="font-size:larger;">RTX-A5000 Benchmark</span></span> ==
 +
<pre>trtexec --loadEngine=models/model_fp16.engine --batch=16</pre>
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">hailortcli benchmark <Hailo's Model .hef ></span></span></span>
+
[[File:EdgeAISDK rtxa5000 trtexec.png|1000x300px|EdgeAISDK rtxa5000 trtexec.png]]
−
<pre>hailortcli benchmark resnet_v1_18.hef</pre>
 
−
 
 
−
<span style="font-size:larger;"><span style="font-size:larger;">[[File:Hailortcli benchmark.png|800x450px|benchmark]]</span></span>
 
  
 
<span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span>
 
<span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span>
Line 69: Line 71:
 
== <span style="font-size:larger;"><span style="font-size:larger;">Edge AI SDK&nbsp;/ Benchmark</span></span> ==
 
== <span style="font-size:larger;"><span style="font-size:larger;">Edge AI SDK&nbsp;/ Benchmark</span></span> ==
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Evaluate the Hailo-8&nbsp;performance&nbsp;with Edge AI SDK.</span></span></span>
+
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Evaluate the RTX-A5000&nbsp;performance&nbsp;with Edge AI SDK.</span></span></span>
  
−
<span style="font-size:larger;"><span style="font-size:larger;">[[File:Benchmark.png|800x500px|Benchmark]]</span></span>
+
[[File:EdgeAISDK rtxa5000 benchmark.png|800x500px|EdgeAISDK rtxa5000 benchmark.png]]
−
 
 
−
<span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span>
 
  
 
&nbsp;
 
&nbsp;
  
−
= <span style="font-size:larger;"><span style="font-size:larger;">HailoRT CLI</span></span> =
+
= <span style="font-size:larger;">NVIDIA System Management Interface</span> =
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">HailoRT CLI - a command line application used to control the Hailo device, run inferences, collect statistics and device events, etc. Use "--help" to exhibit more usages.</span></span></span>
+
<span style="font-size:larger;">The NVIDIA System Management Interface (nvidia-smi) is a command line utility, based on top of the&nbsp;[https://developer.nvidia.com/nvidia-management-library-nvml NVIDIA Management Library (NVML)], intended to aid in the management and monitoring of NVIDIA GPU devices.&nbsp;</span>
−
<pre>hailortcli --help</pre>
 
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Refer to [https://github.com/hailo-ai/hailort/tree/master github-hailort]</span></span></span>
+
<span style="font-size:larger;">This utility allows administrators to query GPU device state and with the appropriate privileges, permits administrators to modify GPU device state.</span>
 +
<pre>nvidia-smi
 +
</pre>
  
 
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span></span>
 
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span></span>
Line 90: Line 91:
 
&nbsp;
 
&nbsp;
  
−
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Hailo-8 Utilization</span></span></span> ==
+
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">RTX-A5000 Utilization</span></span></span> ==
−
<pre>hailortcli monitor</pre>
+
<pre>nvidia-smi</pre>
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">[[File:EdgeAISDK Hailo Usage.jpg|800x400px|Hailo VPU Usage]]</span></span></span>
+
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">[[File:EdgeAISDK rtxa5000 utility.png|800x400px|EdgeAISDK rtxa5000 utility.png]]</span></span></span>
  
 
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span></span>
 
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp;</span></span></span>
Line 99: Line 100:
 
&nbsp;
 
&nbsp;
  
−
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Hailo-8 Temperature</span></span></span> ==
+
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">RTX-A5000 Temperature</span></span></span> ==
−
<pre>python ts_monitoring_2023Sep07.py</pre>
+
<pre>nvidia-smi</pre>
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp; [[File:Thermal in hailortcli.png|800x150px|Thermal in hailortcli.png]]</span></span></span>
+
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">&nbsp;&nbsp;[[File:EdgeAISDK rtxa5000 thermal.png|800x400px|EdgeAISDK rtxa5000 thermal.png]]</span></span></span>
  
 
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Edge AI SDK&nbsp;/&nbsp;Monitoring</span></span></span> ==
 
== <span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">Edge AI SDK&nbsp;/&nbsp;Monitoring</span></span></span> ==
  
−
<span style="font-size:larger;"><span style="font-size:larger;"><span style="font-size:larger;">[[File:Thermal.png|800x450px|thermal]]</span></span></span>
+
[[File:EdgeAISDK rtxa5000 UI.png|800x450px|EdgeAISDK rtxa5000 UI.png]]
−
 
 
−
[[Category:Editor]]
 

Latest revision as of 09:45, 2 July 2024

DeepStream

DeepStream is a complete streaming analytics toolkit based on GStreamer for AI-based multi-sensor processing, video, audio, and image understanding. It’s ideal for vision AI developers, software partners, startups, and OEMs building IVA apps and services. Developers can now create stream processing pipelines that incorporate neural networks and other complex processing tasks such as tracking, video encoding/decoding, and video rendering. DeepStream pipelines enable real-time analytics on video, image, and sensor data.

DeepStream’s multi-platform support gives you a faster, easier way to develop vision AI applications and services. You can even deploy them on-premises, on the edge, and in the cloud with the click of a button. 

More Info refer to https://developer.nvidia.com/deepstream-sdk

DeepStream 6.4 Prerequisites:

You must install the following components:

  • Ubuntu 22.04
  • GStreamer 1.20.3
  • Nvidia Driver R535.104.12
  • CUDA 12.2
  • TensorRT 8.6.1.6

TensorRT

TensorRT is a high performance deep learning inference runtime for image classification, segmentation, and object detection neural networks. TensorRT is built on CUDA, NVIDIA’s parallel programming model, and enables you to optimize inference for all deep learning frameworks. It includes a deep learning inference optimizer and runtime that delivers low latency and high-throughput for deep learning inference applications.

 

Edge AI SDK / Application

Quick Start (Vision) / Application / Video or WebCam / dGPU

EdgeAISDK rtxa5000.png

 

Application Model
Object Detection yolov3.weights
Person Detection sample_ssd_relu6.uff
Face Detection facenet.etlt
Pose Estimation model.etlt

Benchmark

In order to measure FPS, power and latency of the RTX-A5000 you can use the command "trtexec" . For more information please refer to the trtexec documentation in link.

 

 

 

RTX-A5000 Benchmark

trtexec --loadEngine=models/model_fp16.engine --batch=16

EdgeAISDK rtxa5000 trtexec.png

 

 

Edge AI SDK / Benchmark

Evaluate the RTX-A5000 performance with Edge AI SDK.

EdgeAISDK rtxa5000 benchmark.png

 

NVIDIA System Management Interface

The NVIDIA System Management Interface (nvidia-smi) is a command line utility, based on top of the NVIDIA Management Library (NVML), intended to aid in the management and monitoring of NVIDIA GPU devices. 

This utility allows administrators to query GPU device state and with the appropriate privileges, permits administrators to modify GPU device state. 

nvidia-smi

 

 

 

RTX-A5000 Utilization

nvidia-smi

EdgeAISDK rtxa5000 utility.png

 

 

RTX-A5000 Temperature

nvidia-smi

  EdgeAISDK rtxa5000 thermal.png

Edge AI SDK / Monitoring

EdgeAISDK rtxa5000 UI.png

Retrieved from "https://ess-wiki.advantech.com.tw/wiki/index.php?title=Edge_AI_SDK/AI_Framework/Nvidia_x86_64&oldid=36124"