Low-Latency-CC-and-Video-Streaming-with-the-NVidia-Jetson-Nano-v0922.pdf

© 2020 Doodle Labs. All rights reserved. September 30, 2022
Page 1
Application Note
Low Latency C&C and Video Streaming with the
Nvidia Jetson Nano
Introduction
This document is a basic tutorial for how to get started with the Doodle Labs Smart Radio and
the Nvidia Jetson Nano in a video streaming application. Additionally, we demonstrate how the
Smart Radio con be configured to optimize a Command and Control while streaming video at
the same time. The Nvidia Jetson Nano is a popular System-on-Module (SOM) used for
emerging IoT applications such as drones, robots, and generally devices which can make use of
its powerful AI capabilities. It includes a GPU which can perform fast H.264/H.265 video
encoding and decoding making it ideal for low latency video streaming. This tutorial is divided
into the following sections:
1. Video System Block Diagram
2. Hardware Setup
3. First Time Jetson Nano Setup
4. Smart Radio Configuration
5. Preparing GStreamer
6. Video streaming and Command and Control
7. Advanced Video Streaming
This tutorial makes use of the gstreamer command-line tools, gst-launch-1.0 and gst-
inspect-1.0. Building a C-application is beyond the scope of the tutorial.
The latency added by the Smart Radio network is less than 10ms for both concurrent HD
video and the C&C data.

Page 2
Application Note
Video System Block Diagram
Fig. 1 – Video Data Flow
The block diagram above illustrates the data flow path of a simple streaming application. The
cloud of Smart Radios represents a mesh situation where the video feed may hop across a node
before arriving at the destination Smart Radio. For a discussion of the different blocks , please
see the document, “Smart Radio Video Streaming Tutorial”.
Hardware Setup
This tutorial makes use of the Nvidia Jetson Nano Developer Kit which has standard interfaces
like USB, HDMI etc. Basic hardware setup for streaming is shown below.

Page 3
Application Note
Fig. 2 – Basic Hardware Setup
Power
The Jetson Nano can be powered over the standard USB Micro device port (J28) or the barrel
connector (J25). Use a 5-V supply with at least a 2-A current rating. The camera is powered over
the MIPI CSI2 or USB interface. Follow the datasheet of your particular Smart Radio model
when choosing a power supply.
Camera Setup
You may use the MIPI CSI2 interface, or a USB interface. USB 2.0 has a maximum link speed of
480 Mbps which is unsuitable for raw 1080p30 video, which means that you will either need a
USB 3 interface, or you will need to transcode from MJPEG to H.264. Transconding MJPEG to
H264 adds about 100ms to the latency in our tests. The MIPI CSI2 interface is only suitable for
short connections, and ideally it should be shielded against EMI. Note that the HDMI port on
Nano is an output port, so you cannot connect an HDMI camera to it.
A global shutter is preferred since a rolling shutter produces a wavy effect when the UAV is
moving. Additionally, some cameras have some form of image stabilization (optical or
electronic), which is useful on UAVs.
We tested the control loop using the Nvidia Jetson Nano with two different cameras [1][2].
1. Raspberry Pi HQ Camera (imx477)
2. E-Con Systems See3CAM_CU135 - 4k USB3.0 Camera (AR1335)

Page 4
Application Note
Smart Radio Setup
In Fig. 2, we are using the OEM Mini form-factor of the Smart Radio on the UAV side, and the
Wearable form-factor on the GCS side. Refer to their respective integration guides for details on
the connections. Regardless of the form-factor, all Smart Radios include an Ethernet interface
which can be connected to the Jetson Nano on the UAV.
First Time Jetson Nano Setup
The first time you use the Jetson Nano, you may find it easier to get it setup using a monitor
and keyboard. Hook up your monitor to the HDMI port, and keyboard/mouse to the USB ports.
Getting the Firmware
There are extensive guides available on the Nvidia website which detail how to prepare your
Jetson Nano. The basic steps are:
1. Download and extract the latest firmware image. In this tutorial, we used JP 4.4.
2. Format your SD card and burn the firmware image to the SD card using a program such
as Balena Etcher.
3. Inset the SD card into the Jetson Nano, and power up.
The first time you boot up the Jetson Nano, you will be asked to setup a username and
password, and these will be required for SSH or Serial access later.
By default, the Jetson Nano is setup as a DHCP client. Therefore, you can connect it to your
office router so that it can get an IP address and access the internet. Once you have connected
to the internet, run
$ sudo apt update
$ sudo apt upgrade
This will make sure that your package lists and packages are up to date. After that make sure
that your Jetson clock is synchronized. This is necessary to access secure websites.
$ timedatectl
Local time: Tue 2020-05-05 16:23:05 +08
Universal time: Tue 2020-05-05 08:23:05 UTC
RTC time: Tue 2020-05-05 08:23:06
Time zone: Asia/Singapore (+08, +0800)
System clock synchronized: yes
systemd-timesyncd.service active: yes
RTC in local TZ: no

Page 5
Application Note
After the initial setup is complete, connect the Jetson Nano to the Smart Radio as shown in the
hardware diagram.
Smart Radio Configuration
In this section, we describe simple steps to setup the Smart Radio for video streaming. If you
are not sure how to make any of these configuration settings, then please consult the Smart
Radio Configuration Guide.
IP Configuration
The main requirement for the IP configuration is that all nodes in the network are on the same
subnet and can therefore reach one another directly. You may choose to enable a DHCP server
on the network, or just use static IP addresses.
The Jetson Nano is pre-configured as a DHCP client. However, by default, the Smart Radio does
not have the DHCP server enabled.
Enabling a DHCP Server
The DHCP server should be enabled on the WAN interface, and the default DHCP client should
be disabled at the same time.
1. In the GUI, navigate to Network → Interfaces and EDIT the WAN interface.
2. Change the protocol from DHCP Client to Static Address and click Switch Protcol
3. We need to choose an IP address and Netmask for this node. In the screenshot below,
we used 192.168.100.1 as the static IP address and 255.255.255.0 as the netmask.
4. Scroll to the bottom and under the DHCP Server section, deselect ignore
interface. The default settings are ok, so just click Save & Apply. At this point you
should make sure all nodes on the network are setup as DHCP clients. This is true by
default for the Smart Radio which are DHCP clients, but also have a fixed IP address.

Page 6
Application Note
Fig. 3 – DHCP Configuration
All nodes in the network should now have an IP address in the 192.168.100.0/24 subnet. You
can find the IP addresses of all nodes on the network by logging into the Smart Radio over SSH
and using the address resolution protocol. You can identify connected devices by their
hardware address.

Page 7
Application Note
root@smartradio-301a4ebb02:~# arp
IP address HW type Flags HW address Mask Device
192.168.100.210 0x1 0x2 b0:25:aa:2d:d3:8e * br-wan
192.168.100.185 0x1 0x2 00:30:1a:4e:aa:01 * br-wan
192.168.100.193 0x1 0x2 00:30:1a:4e:aa:09 * br-wan
Note that the above steps apply for both WDS AP/Client and Mesh modes.
Traffic Prioritization
Figure 4 shows the configuration menu for Differentiated Services. This page can be found by
navigating to network → Traffic Prioritization in the web GUI. Traffic optimization
works by filtering packets based on their network port, IP address or transport layer protocol
and placing them in one of four different queues – best effort, command/control and voice,
video, and background. Doodle Labs Smart Radios include additional optimizations for video
and command/control data which can be enabled by checking the relevant radio buttons in the
Traffic Prioritization configuration menu.
The video bad link threshold is an additional failsafe where the video stream is guaranteed to
be dropped if the signal strength is lower than the defined threshold. The default numbers
make it disabled.
You may also enable Diversity Rates Only which limits the radio’s modulation rate to slower but
more robust rates. This is recommended for highly dynamically changing channel conditions or
UAV movement.
Lastly, make sure to create a rule for your RC/telemetry connection like the ones already
defined. By default, port 14550 (MAVLink) is sent to the command and control queue. You do
not need to create a rule for your video stream.
After making your changes, click Save & Apply.

Page 8
Application Note
Fig. 4 – Differentiated Services Configuration Menu
Preparing GStreamer
GStreamer is a framework for creating multimedia streaming applications and is available in
multiple platforms including Windows, iOS, Android, and Linux [3]. GStreamer is installed in the
Jetson Nano by default and you can write simple pipelines to steam video without any
additional setup. This guide focuses on using RTSP streaming, which is commonly used for real-
time streaming applications.
In order to stream using RTSP, you either need to write your own application, or use gst-rtsp-
server [4]. gst-rtsp-server requires the gtk-doc-tools package to be installed.
$ sudo apt install gtk-doc-tools
In order to use gst-rtsp-server, you need to clone the repository, checkout the version of
gst-rtsp-server suitable for your GStreamer version, and the build the application. Start by
creating a working directory.
$ mkdir workingDir
$ cd workingDir
$ git clone https://github.com/GStreamer/gst-rtsp-server.git

Page 9
Application Note
$ cd gst-rtsp-server
$ gst-launch-1.0 –version
GStreamer 1.14.5
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0
The version of GStreamer we have is 1.14.5, so checkout the corresponding git branch.
$ git checkout 1.14.5
$ ./autogen.sh
$ ./configure.sh
$ make
$ sudo make install
gst-rtsp-server is now ready to be used.
Video Streaming and Command and Control
Before starting a video stream, first get the information on your video camera’s capabilities.
You can list the cameras attached to the Jetson Nano and check their capabilities using v4l2-
ctl.
$ v4l2-ctl --list-devices
vi-output, imx219 6-0010 (platform:54080000.vi:0):
/dev/video0
$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'RG10'
Name : 10-bit Bayer RGRG/GBGB
Size: Discrete 3264x2464
Interval: Discrete 0.048s (21.000 fps)
In our case, we have one camera which is attached and it is exposed to the user as
/dev/video0. Our tests will be conducted using 1920x1080 at 30 fps. Fast encoding at H.264
can be accomplished using the omxh264enc plugin. You can see details and options of the
omxh264enc plugin using
$ gst-inspect-1.0 omxh264enc

Page 10
Application Note
The output is very long and is not shown. The equivalent H.265 encoder plugin is omxh265enc.
Nvidia has an Accelerated GStreamer User Guide available online which details some of the
capabilities of the Jetson Nano when used with GStreamer.
RTP Streaming
Before diving into RTSP streaming, we can test a simple RTP video stream. In this case, the
Nvidia Jetson Nano will act as the client, and it will directly stream video packets to a known
listening server.
The RTP client can be started on the Jetson Nano using
$ gst-launch-1.0 nvarguscamerasrc ! "video/x-
raw(memory:NVMM)",width=1920,height=1080,framerate=30/1,format=NV12 !
videoconvert ! omxh264enc control-rate=constant bitrate=5000000
iframeinterval=15 ! h264parse ! rtph264pay name=pay0 pt=96 config-interval=-
1 ! udpsink host=<IP ADDRESS> port=5000 sync=false
where <IP Address> is the IP address of the receiving PC. The video feed can be picked up on
the receiving PC with
$ gst-launch-1.0 -v udpsrc port=5000 caps='application/x-rtp,
media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264,
framerate=(fraction)30/1, width=(string) 1920, height=(string) 1080,
playload=(int)96' ! rtpjitterbuffer latency=100 ! rtph264depay ! avdec_h264 !
videoconvert ! autovideosink sync=false
All instances of h264 above also work with h265.
RTSP Streaming
RTSP streaming can be started using
$ ./gst-rtsp-server/examples/test-launch "nvarguscamerasrc ! video/x-
raw(memory:NVMM) width=1920 height=1080 framerate=30/1 format=NV12 !
omxh264enc control-rate=constant bitrate=5000000 iframeinterval=15!
h264parse ! rtph264pay name=pay0 pt=96 config-interval=-1"
Note that the command points to the gst-rtsp-server directory which was cloned earlier.
The stream can be picked up on the receiving PC using
$ gst-launch-1.0 -v rtspsrc buffer-mode=0 do-retransmissions=0 drop-on-
latency=1 latency=100 location=rtsp://<IP Address>:8554/test ! application/x-
rtp, payload=96 ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink
sync=false

Page 11
Application Note
where <IP Address> is the IP address of the Jetson Nano. All instances of h264 above also
work with h265. In the command above we use UDP as the transport protocol. To use TCP, the
“location=rtsp://” part should be changed to “location=rtspt://”.
QGroundcontrol
Important Note: It is not possible to change the RTSP client settings in QGroundControl, and
we have found that many of the default settings are not suitable for real-time video
streaming. If you are using QGroundControl and are experiencing video issues, then we
recommend avoiding RTSP, and using the simple RTP streaming discussed in the section
above.
The RTSP stream can be picked up using QGroundcontrol with either H.264 or H.265 encoding.
Figure 5 shows the relevant settings, and Fig. 6 shows a screenshot of the video feed with
2Mbps H.265 encoding. Note that QGroundcontrol does not support RTSP over TCP by default.
Fig. 5 – QGroundcontrol RTSP settings

Page 12
Application Note
Fig – 6 RTSP Stream in QGC 2Mbps H.265 Encoding
Results
With the above settings, we measured the glass-to-glass latency while running hping3 at the
same time. hping3 is a Linux command-line utility which can be used to measure UDP latency
(among other things). You can install and run hping3 by running
$ sudo apt install hping3
$ sudo hping3 --data 500 --destport 14550 10.223.0.2
HPING 10.223.0.2 (enp4s0 10.223.0.1): NO FLAGS are set, 40 headers + 500 data
bytes
len=40 ip=10.223.0.2 ttl=64 DF id=43729 sport=14550 flags=RA seq=0 win=0
rtt=3.8 ms
With the above settings, the glass-to-glass latency was typically 110ms with only about 10ms
for the transport through the Smart Radios. The difference between TCP and UDP was around 3
ms. Figure 7 shows the results.
Note that Doodle Labs Mesh Rider uses special radio and parameters to optimize the video
transmission over wireless medium in high interference areas. For video transmission within
Smart Radio private network, we recommend use of TCP. We can see that the latency added
by the Smart Radio network amounted to less than around 10ms.

Page 13
Application Note
Fig. 7 – Glass to glass latency
Advanced Video Streaming
In this section, we will cover some advanced video streaming protocols which can improve
video performance in various adverse conditions. We present some example gstreamer
pipelines as-is, but bear in mind that many of these may not be integrated into applications
such as QGroundControl.
Pipeline Configuration
The RTSP pipelines which we used above can be tailored for different applications. Some of the
settings which are useful to configure in highly dynamic applications (UAVs) are discussed
below.
1. Encoder Bitrate – The encoder bitrate controls the bandwidth used to send the video.
Any wireless link has a limited network capacity, and if the bitrate exceeds this capacity,
frames will be dropped.
2. Encoder i-frame interface – The i-frame interval determines how often key interval
frames are sent. In H264/H265 encoding, image frames are compressed in image-space
and time. I-frames, however, are not compressed in time. In contrast to i-frames, p-
frames and b-frames require information from previous and future frames to be
decoded, so if an i-frame is dropped, all subsequent frames until the next i-frame could
be affected.

Page 14
Application Note
a. We recommend keeping a very low i-frame interval for highly dynamic channel
conditions. This will result in a general degradation in picture quality, but fewer
frame drops.
3. Intra-refresh type – Some encoders allow you to divide i-frames into multiple sub-
frames. Rather than encoding the entire i-frame at once, the frame is split into different
regions, and the i-frame of each region is encoded at cyclically. Since i-frames are
significantly larger than p/b-frames, this has the effect of making the bit-stream more
stable which is easier on the radio link.
4. Region-of-interest – Some encoders allow you too choose a region-of-interest within the
frame. Usually is is a rectangle in the center of the frame, and it will be encoded at a
higher resolution than the rest of the frame.
Forward-Error-Correction
Forward-Error-Correction (FEC) is an encoding method whereby data is sent redundantly so
that missing or erroneous data can be recovered at the receiver. FEC is already used at the bit
level by the radio, but it can be added at either the transport or application level. Gstreamer
integrates the FEC encoder/decoder elements gstfeculpenc and gstfeculpdec [5][6].
An example RTP sending pipeline using the raspberry pi is
gst-launch-1.0 --gst-debug-level=3 rpicamsrc bitrate=2000000 exposure-
mode=sports awb-mode=1 keyframe-interval=15 rotation=180 preview=false
sensor-mode=5 ! video/x-h264,width1280,height=720,framerate=30/1 !
h264parse ! rtph264pay config-interval=-1 name=pay0 pt=96 ! rtpulpfecenc
percentage=100 pt=122 ! udpsink host=10.223.0.1 port=5000
An example RTP receiving pipeline is
gst-launch-1.0 -v udpsrc port=5000 caps='application/x-rtp,
media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264,
framerate=(fraction)30/1, width=(string)1920, height=(string)1080,
playload=(int)96' ! rtpulpfecdec pt=122 ! rtph264depay ! avdec_h264 !
videoconvert ! autovideosink sync=false
MTU Sizing
The Maximum Transmission Unit (MTU) is the largest data packet that the network will support.
Reducing the MTU size used by the video streaming application will result in smaller over-the-
air packets. A smaller packet is in the air for a shorter duration and therefore is less likely to be
corrupted. However, the MTU size also directly affects the network capacity. As each packet in
a unicast transmission needs to be acknowledged, it is less efficient to send smaller packets and
then wait for an acknowledgement.

Page 15
Application Note
In GStreamer, you can configure the MTU in an RTSP stream using the rtph264pay element.
An example configuration would be
rtph264pay config-interval=-1 name=pay0 pt=96 mtu=250
Adaptive Bitrates
The network capacity is the throughput that the network can support at any particular time. In
a UAV scenario, the network capacity can change throughout the mission. The most obvious
case is where the UAV is flying away from the GCS, which results in a steady drop in the
network capacity over time. Using dynamically adaptive bitrates to modify the video encoder’s
bitrate based on the network capacity can result in a stable video feed even in harsh RF
environments or at long range.
References
[1] Raspberry Pi HQ Camera, https://www.raspberrypi.com/products/raspberry-pi-high-
quality-camera/, 18-8-2022
[2] E-Con Systems 4K USB Camera, https://www.e-consystems.com/4k-usb-camera.asp, 18-
8-2022
[3] Gstreamer, https://gstreamer.freedesktop.org/, 18-8-2022
[4] gst-rtsp-server, https://github.com/GStreamer/gst-rtsp-server, 18-8-2022
[5] Gstreamer gstulpfecenc,
https://gstreamer.freedesktop.org/documentation/rtp/rtpulpfecenc.html?gi-
language=c, 26-7-2022
[6] Gstreeamer gstulpfecdec,
https://gstreamer.freedesktop.org/documentation/rtp/rtpulpfecdec.html?gi-
language=c, 26-7-2022

Low-Latency-CC-and-Video-Streaming-with-the-NVidia-Jetson-Nano-v0922.pdf

More Related Content

Recently uploaded (20)

Featured (20)

Low-Latency-CC-and-Video-Streaming-with-the-NVidia-Jetson-Nano-v0922.pdf