Embedded Linux Development

SI/SW
2024/9/6

© 2024 Realtek Semiconductor Corp. All rights reserved

© 2024 Realtek Semiconductor Corp. All rights reserved 1
Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 2

Birth of Free Software
 1983, Richard Stallman, GNU project and the free software concept. Beginning of the
development of gcc, gdb, glibc and other important tools
 1991, Linus Torvalds, Linux kernel project, a UNIX-like operating system kernel.
Together with GNU software and many other open-source components: a completely
free operating system, GNU/Linux

© 2024 Realtek Semiconductor Corp. All rights reserved 3

What is embedded Linux?
 Embedded Linux is the usage of the Linux kernel and various open-source
components in embedded systems
 Ability to reuse components
 Low cost
 Full control
 Easy testing of new features
 Quality
 Security
 Community support
 Participation in community work

© 2024 Realtek Semiconductor Corp. All rights reserved 4

A few examples

© 2024 Realtek Semiconductor Corp. All rights reserved 5

A few examples

© 2024 Realtek Semiconductor Corp. All rights reserved 6

Embedded Linux System Architecture

© 2024 Realtek Semiconductor Corp. All rights reserved 7

Software components
 Cross-compilation toolchain
 Compiler that runs on the development machine, but generates code for the target
 Bootloader
 Started by the hardware, responsible for basic initialization, loading and executing the kernel
 Linux Kernel
 Contains the process and memory management, network stack, device drivers and provides services to
user space applications
 C library
 Of course, a library of C functions
 Also the interface between the kernel and the user space applications

 Libraries and applications

 Third-party or in-house

© 2024 Realtek Semiconductor Corp. All rights reserved 8

Embedded Linux work
 Several distinct tasks are needed when deploying embedded Linux in a product:
 Board Support Package development
 A BSP contains a bootloader and kernel with the suitable device drivers for the targeted
hardware
 System integration
 Integrate all the components, bootloader, kernel, third-party libraries and applications and
in-house applications into a working system
 Development of applications
 Normal Linux applications

© 2024 Realtek Semiconductor Corp. All rights reserved 9

OS for Linux development
 We strongly recommend to use GNU/Linux as the desktop operating system to
embedded Linux developers, for multiple reasons.
 All community tools are developed and designed to run on Linux. Trying to use them on
other operating systems (Windows, macOS) will lead to trouble.
 As Linux also runs on the embedded device, all the knowledge gained from using Linux on
the desktop will apply similarly to the embedded device.

© 2024 Realtek Semiconductor Corp. All rights reserved 10

Host vs. target
 When doing embedded development, there is always a split between
 The host, the development workstation, which is typically a powerful PC
 The target, which is the embedded system under development
 They are connected by various means: almost always a serial line for debugging
purposes, frequently a networking connection, sometimes a JTAG interface for
low-level debugging

© 2024 Realtek Semiconductor Corp. All rights reserved 11

Toolchain

© 2024 Realtek Semiconductor Corp. All rights reserved 12

Architecture tuple and toolchain prefix
 Many UNIX/Linux build mechanisms rely on architecture tuple names to identify
machines.
 arm-linux-gnueabihf, mips64el-linux-gnu, arm-vendor-none-eabihf  cross compiler
 gcc  native compiler
 These tuples are 3 or 4 parts:
 The architecture name: arm, riscv, mips64el, etc.
 Optionally, a vendor name, which is a free-form string
 An operating system name, or none when not targeting an operating system
 The ABI/C library

© 2024 Realtek Semiconductor Corp. All rights reserved 13

Components of gcc toolchains

© 2024 Realtek Semiconductor Corp. All rights reserved 14

Binutils
 Binutils is a set of tools to generate and manipulate binaries (usually with the ELF
format) for a given CPU architecture
 as, the assembler, that generates binary code from assembler source code
 ld, the linker
 ar, ranlib, to generate .a archives (static libraries)
 objdump, readelf, size, nm, strings, to inspect binaries. Very useful analysis tools!
 objcopy, to modify binaries
 strip, to strip parts of binaries that are just needed for debugging (reducing their size).

© 2024 Realtek Semiconductor Corp. All rights reserved 15

C/C++ compiler
 GCC: GNU Compiler Collection, the famous free software compiler
 https://gcc.gnu.org/
 Can compile C, C++, Ada, Fortran, Java, Objective-C, Objective-C++, Go, etc. Can
generate code for a large number of CPU architectures, including x86, ARM,
RISC-V, and many others
 Available under the GPL license, libraries under the GPL with linking exception.

© 2024 Realtek Semiconductor Corp. All rights reserved 16

Kernel headers
 The C standard library and compiled programs need to
interact with the kernel
 Available system calls and their numbers
 Constant definitions
 Data structures, etc.
 Therefore, compiling the C standard library requires
kernel headers, and many applications also require them
 Available in <linux/...> and <asm/...> and a few other
directories corresponding to the ones visible in
include/uapi/ and in arch/<arch>/include/uapi in the
kernel sources
 The kernel headers are extracted from the kernel sources
using the headers_install kernel Makefile target

© 2024 Realtek Semiconductor Corp. All rights reserved 17

C standard library
 The C standard library is an essential component
of a Linux system.
 Interface between the applications and the kernel
 Provides the well-known standard C API to ease
application development
 Several C standard libraries are available: glibc,
uClibc, musl, klibc, newlib...
 The choice of the C standard library must be made
at cross-compiling toolchain generation time, as
the GCC compiler is compiled against a specific C
standard library.

© 2024 Realtek Semiconductor Corp. All rights reserved 18

An alternate compiler suite: LLVM
 The LLVM project has been developing an alternative compiler suite:
 Clang, C/C++ compiler, https://clang.llvm.org/
 LLDB, debugger, https://lldb.llvm.org/
 LLD, linker, https://lld.llvm.org/
 and more, see https://llvm.org/
 Available under MIT/BSD licenses

© 2024 Realtek Semiconductor Corp. All rights reserved 19

ABI
 ABI, for Application Binary Interface, defines
the calling conventions (how function
arguments are passed, how the return value is
passed, how system calls are made) and the
organization of structures (alignment, etc.)
 On ARM 32-bit, two main ABIs: EABI and
EABIhf
 EABIhf passes floating-point arguments in
floating-point registers → needs an ARM
processor with a FPU
 On RISC-V, several ABIs: ilp32, ilp32f, ilp32d,
lp64, lp64f, and lp64d

© 2024 Realtek Semiconductor Corp. All rights reserved 20

Floating point support
 For processors with a floating point unit, sometimes different FPU are possible.
For example on ARM: VFPv3, VFPv3-D16, VFPv4, VFPv4-D16, etc.
 For processors without a floating point unit, two solutions for floating point
computation:
 Generate hard float code and rely on the kernel to emulate the floating point instructions.
This is very slow
 Generate soft float code, so that instead of generating floating point instructions, calls to a
user space library are generated

© 2024 Realtek Semiconductor Corp. All rights reserved 21

CPU optimization flags
 GNU tools (gcc, binutils) can only be compiled for a specific target architecture at
a time (ARM, x86, RISC-V...)
 gcc offers further options:
 -march allows to select a specific target instruction set
 -mtune allows to optimize code for a specific CPU
 For example: -march=armv7 -mtune=cortex-a8
 -mcpu=cortex-a8 can be used instead to allow gcc to infer the target instruction set (-
march=armv7) and cpu optimizations (-mtune=cortex-a8)
 https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html
 Note: LLVM (Clang, LLD...) utilities support multiple target architectures at once.

© 2024 Realtek Semiconductor Corp. All rights reserved 22

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 23

Bootloader role
 The bootloader is a piece of code responsible for
 Basic hardware initialization
 Loading of an application binary, usually an operating system kernel, from flash storage,
from the network, or from another type of non-volatile storage
 Possibly decompression of the application binary
 Execution of the application
 Besides these basic functions, most bootloaders provide a shell or menu
 Menu to select the operating system to load
 Shell with commands to load data from storage or network, inspect memory, perform
hardware testing/diagnostics

© 2024 Realtek Semiconductor Corp. All rights reserved 24

Booting on x86 platforms: Legacy BIOS
 Basic Input Output System

© 2024 Realtek Semiconductor Corp. All rights reserved 25

Booting on x86 platforms: UEFI
 Unified Extensible Firmware Interface

© 2024 Realtek Semiconductor Corp. All rights reserved 26

GRUB
 Grand Unified Bootloader, from the GNU
project
 De-facto standard in most Linux
distributions for x86 platforms
 Supports x86 legacy and UEFI systems
 Also supports ARM, ARM64, RISC-V,
PowerPC, but less popular than other
bootloaders on those platforms

© 2024 Realtek Semiconductor Corp. All rights reserved 27

ACPI
 Advanced Configuration and Power Interface
 Open standard that operating systems can use to discover and configure
computer hardware components, to perform power management, to perform
auto configuration, and to perform status monitoring
 Tables with descriptions of the hardware that cannot be dynamically discovered at
runtime
 Tables provided by the firmware (UEFI or legacy) and used by the operating
system (Linux kernel in our case)

© 2024 Realtek Semiconductor Corp. All rights reserved 28

UEFI and ACPI on ARM
 Historically UEFI and ACPI are technologies coming from the Intel/x86 world
 ARM is also pushing for the adoption of UEFI and ACPI as part of its ARM System
Ready certification
 Mainly for servers/workstations SoCs
 Does not impact embedded SoCs
 Also some on-going effort to use UEFI on RISC-V, but not the de-facto standard

© 2024 Realtek Semiconductor Corp. All rights reserved 29

Booting on embedded platforms

© 2024 Realtek Semiconductor Corp. All rights reserved 30

U-Boot
 The de-facto standard and most widely used bootloader on embedded
architectures: ARM, ARM64, RISC-V, PowerPC, MIPS, and more.
 License: GPLv2 (same as Linux)
 Follows a regular release schedule. Every 2 or 3 months, a new version is
released. Versions are named YYYY.MM.

© 2024 Realtek Semiconductor Corp. All rights reserved 31

Trusted firmware
 Modern SoCs have advanced security mechanisms that require running some sort
of trusted firmware
 This firmware is loaded by the bootloader, or part of the boot chain itself
 This trusted firmware stays resident after control has been passed to the OS
 It is stored in a dedicated portion of the DDR, or some specific SRAM, inaccessible from the
OS
 It provides services to the OS, which the OS cannot perform directly
 Can also be responsible for running a secure OS alongside the regular OS

© 2024 Realtek Semiconductor Corp. All rights reserved 32

ARM exception levels and worlds

Secure EL2 from ARMv8.4-A

© 2024 Realtek Semiconductor Corp. All rights reserved 33

Interfaces with secure firmware
 Prevents the operating system running in normal world
from directly accessing critical hardware resources
 PSCI, Power State Coordination Interface
 Power management related: turn CPUs on/off, CPU idle state,
platform shutdown/reset
 SCMI, System Control and Management Interface
 Power domain, clocks, sensor, performance
 Secure firmware implementing these interfaces is
 Mandatory to run Linux on ARMv8
 Mandatory to run Linux on some ARMv7 platforms, but not all

© 2024 Realtek Semiconductor Corp. All rights reserved 34

SCMI
 System Control & Management Interface

© 2024 Realtek Semiconductor Corp. All rights reserved 35

TF-A
 Trusted Firmware-A (TF-A) provides a reference implementation of secure world
software for Armv7-A and Armv8-A, including a Secure Monitor executing at
Exception Level 3 (EL3)
 Formerly known as ATF, for ARM Trusted Firmware
 Implements the various standard interfaces that operating systems need from the
secure firmware
 https://www.trustedfirmware.org/projects/tf-a/

© 2024 Realtek Semiconductor Corp. All rights reserved 36

Trusted OS, OP-TEE
 A trusted operating system can run in the secure world, also called Trusted
Execution Environment or TEE
 Hardware partitioning between secure world and normal world
 Some hardware resources only available in the secure world, by the trusted OS
 Allows to run trusted applications/services
 isolated from Linux
 can provide services to Linux applications
 Most common open-source implementation: OP-TEE
 https://www.op-tee.org/

© 2024 Realtek Semiconductor Corp. All rights reserved 37

ARM Boot Summary

© 2024 Realtek Semiconductor Corp. All rights reserved 38

Allwinner ARMv8 cores

© 2024 Realtek Semiconductor Corp. All rights reserved 39

TI AM62x (BeaglePlay): ARMv7 and ARMv8 cores

© 2024 Realtek Semiconductor Corp. All rights reserved 40

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 41

Linux kernel in the system

© 2024 Realtek Semiconductor Corp. All rights reserved 42

Linux kernel main roles
 Manage all the hardware resources: CPU, memory, I/O.
 Provide a set of portable, architecture and hardware independent APIs to
allow user space applications and libraries to use the hardware resources.
 Handle concurrent accesses and usage of hardware resources from different
applications.
 Example: a single network interface is used by multiple user space applications through
various network connections. The kernel is responsible for “multiplexing” the hardware
resource.

© 2024 Realtek Semiconductor Corp. All rights reserved 43

Inside the Linux kernel

© 2024 Realtek Semiconductor Corp. All rights reserved 44

Linux versioning scheme
 Until 2003, there was a new stable release branch of Linux every 2 or 3 years (2.0,
2.2, 2.4). New development branches took 2-3 years to become stable (too slow!).
 Since 2003, there is a new stable release of Linux about every 10 weeks:
 Versions 2.6 (Dec. 2003) to 2.6.39 (May 2011)
 Versions 3.0 (Jul. 2011) to 3.19 (Feb. 2015)
 Versions 4.0 (Apr. 2015) to 4.20 (Dec. 2018)
 Versions 5.0 (Mar. 2019) to 5.19 (July 2022)
 Version 6.0 was released in Oct. 2022.
 Features are added to the kernel in a progressive way. Since 2003, kernel
developers have managed to do so without having to introduce a massively
incompatible development branch.
 For each release, there are bugfix and security updates: 6.0.1, 6.0.2, etc.

© 2024 Realtek Semiconductor Corp. All rights reserved 45

Linux version history

© 2024 Realtek Semiconductor Corp. All rights reserved 46

 Linus Torvalds Quotes

I decided to just bite the bullet, and call the next version 3.0. It will get released close enough
to the 20-year mark, which is excuse enough for me, although honestly, the real reason is
just that I can no longer comfortably count as high as 40.

The strongest argument for some people advocating 4.0 seems to have been a wish to see
4.1.15 - because "that was the version of Linux skynet used for the T-800 terminator"

The numbering change is not indicative of anything special. If you want to have an official
reason, it's that I ran out of fingers and toes to count on, so 4.21 became 5.0

So, as is hopefully clear to everybody, the major version number change is more about me
running out of fingers and toes than it is about any big fundamental changes.

© 2024 Realtek Semiconductor Corp. All rights reserved 47

Linux 4.1.15-1.1381_SKYN12nnmp(slawes.sky.net)

© 2024 Realtek Semiconductor Corp. All rights reserved 48

Location of official kernel sources
 The mainline versions of the Linux kernel, as released by Torvalds
 These versions follow the development model of the kernel
 They may not contain the latest developments from a specific area yet
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 The stable versions of the Linux kernel, as maintained by a maintainers group
 These versions do not bring new features compared to Linus’ tree
 Only bug fixes and security fixes are pulled there
 Each version is stabilized during the development period of the next mainline kernel
 Certain versions can be maintained for much longer, 2+ years
 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

© 2024 Realtek Semiconductor Corp. All rights reserved 49

Getting Linux source
 The kernel sources are available from https://www.kernel.org
 Fetch the entire kernel sources and history
 git clone git: //git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 Create a branch that starts at a specific stable version
 git checkout -b <name-of-branch> v5.18
 Web interface available at
 http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/

© 2024 Realtek Semiconductor Corp. All rights reserved 50

Linux kernel size and structure
 Linux v5.18 sources: close to 80k files, 35M lines, 1.3GiB
 But a compressed Linux kernel just sizes a few megabytes.
 So, why are these sources so big?
Because they include numerous device drivers, network protocols, architectures,
filesystems... The core is pretty small!
 As of kernel version v5.18 (in percentage of total number of lines):

© 2024 Realtek Semiconductor Corp. All rights reserved 51

Linux git trees

© 2024 Realtek Semiconductor Corp. All rights reserved 52

Linux development model
 Using merge and bug fixing windows

© 2024 Realtek Semiconductor Corp. All rights reserved 53

Need for long term support (1)
 Issue: bug and security fixes only released for most recent kernel versions.
 Solution: the last release of each year is made an LTS (Long Term Support) release,
and is supposed to be supported (and receive bug and security fixes) for at least 2
years.

 Example at Google: starting from Android O (2017), all new Android devices have
to run such an LTS kernel.

© 2024 Realtek Semiconductor Corp. All rights reserved 54

Need for long term support (2)
 You could also get long term support from a commercial embedded Linux
provider.
 Wind River Linux can be supported for up to 15 years.
 Ubuntu Core can be supported for up to 10 years.
 ”If you are not using a supported distribution kernel, or a stable / longterm kernel,
you have an insecure kernel” - Greg KH, 2019
Some vulnerabilities are fixed in stable without ever getting a CVE.
 The Civil Infrastructure Platform project is an industry / Linux Foundation effort to
support much longer (at least 10 years) selected LTS versions.

© 2024 Realtek Semiconductor Corp. All rights reserved 55

Programming language (1)
 Implemented in C like all UNIX systems
 A little Assembly is used too:
 CPU and machine initialization, exceptions
 Critical library routines.
 All the code compiled with gcc
 Many gcc specific extensions used in the kernel code, any ANSI C compiler will not compile
the kernel
 See https://gcc.gnu.org/onlinedocs/gcc-10.2.0/gcc/C-Extensions.html

© 2024 Realtek Semiconductor Corp. All rights reserved 56

Programming language (2)
 A subset of the supported architectures can be built with the LLVM C
compiler (Clang) too: https://clangbuiltlinux.github.io/
 Rust support is currently being introduced:
drivers/net/phy/ax88796b_rust.rs is a first driver written in Rust.
 No C++ used, see http://vger.kernel.org/lkml/#s15-3

© 2024 Realtek Semiconductor Corp. All rights reserved 57

No C library
 The kernel has to be standalone and can’t use user space code.
 Architectural reason: user space is implemented on top of kernel services, not the
opposite.
 Technical reason: the kernel is on its own during the boot up phase, before it has
accessed a root filesystem.
 Hence, kernel code has to supply its own library implementations (string utilities,
cryptography, uncompression...)
 So, you can’t use standard C library functions in kernel code. (printf(), memset(),
malloc(),...).
 Fortunately, the kernel provides similar C functions for your convenience, like
printk(), memset(), kmalloc(), ...

© 2024 Realtek Semiconductor Corp. All rights reserved 58

Portability
 The Linux kernel code is designed to be portable
 All code outside arch/ should be portable
 To this aim, the kernel provides macros and functions to abstract the architecture
specific details
 Endianness
 cpu_to_be32()
 cpu_to_le32()
 be32_to_cpu()
 le32_to_cpu()
 I/O memory access
 Memory barriers to provide ordering guarantees if needed
 DMA API to flush and invalidate caches if needed

© 2024 Realtek Semiconductor Corp. All rights reserved 59

Linux kernel to user API/ABI stability
 Linux kernel to userspace API is stable
 Source code for userspace applications will not have to
be updated when compiling for a more recent kernel
 System calls, /proc and /sys content cannot be
removed or changed. Only new entries can be added.
 Linux kernel to userspace ABI is stable
 Binaries are portable and can be executed on a more
recent kernel
 The way memory is accessed, the size of the variables
in memory, how structures are organized, the calling
convention, etc, are all stable over time.

© 2024 Realtek Semiconductor Corp. All rights reserved 60

Linux internal API/ABI instability
 Linux internal API is not stable
 The source code of a driver is not portable across
versions
 In-tree drivers are updated by the developer
proposing the API change: works great for
mainline code
 An out-of-tree driver compiled for a given
version may no longer compile or work on a
more recent one
 See process/stable-api-nonsense in kernel
sources for reasons why.
 Linux internal ABI is not stable
 A binary module compiled for a given kernel
version cannot be used with another version
 The module loading utilities will perform this
check prior to the insertion

© 2024 Realtek Semiconductor Corp. All rights reserved 61

Kernel memory constraints
 No memory protection
 The kernel doesn’t try to recover from attempts to access illegal memory
locations. It just dumps oops messages on the system console.
 Fixed size stack (8 or 4 KB). Unlike in user space, no mechanism was implemented
to make it grow.
 Swapping is not implemented for kernel memory either.
(except tmpfs which lives completely in the page cache and on swap)

© 2024 Realtek Semiconductor Corp. All rights reserved 62

Linux kernel licensing constraints
 The Linux kernel is licensed under the GNU General Public License version 2
 This license gives you the right to use, study, modify and share the software freely
 However, when the software is redistributed, either modified or unmodified, the
GPL requires that you redistribute the software under the same license, with the
source code
 If modifications are made to the Linux kernel (for example to adapt it to your hardware), it
is a derivative work of the kernel, and therefore must be released under GPLv2
 The GPL license has been successfully enforced in courts:
https://en.wikipedia.org/wiki/Gpl-violations.org#Notable_victories
 However, you’re only required to do so
 At the time the device starts to be distributed
 To your customers, not to the entire world

© 2024 Realtek Semiconductor Corp. All rights reserved 63

Proprietary code and the kernel
 It is illegal to distribute a binary kernel that includes statically compiled
proprietary drivers
 The kernel modules are a gray area: are they derived works of the kernel or not?
 The general opinion of the kernel community is that proprietary modules are bad:
process/kernel-driver-statement
 From a legal point of view, each driver is probably a different case:
 Are they derived works of the kernel?
 Are they designed to be used with another operating system?

© 2024 Realtek Semiconductor Corp. All rights reserved 64

Abusing the kernel licensing constraints
 There are some examples of  The current trend is to hide the
proprietary drivers logic in the firmware or in
 Nvidia uses a wrapper between their userspace. The GPL kernel driver is
drivers and the kernel almost empty and either:
 They claim the drivers could be used  Blindly writes an incoming flow of
with a different OS with another bytes in the hardware
wrapper  Exposes a huge MMIO region to
userspace through mmap

© 2024 Realtek Semiconductor Corp. All rights reserved 65

Kernel configuration
 Specify the architecture for the kernel to
build
 Set it to the name of a directory under arch/:
export ARCH=arm
 By default, the kernel build system assumes that
the kernel is configured and built for the host
architecture
 make menuconfig
 Kernel config options depends:
 On the target architecture and on your hardware
(for device drivers, etc.)
 On the capabilities you would like to give to your
kernel (network capabilities, filesystems, real-time,
etc.). Such generic options are available in all
architectures.

© 2024 Realtek Semiconductor Corp. All rights reserved 66

Kernel configuration options

© 2024 Realtek Semiconductor Corp. All rights reserved 67

Kernel compilation
 Specify cross compiler:
 export PATH=$PATH:[path/to/your/cross/compiler/bin]
 export CROSS_COMPILE=arm-linux-

 make
 Only works from the top kernel source directory
 Run several jobs in parallel. Our advice: ncpus * 2 to fully
load the CPU and I/Os at all times
Example: make -j 8
 To recompile faster (7x according to some
benchmarks), use the ccache compiler cache
 export CROSS_COMPILE="ccache arm-linux-"

© 2024 Realtek Semiconductor Corp. All rights reserved 68

Kernel compilation results
 arch/<arch>/boot/Image, uncompressed kernel image that can be booted
 arch/<arch>/boot/*Image*, compressed kernel images that can also be booted
 bzImage for x86, zImage for ARM, Image.gz for RISC-V, etc
 arch/<arch>/boot/dts/*.dtb, compiled Device Tree Blobs
 All kernel modules, spread over the kernel source tree, as .ko (Kernel Object) files.
 vmlinux, a raw uncompressed kernel image in the ELF format, useful for
debugging purposes but generally not used for booting purposes

© 2024 Realtek Semiconductor Corp. All rights reserved 69

Kernel building overview

© 2024 Realtek Semiconductor Corp. All rights reserved 70

Overall booting process

© 2024 Realtek Semiconductor Corp. All rights reserved 71

Kernel command line
 In addition to the compile time configuration, the kernel behavior can be adjusted
with no recompilation using the kernel command line
 The kernel command line is a string that defines various arguments to the kernel
 root= for the root filesystem
 console= for the destination of kernel messages
 Example: console=ttyS1,57600 root=/dev/mtdblock4 rootfstype=squashfs
 Many more exist. The most important ones are documented in admin-guide/kernel-
parameters in kernel documentation.

© 2024 Realtek Semiconductor Corp. All rights reserved 72

Passing the kernel command line
 U-Boot carries the Linux kernel command line string in its bootargs environment
variable
 Right before starting the kernel, it will store the contents of bootargs in the
chosen section of the Device Tree
 The kernel will behave differently depending on its configuration:
 If CONFIG_CMDLINE_FROM_BOOTLOADER is set:
The kernel will use only the string from the bootloader
 If CONFIG_CMDLINE_FORCE is set:
The kernel will only use the string received at configuration time in CONFIG_CMDLINE
 If CONFIG_CMDLINE_EXTEND is set:
The kernel will concatenate both strings

© 2024 Realtek Semiconductor Corp. All rights reserved 73

Using kernel modules
 Modules make it easy to develop drivers without rebooting: load, test, unload,
rebuild, load...
 Useful to keep the kernel image size to the minimum (essential in GNU/Linux
distributions for PCs).
 Also useful to reduce boot time: you don’t spend time initializing devices and
kernel features that you only need later.
 Caution: once loaded, have full control and privileges in the system. No particular
protection. That’s why only the root user can load and unload modules.
 To increase security, possibility to allow only signed modules, or to disable
module support entirely.

© 2024 Realtek Semiconductor Corp. All rights reserved 74

Module installation and metadata
 Normally, modules are installed in
/lib/modules/<kernel-version>/
 Compiled kernel modules are stored in .ko
(Kernel Object) files $ ls /lib/modules/5.19.0-46-generic/
build modules.dep
 Metadata files: initrd
kernel
modules.dep.bin
modules.devname
 modules.dep modules.alias modules.order
modules.alias.bin modules.softdep
 modules.alias modules.builtin modules.symbols
modules.builtin.alias.bin modules.symbols.bin
 modules.symbols modules.builtin.bin vdso
modules.builtin.modinfo
 modules.builtin
 Each file has a corresponding .bin version,
which is an optimized version of the
corresponding text file

© 2024 Realtek Semiconductor Corp. All rights reserved 75

Module dependencies: modules.dep
 Some kernel modules can depend on other modules, based on the symbols
(functions and data structures) that they use.
 Example: the ubifs module depends on the ubi and mtd modules.
 mtd and ubi need to be loaded before ubifs
 These dependencies are described both in
/lib/modules/<kernel-version>/modules.dep and in
/lib/modules/<kernel-version>/modules.dep.bin
 Will be used by module loading tools.

© 2024 Realtek Semiconductor Corp. All rights reserved 76

Module alias: modules.alias

© 2024 Realtek Semiconductor Corp. All rights reserved 77

Module utilities: modinfo
 modinfo <module_name>, for modules in /lib/modules
 modinfo /path/to/module.ko

# modinfo usb_storage
filename: /lib/modules/5.18.13-200.fc36.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz
license: GPL
description: USB Mass Storage driver for Linux
author: Matthew Dharm <[email protected]>
alias: usb:v*p*d*dc*dsc*dp*ic08isc06ip50in*
alias: usb:v*p*d*dc*dsc*dp*ic08isc05ip50in*
alias: usb:v*p*d*dc*dsc*dp*ic08isc04ip50in*
[...]
intree: Y
name: usb_storage
[...]
parm: option_zero_cd:ZeroCD mode (1=Force Modem (default), 2=Allow CD-Rom (uint)
parm: swi_tru_install:TRU-Install mode (1=Full Logic (def), 2=Force CD-Rom, 3=Force Modem) (uint)
parm: delay_use:seconds to delay before using a new device (uint)
parm: quirks:supplemental list of device IDs and their quirks (string)

© 2024 Realtek Semiconductor Corp. All rights reserved 78

Module utilities: lsmod
 Lists currently loaded kernel modules
$ lsmod
 Includes Module Size Used by
btrfs 1536000 0
 The reference count: incremented when the blake2b_generic 20480 0
module is used by another module or by a xor 24576 1 btrfs
user-space process, prevents from unloading zstd_compress 225280 1 btrfs
raid6_pq 122880 1 btrfs
modules that are in-use ufs 106496 0
 Dependant modules: modules that depend on qnx4 16384 0
hfsplus 118784 0
us hfs 65536 0
minix 49152 0
 Information retrieved through ntfs 122880 0
/proc/modules msdos
jfs
20480
233472
0
0
xfs 1753088 0
libcrc32c 16384 2 btrfs,xfs

© 2024 Realtek Semiconductor Corp. All rights reserved 79

Module utilities: insmod and rmmod
 Basic tools to:
 load a module: insmod
 unload a module: rmmod
 Basic because:
 Need a full path to the module .ko file
 Do not handle module dependencies

# insmod /lib/modules/`uname -r`/kernel/fs/fuse/cuse.ko.xz

# rmmod cuse

© 2024 Realtek Semiconductor Corp. All rights reserved 80

Module utilities: modprobe
 modprobe is the more advanced tool for loading/unloading modules
 Takes just a module name as argument: modprobe <module-name>
 Takes care of dependencies automatically, using the modules.dep file
 Supports removing modules using modprobe -r, including its no longer used
dependencies

# modinfo fat_test | grep depends

depends: kunit,fat
# lsmod | grep -E "^(kunit|fat|fat_test)"
fat 86016 1 vfat
# modprobe fat_test
# lsmod | grep -E "^(kunit|fat|fat_test)"
fat_test 24576 0
kunit 36864 1 fat_test
fat 86016 2 fat_test,vfat
# sudo modprobe -r fat_test
# lsmod | grep -E "^(kunit|fat|fat_test)"
fat 86016 1 vfat

© 2024 Realtek Semiconductor Corp. All rights reserved 81

Passing parameters to modules
 Find available parameters:
 modinfo usb-storage
 Through insmod:
 sudo insmod ./usb-storage.ko delay_use=0
 Through modprobe:
 Set parameters in /etc/modprobe.conf or in any file in /etc/modprobe.d/: options usb-
storage delay_use=0
 Through the kernel command line, when the driver is built statically into the
kernel:
 usb-storage.delay_use=0

© 2024 Realtek Semiconductor Corp. All rights reserved 82

Modules in sysfs
 All modules are visible in sysfs, under /sys/module/<name>
 Lots of information available about each module
 For example, the /sys/module/<name>/parameters directory contains one file per
module parameter
 Can read the current value of module parameters
 Some of them can even be changed at runtime (determined by the module code)
 Example:
echo 0 > /sys/module/usb_storage/parameters/delay_use

© 2024 Realtek Semiconductor Corp. All rights reserved 83

Hello module
// SPDX-License-Identifier: GPL-2.0
/* hello.c */
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

static int __init hello_init(void)

{
pr_info(”Hello World!\n");
return 0;
}

static void __exit hello_exit(void)

{
pr_info(”Goodbye World!\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION(”Hello module");
MODULE_AUTHOR(”[email protected]");

© 2024 Realtek Semiconductor Corp. All rights reserved 84

Hello module
 Code marked as __init:
 Removed after initialization (static kernel or module.)
 See how init memory is reclaimed when the kernel finishes booting:

[ 2.710026] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.

[ 2.730844] devtmpfs: mounted
[ 2.743340] Freeing unused kernel memory: 1240K

 Code marked as __exit:

 Discarded when module compiled statically into the kernel, or when module unloading
support is not enabled.

© 2024 Realtek Semiconductor Corp. All rights reserved 85

Symbols exported to modules (1)
 From a kernel module, only a limited number of kernel functions can be called
 Functions and variables have to be explicitly exported by the kernel to be visible
to a kernel module
 Two macros are used in the kernel to export functions and variables:
 EXPORT_SYMBOL(symbolname), which exports a function or variable to all modules
 EXPORT_SYMBOL_GPL(symbolname), which exports a function or variable only to GPL
modules

© 2024 Realtek Semiconductor Corp. All rights reserved 86

Symbols exported to modules (2)

© 2024 Realtek Semiconductor Corp. All rights reserved 87

Compiling a module
 Two solutions
 Out of tree , When the code is outside of the kernel source tree, in a different directory
 Not integrated into the kernel configuration/compilation process
 Needs to be built separately
 The driver cannot be built statically, only as a module
 Inside the kernel tree
 Well integrated into the kernel configuration/compilation process
 The driver can be built statically or as a module

© 2024 Realtek Semiconductor Corp. All rights reserved 88

Compiling an out-of-tree Module

obj-m := hello.o

PWD := $(shell pwd)

KDIR := /lib/modules/$(shell uname -r)/build

all:
${MAKE} -C ${KDIR} M=${PWD} modules

install:
${MAKE} -C ${KDIR} M=${PWD} modules_install

clean:
${MAKE} -C ${KDIR} M=${PWD} clean

© 2024 Realtek Semiconductor Corp. All rights reserved 89

New module in Kernel Sources (1)
 To add a new module to the kernel sources
 Add your new source file to the appropriate source directory. Example:
sample/hello.c
 Describe the configuration interface for your new driver by adding the following lines to
the Kconfig file in this directory:
config SAMPLE_HELLO
bool "Sample hello world"
help
Say hello to the world.

 Add a line in the Makefile file based on the Kconfig setting:

obj-$(CONFIG_SAMPLE_HELLO) += hello.o

© 2024 Realtek Semiconductor Corp. All rights reserved 90

New Module in Kernel Sources (2)

© 2024 Realtek Semiconductor Corp. All rights reserved 91

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 92

Describing hardware device
 Discoverable hardware
 Some busses have built-in hardware discoverability mechanisms
 Most common busses: USB and PCI
 Non-discoverable hardware
 Many embedded architectures have a lot of non-discoverable hardware (serial, Ethernet,
I2C, Nand flash, USB controllers...)
 This hardware needs to be described and passed to the Linux kernel
 The bootloader/firmware is expected to provide this description when starting the kernel:
 Directly in the OS/bootloader code
 Using ACPI tables: On x86 systems, but also on a subset of ARM64 platforms
 Using a Device Tree: On most embedded devices, originates from OpenFirmware

© 2024 Realtek Semiconductor Corp. All rights reserved 93

Device Tree: from source to blob
 A tree data structure describing the
hardware is written by a developer in a
Device Tree Source file, .dts
 Processed by the Device Tree Compiler, dtc
 Produces a more efficient representation:
Device Tree Blob, .dtb
 Additional C preprocessor pass

© 2024 Realtek Semiconductor Corp. All rights reserved 94

Where are Device Tree Sources located?
 Even though they are OS-agnostic, no central and OS-neutral place to host
Device Tree sources and share them between projects
 In practice, the Linux kernel sources can be considered as the canonical location
for Device Tree Source files
 arch/<ARCH>/boot/dts/<vendor>/
 arch/arm/boot/dts (on ARM 32 architecture before Linux 6.5)
 ≈ 4500 Device Tree Source files (.dts and .dtsi) in Linux as of 6.0.
 Duplicated/synced in various projects
 U-Boot, Barebox, TF-A

© 2024 Realtek Semiconductor Corp. All rights reserved 95

Device Tree base syntax
 Tree of nodes
 Nodes with properties
 Node ≈ a device or IP block
 Properties ≈ device characteristics
 Notion of cells in property values
 Notion of phandle to point to other nodes
 dtc only does syntax checking, no semantic
validation

© 2024 Realtek Semiconductor Corp. All rights reserved 96

DT overall structure: simplified example

© 2024 Realtek Semiconductor Corp. All rights reserved 97

DT overall structure: simplified example

© 2024 Realtek Semiconductor Corp. All rights reserved 98

DT overall structure: simplified example

© 2024 Realtek Semiconductor Corp. All rights reserved 99

DT overall structure: simplified example

© 2024 Realtek Semiconductor Corp. All rights reserved 100

DT overall structure: simplified example

© 2024 Realtek Semiconductor Corp. All rights reserved 101

Device Tree inheritance
 Device Tree files are not monolithic, they can be split in several files, including
each other.
 .dtsi files are included files, while .dts files are final Device Trees
 Only .dts files are accepted as input to dtc
 Typically, .dtsi will contain
 definitions of SoC-level information
 definitions common to several boards
 The .dts file contains the board-level information
 The inclusion works by overlaying the tree of the including file over the tree of
the included file, according to the order of the #include directives.
 Allows an including file to override values specified by an included file.
 Uses the C pre-processor #include directive

© 2024 Realtek Semiconductor Corp. All rights reserved 102

Device Tree inheritance example

© 2024 Realtek Semiconductor Corp. All rights reserved 103

DT inheritance in Bone Black support

© 2024 Realtek Semiconductor Corp. All rights reserved 104

Device Tree design principles
 Describe hardware (how the hardware is), not configuration (how I choose to use
the hardware)
 OS-agnostic
 For a given piece of HW, Device Tree should be the same for U-Boot, FreeBSD or Linux
 There should be no need to change the Device Tree when updating the OS
 Describe integration of hardware components, not the internals of hardware
components
 The details of how a specific device/IP block is working is handled by code in device drivers
 The Device Tree describes how the device/IP block is connected/integrated with the rest of
the system: IRQ lines, DMA channels, clocks, reset lines, etc.
 Like all beautiful design principles, these principles are sometimes violated.

© 2024 Realtek Semiconductor Corp. All rights reserved 105

The properties
 Device tree properties can be:
 Fully generic
 Their meaning is usually described in one place: the core DT schema available at
https://github.com/devicetree-org/dt-schema
 compatible, reg, #address-cells, etc
 Subsystem specific and cover generic consumer bindings
 Their meaning is either described in the dt-schema GitHub repository or under
Documentation/device-tree/bindings
 clocks, interrupts, regulators, etc
 Subsystem specific
 spi-cpha, i2c-scl-internal-delay-ns, nand-ecc-engine, mac-address, etc
 Vendor/device specific
 To describe uncommon or very specific properties
 Always described in the device’s binding file and prefixed with <vendor>,
 ti,hwmods, xlnx,num-channels, nxp,tx-output-mode, etc

© 2024 Realtek Semiconductor Corp. All rights reserved 106

The compatible property
 Is a list of strings
 Describes the specific binding to which the node complies.
 It uniquely identifies the programming model of the device.
 Practically speaking, it is used by the operating system to find the appropriate
driver for this device.
 When describing real hardware, the typical form is vendor,model
 Examples:
 compatible = "arm,armv7-timer";
 compatible = "st,stm32mp1-dwmac", "snps,dwmac-4.20a";
 compatible = "gpio-keys";
 Special value: simple-bus → bus where all sub-nodes are memory-mapped
devices

© 2024 Realtek Semiconductor Corp. All rights reserved 107

compatible property and Linux kernel drivers
 Linux identifies as platform devices:
 Top-level DT nodes with a compatible string
 Sub-nodes of simple-bus
 Sub-nodes of I2C controllers → I2C devices
 Sub-nodes of SPI controllers → SPI devices
 Each Linux driver has a table of compatible
strings it supports
 struct of_device_id[]
 When a DT node compatible string matches a
given driver, the device is bound to that
driver.

© 2024 Realtek Semiconductor Corp. All rights reserved 108

Matching with drivers in Linux: platform driver

© 2024 Realtek Semiconductor Corp. All rights reserved 109

Matching with drivers in Linux: I2C driver

© 2024 Realtek Semiconductor Corp. All rights reserved 110

reg property
 Memory-mapped devices: base physical address and size of the memory-
mapped registers. Can have several entries for multiple register areas.

 I2C devices: address of the device on the I2C bus.

© 2024 Realtek Semiconductor Corp. All rights reserved 111

reg property
 SPI devices: chip select number

 The unit address must be the address of the first reg entry.

© 2024 Realtek Semiconductor Corp. All rights reserved 112

cells property (1)
 Property numbers shall fit into 32-bit containers called cells
 Example with a reg property using 2 entries of 2 cells:

reg = <0x50027000 0x4>, <0x500273f0 0x10>;

 The OS cannot make the difference with:

reg = <0x50027000>, <0x4>, <0x500273f0>, <0x10>;

reg = <0x50027000 0x4 0x500273f0>, <0x10>;
reg = <0x50027000>, <0x4 0x500273f0 0x10>;
reg = <0x50027000 0x4 0x500273f0 0x10>;

© 2024 Realtek Semiconductor Corp. All rights reserved 113

cells property (2)
 Need for other properties to declare the right formatting:
 #address-cells: Indicates the number of cells used to carry the address
 #size-cells: Indicates the number of cells used to carry the size
 The parent-node declares the children reg property formatting
 Platform devices need memory ranges
 SPI devices need chip-selects

module@a0000 { spi@300000 {
#address-cells = <1>; #address-cells = <1>;
#size-cells = <1>; #size-cells = <0>;

serial@1000 { flash@1 {
reg = <0x1000 0x10>, <0x2000 0x10>; reg = <1>;
}; };
}; };

© 2024 Realtek Semiconductor Corp. All rights reserved 114

status property
 The status property indicates if the device is really in use or not
 In .dtsi files describing SoCs: all devices that interface to the outside world have
status = "disabled";
 Enabled on a per-device basis in the board .dts

© 2024 Realtek Semiconductor Corp. All rights reserved 115

Resources: interrupts, clocks, DMA, reset lines, ...
 Common pattern for resources shared by
multiple hardware blocks
 Interrupt lines
 Clock controllers
 DMA controllers
 Reset controllers
 …
 A Device Tree node describing the
controller as a device
 References from other nodes that use
resources provided by this controller

© 2024 Realtek Semiconductor Corp. All rights reserved 116

Typical software stack for hardware access
 From the bottom to the top:
 A bus controller driver in the kernel drives an
I2C, SPI, USB, PCI controller
 A bus subsystem provides an API for drivers to
access a particular type of bus: I2C, SPI, PCI,
USB, etc.
 A device driver in the kernel drives a particular
device connected to a given bus
 A driver subsystem exposes features of certain
class of devices, through a standard
kernel/user-space interface
 An application can access the device through
this standard kernel/user-space interface either
directly or through a library.

© 2024 Realtek Semiconductor Corp. All rights reserved 117

Stack illustrated with a GPIO expander

© 2024 Realtek Semiconductor Corp. All rights reserved 118

Kernel and Device Drivers
 In Linux, a driver is always interfacing with:
 a framework that allows the driver to expose the
hardware features in a generic way.
 a bus infrastructure, part of the device model, to
detect/communicate with the hardware.

© 2024 Realtek Semiconductor Corp. All rights reserved 119

Device Model data structures
 The device model is organized around three main data structures:
 The struct bus_type structure, which represents one type of bus (USB, PCI, I2C, etc.)
 The struct device_driver structure, which represents one driver capable of handling certain
devices on a certain bus.
 The struct device structure, which represents one device connected to a bus
 The kernel uses inheritance to create more specialized versions of struct
device_driver and struct device for each bus subsystem.

© 2024 Realtek Semiconductor Corp. All rights reserved 120

Bus Drivers
 The first component of the device model is the bus driver
 One bus driver for each type of bus: USB, PCI, SPI, MMC, I2C, etc.
 It is responsible for
 Registering the bus type (struct bus_type)
 Allowing the registration of adapter drivers (USB controllers, I2C adapters, etc.), able to
detect the connected devices (if possible), and providing a communication mechanism with
the devices
 Allowing the registration of device drivers (USB devices, I2C devices, PCI devices, etc.),
managing the devices
 Matching the device drivers against the devices detected by the adapter drivers.
 Provides an API to implement both adapter drivers and device drivers
 Defining driver and device specific structures.
e.g. struct usb_driver and struct usb_interface

© 2024 Realtek Semiconductor Corp. All rights reserved 121

sysfs
 The bus, device, drivers, etc. structures are internal to the kernel
 The sysfs virtual filesystem offers a mechanism to export such information to user
space
 Used for example by udev to provide automatic module loading, firmware
loading, mounting of external media, etc.
 sysfs is usually mounted in /sys
 /sys/bus/ contains the list of buses
 /sys/devices/ contains the list of devices
 /sys/class enumerates devices by the framework they are registered to (net, input, block...),
whatever bus they are connected to

© 2024 Realtek Semiconductor Corp. All rights reserved 122

Example of the USB bus

Hardware view of the bus Device model view of the bus

© 2024 Realtek Semiconductor Corp. All rights reserved 123

Example of Device Driver
 To illustrate how drivers are implemented to
work with the device model, we will study the
source code of a driver for a USB network card
 It is USB device, so it has to be a USB device driver
 It exposes a network device, so it has to be a
network driver
 Most drivers rely on a bus infrastructure (here, USB)
and register themselves in a framework (here,
network)
 Example: drivers/net/usb/rtl8150.c

© 2024 Realtek Semiconductor Corp. All rights reserved 124

Device Identifiers
 Defines the set of devices that this driver can manage, so that the USB core
knows for which devices this driver should be used
 The MODULE_DEVICE_TABLE() macro allows depmod (run by make
modules_install) to extract the relationship between device identifiers and
drivers, so that drivers can be loaded automatically by udev. See
/lib/modules/$(uname -r)/modules.{alias,usbmap}

© 2024 Realtek Semiconductor Corp. All rights reserved 125

Instantiation of usb_driver
 struct usb_driver is a structure defined by the USB core. Each USB device driver
must instantiate it, and register itself to the USB core using this structure
 This structure inherits from struct device_driver, which is defined by the device
model.

© 2024 Realtek Semiconductor Corp. All rights reserved 126

Driver registration and unregistration
 When the driver is loaded / unloaded, it must register / unregister itself to / from
the USB core
 Done using usb_register() and usb_deregister(), provided by the USB core.

 All this code is actually replaced by a call to the module_usb_driver() macro:

© 2024 Realtek Semiconductor Corp. All rights reserved 127

At Initialization
 The USB adapter driver that corresponds to the USB controller of the system
registers itself to the USB core
 The rtl8150 USB device driver registers itself to the USB core

 The USB core now knows the association between the vendor/product IDs of
rtl8150 and the struct usb_driver structure of this driver

© 2024 Realtek Semiconductor Corp. All rights reserved 128

When a device is detected

© 2024 Realtek Semiconductor Corp. All rights reserved 129

Probe Method
 Invoked for each device bound to a driver
 The probe() method receives as argument a structure describing the device,
usually specialized by the bus infrastructure (struct pci_dev, struct usb_interface,
etc.)
 This function is responsible for
 Initializing the device, mapping I/O memory, registering the interrupt handlers. The bus
infrastructure provides methods to get the addresses, interrupt numbers and other device-
specific information.
 Registering the device to the proper kernel framework, for example the network
infrastructure.

© 2024 Realtek Semiconductor Corp. All rights reserved 130

Example: probe() and disconnect() methods

© 2024 Realtek Semiconductor Corp. All rights reserved 131

Platform devices
 Amongst the non-discoverable devices, a huge family are the devices that are
directly part of a system-on-chip: UART controllers, Ethernet controllers, SPI or
I2C controllers, graphic or audio devices, etc.
 In the Linux kernel, a special bus, called the platform bus has been created to
handle such devices.
 It supports platform drivers that handle platform devices.
 It works like any other bus (USB, PCI), except that devices are enumerated
statically instead of being discovered dynamically.

© 2024 Realtek Semiconductor Corp. All rights reserved 132

Implementation of a Platform Driver
 Define platform_driver structure

 Register its driver to the platform driver infrastructure

or

© 2024 Realtek Semiconductor Corp. All rights reserved 133

Platform device instantiation
 As platform devices cannot be detected dynamically, they are defined statically
 Legacy way: by direct instantiation of struct platform_device structures, as done on a few
old ARM platforms.
 Current way: by parsing an ”external” description, like a device tree on most embedded
platforms today, from which struct platform_device structures are created.

© 2024 Realtek Semiconductor Corp. All rights reserved 134

Hardware resources
 DTS resources
uart3: serial@5000c000 {
compatible = "fsl,imx53-uart", "fsl,imx21-uart";
reg = <0x5000c000 0x4000>;
interrupts = <33>;
clocks = <&clks IMX5_CLK_UART3_IPG_GATE>,
<&clks IMX5_CLK_UART3_PER_GATE>;
clock-names = "ipg", "per";
dmas = <&sdma 42 4 0>, <&sdma 43 4 0>;
dma-names = "rx", "tx";
status = "disabled";
};

 The platform driver has access to the resources

base = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
rxirq = platform_get_irq(pdev, 0);
sport->clk_ipg = devm_clk_get(&pdev->dev, "ipg");
sport->dma_chan_tx = dma_request_chan(dev, "tx");

© 2024 Realtek Semiconductor Corp. All rights reserved 135

Kernel and Device Drivers
 In Linux, a driver is always interfacing with:
 a framework that allows the driver to expose the
hardware features in a generic way.
 a bus infrastructure, part of the device model, to
detect/communicate with the hardware.

© 2024 Realtek Semiconductor Corp. All rights reserved 136

Types of devices
 Under Linux, there are essentially three types of devices:
 Network devices. They are represented as network interfaces, visible in user space using
ip a
 Block devices. They are used to provide user space applications access to raw storage
devices (hard disks, USB keys). They are visible to the applications as device files in /dev.
 Character devices. They are used to provide user space applications access to all other
types of devices (input, sound, graphics, serial, etc.). They are also visible to the
applications as device files in /dev.
 Most devices are character devices.

© 2024 Realtek Semiconductor Corp. All rights reserved 137

Major and minor numbers
 Within the kernel, all block and character devices are identified using a major and
a minor number.
 The major number typically indicates the family of the device.
 The minor number allows drivers to distinguish the various devices they manage.
 Most major and minor numbers are statically allocated, and identical across all
Linux systems.
 They are defined in admin-guide/devices.

© 2024 Realtek Semiconductor Corp. All rights reserved 138

Devices: everything is a file
 A very important UNIX design decision was to represent most system objects as
files
 It allows applications to manipulate all system objects with the normal file API
(open, read, write, close, etc.)
 So, devices had to be represented as files to the applications
 This is done through a special artifact called a device file
 It is a special type of file, that associates a file name visible to user space
applications to the triplet (type, major, minor) that the kernel understands
 All device files are by convention stored in the /dev directory

© 2024 Realtek Semiconductor Corp. All rights reserved 139

Device files examples
 Example of device files in a Linux system
$ ls -l /dev/ttyS0 /dev/tty1 /dev/sda1 /dev/sda2 /dev/zero
brw-rw---- 1 root disk 8, 1 Apr 9 10:54 /dev/sda1
brw-rw---- 1 root disk 8, 2 Apr 9 10:54 /dev/sda2
crw--w---- 1 root tty 4, 1 Apr 9 10:54 /dev/tty1
crw-rw---- 1 root dialout 4, 64 Apr 9 10:54 /dev/ttyS0
crw-rw-rw- 1 root root 1, 5 Apr 9 10:54 /dev/zero

 Example C code that uses the usual file API to write data to a serial port
int fd;
fd = open("/dev/ttyS0", O_RDWR);
write(fd, "Hello", 5);
close(fd);

© 2024 Realtek Semiconductor Corp. All rights reserved 140

Creating device files
 Before Linux 2.6.32, on basic Linux systems, the device files had to be created
manually using the mknod command
 mknod /dev/<device> [c|b] major minor
 Needed root privileges
 Coherency between device files and devices handled by the kernel was left to the system
developer
 The devtmpfs virtual filesystem can be mounted on /dev → the kernel
automatically creates/removes device files
 CONFIG_DEVTMPFS_MOUNT → asks the kernel to mount devtmpfs automatically at boot
time (except when booting on an initramfs).
 devtmpfs can be supplemented by userspace tools like udev or mdev to adjust
permission/ownership, load kernel modules automatically and create symbolic
links to devices.

© 2024 Realtek Semiconductor Corp. All rights reserved 141

A character driver in the kernel
 From the point of view of an application, a character device is essentially a file.
 The driver of a character device must therefore implement operations that let
applications think the device is a file: open, close, read, write, etc.
 In order to achieve this, a character driver must implement the operations
described in the struct file_operations structure and register them.
 The Linux filesystem layer will ensure that the driver’s operations are called when
a user space application makes the corresponding system call.

© 2024 Realtek Semiconductor Corp. All rights reserved 142

From user space to the kernel: character devices

© 2024 Realtek Semiconductor Corp. All rights reserved 143

File operations
 Here are the most important operations for a character driver. All of them are
optional.

#include <linux/fs.h>

struct file_operations {
struct module *owner;
ssize_t (*read) (struct file *, char __user *,
size_t, loff_t *);
ssize_t (*write) (struct file *, const char __user *,
size_t, loff_t *);
long (*unlocked_ioctl) (struct file *, unsigned int,
unsigned long);
int (*mmap) (struct file *, struct vm_area_struct *);
int (*open) (struct inode *, struct file *);
int (*release) (struct inode *, struct file *);
……
};

© 2024 Realtek Semiconductor Corp. All rights reserved 144

open() and release()
 int foo_open(struct inode *i, struct file *f)
 Called when user space opens the device file.
 Only implement this function when you do something special with the device at open()
time.
 int foo_release(struct inode *i, struct file *f)
 Called when user space closes the device file.
 Only implement this function when you do something special with the device at close()
time.

© 2024 Realtek Semiconductor Corp. All rights reserved 145

read()
 ssize_t foo_read(struct file *f, char __user *buf,
size_t sz, loff_t *off)
 Called when user space uses the read() system call on the device.
 Must read data from the device, write at most sz bytes to the user space buffer buf, and
update the current position in the file off. f is a pointer to the same file structure that was
passed in the open() operation
 Must return the number of bytes read.
0 is usually interpreted by userspace as the end of the file.
 On UNIX, read() operations typically block when there isn’t enough data to read from the
device

© 2024 Realtek Semiconductor Corp. All rights reserved 146

write()
 ssize_t foo_write(struct file *,
const char __user *buf, size_t sz, loff_t *off)
 Called when user space uses the write() system call on the device
 The opposite of read, must read at most sz bytes from buf, write it to the device, update
off and return the number of bytes written.

© 2024 Realtek Semiconductor Corp. All rights reserved 147

Exchanging data with user space 1/3
 Kernel code isn’t allowed to directly access user space memory, using memcpy()
or direct pointer dereferencing
 Doing so does not work on some architectures
 If the address passed by the application was invalid, the application would segfault.
 Never trust user space. A malicious application could pass a kernel address which you
could overwrite with device data (read case), or which you could dump to the device (write
case).
 To keep the kernel code portable, secure, and have proper error handling, your
driver must use special kernel functions to exchange data with user space.

© 2024 Realtek Semiconductor Corp. All rights reserved 148

Exchanging data with user space 2/3
 A single value
 get_user(v, p);
 The kernel variable v gets the value pointed by the user space pointer p
 put_user(v, p);
 The value pointed by the user space pointer p is set to the contents of the kernel variable v.
 A buffer
 unsigned long copy_to_user(void __user *to,
const void *from, unsigned long n);
 unsigned long copy_from_user(void *to,
const void __user *from, unsigned long n);
 The return value must be checked. Zero on success, non-zero on failure. If non-
zero, the convention is to return -EFAULT.

© 2024 Realtek Semiconductor Corp. All rights reserved 149

Exchanging data with user space 3/3

© 2024 Realtek Semiconductor Corp. All rights reserved 150

Zero copy access to user memory
 Having to copy data to or from an intermediate kernel buffer can become
expensive when the amount of data to transfer is large (video).
 Zero copy options are possible:
 mmap() system call to allow user space to directly access memory mapped I/O space.
 get_user_pages() and related functions to get a mapping to user pages without having to
copy them.

© 2024 Realtek Semiconductor Corp. All rights reserved 151

unlocked_ioctl()
 long unlocked_ioctl(struct file *f,
unsigned int cmd, unsigned long arg);
 Associated to the ioctl() system call.
 Called unlocked because it didn’t hold the Big Kernel Lock (gone now).
 Allows to extend the driver capabilities beyond the limited read/write API.
 cmd is a number identifying the operation to perform
 arg is the optional argument passed as third argument of the ioctl() system call. Can be an
integer, an address, etc.
 The semantic of cmd and arg is driver-specific.

© 2024 Realtek Semiconductor Corp. All rights reserved 152

ioctl() example: kernel side
static long dbg_iomem_ioctl(struct file *file,
unsigned int cmd, unsigned long arg)
{
switch (cmd) {
case DBG_IOMEM_IOC_IOREAD32:
{
u8 buf[4];

if (copy_from_user(buf, (char *)arg, 4)) {

retval = -EFAULT;
break;
}
[…]

break;
}

[…]
default:
return -EINVAL;
}

return retval;
}

© 2024 Realtek Semiconductor Corp. All rights reserved 153

ioctl() Example: Application Side
int main(int argc, char *argv[])
{
int fd, err;
u8 buf[4];
u32 addr = (u32)strtoul(argv[1], NULL, 0);

fd = open("/dev/dbg_iomem", O_RDWR);
if (fd < 0)
return -1;

buf[0] = (u8)(addr >> 24);

buf[1] = (u8)(addr >> 16);
buf[2] = (u8)(addr >> 8);
buf[3] = (u8)addr;

err = ioctl(fd, DBG_IOMEM_IOC_IOREAD32, buf);

if (err < 0)
return err;

close(fd);
return 0;
}

© 2024 Realtek Semiconductor Corp. All rights reserved 154

Beyond character drivers: kernel frameworks
 Many device drivers are not implemented directly as character drivers
 They are implemented under a framework, specific to a given device type
(framebuffer, V4L, serial, etc.)
 The framework allows to factorize the common parts of drivers for the same type of
devices
 From user space, they are still seen as character devices by the applications
 The framework allows to provide a coherent user space interface (ioctl, etc.) for every type
of device, regardless of the driver

© 2024 Realtek Semiconductor Corp. All rights reserved 155

Example: Some Kernel Frameworks

© 2024 Realtek Semiconductor Corp. All rights reserved 156

Example: MMC framework
 Kernel option CONFIG_MMC
 menuconfig MMC
 tristate "MMC/SD/SDIO card support"
 Implemented in C files in drivers/mmc/core/
 Defines the set of operations a MMC host
driver must implement
 struct mmc_host_ops

© 2024 Realtek Semiconductor Corp. All rights reserved 157

MMC host driver operations
 Here are the operations a MMC host driver can or must implement, and define
them in a struct mmc_host_ops structure
(excerpt from drivers/mmc/host/rtsx_pci_sdmmc.c)

static const struct mmc_host_ops realtek_pci_sdmmc_ops = {

.pre_req = sdmmc_pre_req,
.post_req = sdmmc_post_req,
.request = sdmmc_request,
.set_ios = sdmmc_set_ios,
.get_ro = sdmmc_get_ro,
.get_cd = sdmmc_get_cd,
.start_signal_voltage_switch = sdmmc_switch_voltage,
.execute_tuning = sdmmc_execute_tuning,
.init_sd_express = sdmmc_init_sd_express,
};

© 2024 Realtek Semiconductor Corp. All rights reserved 158

MMC host driver code
 In the probe() function, registration of the MMC host device and operations

static int rtsx_pci_sdmmc_drv_probe(struct platform_device *pdev)

{
struct mmc_host *mmc;
[...]
mmc = mmc_alloc_host(sizeof(*icr), &pdev->dev);
[...]
mmc->ops = &sdmmc_ops;
[...]
mmc_add_host(mmc);
[...]
}

MMC host driver: drivers/mmc/host/rtsx_pci_sdmmc.c

MFD driver: drivers/misc/cardreader/rtsx_pcr.c

© 2024 Realtek Semiconductor Corp. All rights reserved 159

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 160

Simple introduction of KBuild
 A build framework based on GNU make and a standard set of cross platform
tools, designed for Linux kernel.
 include a configuration framework called Kconfig
 Powerful build system
 Highly modular and customizable
 The same code base is used for a different range of computing systems, from
supercomputers to very tiny embedded devices
 Not just Linux kernel who use kbuild/kconfig
 U-boot
 Buildroot
 …
 Refer to:
 Documentation/kbuild/*
 https://www.kernel.org/doc/html/latest/kbuild/index.html

© 2024 Realtek Semiconductor Corp. All rights reserved 161

Overview of Kconfig
 MANY targets for Kconfig

© 2024 Realtek Semiconductor Corp. All rights reserved 162

Kconfig: menu entries
 Most entries define a config option

config MISC_RTSX_PCI
tristate "Realtek PCI-E card reader"
depends on PCI
select MFD_CORE
help
This supports for Realtek PCI-Express card reader including rts5209,
rts5227, rts522A, rts5229, rts5249, rts524A, rts525A, rtl8411, rts5260.
Realtek card readers support access to many types of memory cards,
such as Memory Stick, Memory Stick Pro, Secure Digital and
MultiMediaCard.

© 2024 Realtek Semiconductor Corp. All rights reserved 163

Kconfig: menu attributes
 A menu entry can have a number of attributes. Not all of them are applicable
everywhere
 type definition: "bool"/"tristate"/"string"/"hex"/"int”
 input prompt: "prompt" <prompt> ["if" <expr>]
 default value: "default" <expr> ["if" <expr>]
 dependencies: "depends on" <expr>
 reverse dependencies: "select" <symbol> ["if" <expr>]
 numerical ranges: "range" <symbol> <symbol> ["if" <expr>]
 help text: "help" or "---help---"

bool "Networking support" bool "foo" if BAR

and default y if BAR
bool and
prompt "Networking support" depends on BAR
bool "foo"
default y

© 2024 Realtek Semiconductor Corp. All rights reserved 164

Kconfig: menu structure
menu “I2C support”
 menu/endmenu
config I2C
 menuconfig tristate “I2C support”

if I2C

onfig I2C_CHARDEV
…

endif # I2C

endmenu

menuconfig SPI
bool “SPI support”

if SPI

config SPI_DEBUG
…

endif # SPI

© 2024 Realtek Semiconductor Corp. All rights reserved 165

Kconfig: source
 This reads the specified configuration file

if MMC

source "drivers/mmc/core/Kconfig"

source "drivers/mmc/card/Kconfig"

source "drivers/mmc/host/Kconfig"

endif # MMC

© 2024 Realtek Semiconductor Corp. All rights reserved 166

Overview of Makefiles
 The Makefiles have five parts:

Makefile the top Makefile.

.config the kernel configuration file.
arch/$(ARCH)/Makefile the arch Makefile.
scripts/Makefile.* common rules etc. for all kbuild Makefiles.
kbuild Makefiles there are about 500 of these.

 The top Makefile reads the .config file, which comes from the kernel configuration
process.
 The top Makefile is responsible for building two major products: vmlinux and
modules
 Each subdirectory has a kbuild Makefile which carries out the commands passed
down from above

© 2024 Realtek Semiconductor Corp. All rights reserved 167

Kbuild: goal definitions
 Goal definitions are the main part (heart) of the kbuild Makefile
 obj-y += foo.o
 obj-m += foo.o
 obj-$(CONFIG_FOO) += foo.o
 Descending down in directories
 obj-$(CONFIG_EXT2_FS) += ext2/

© 2024 Realtek Semiconductor Corp. All rights reserved 168

Kbuild: compilation flags (1)
 ccflags-y, asflags-y and ldflags-y
 These three flags apply only to the kbuild makefile in which they are assigned
 Note: Flags with the same behaviour were previously named:
EXTRA_CFLAGS, EXTRA_AFLAGS and EXTRA_LDFLAGS (Deprecated)
ccflags-y := -Os
ccflags-$(CONFIG_ACPI_DEBUG) += -DACPI_DEBUG_OUTPUT

asflags-y := -ansi

ldflags-y += -T $(srctree)/$(src)/decompress_$(arch-y).lds

© 2024 Realtek Semiconductor Corp. All rights reserved 169

Kbuild: compilation flags (2)
 subdir-ccflags-y, subdir-asflags-y
 The two flags listed above are similar to ccflags-y and asflags-y
 The difference is that the subdir- variants have effect for the kbuild file where they are
present and all subdirectories
subdir-ccflags-y := -Werror

 CFLAGS_$@, AFLAGS_$@
 $(CFLAGS_$@) and $(AFLAGS_$@) specifies per-file options

CFLAGS_aha152x.o = -DAHA152X_STAT –DAUTOCONF

AFLAGS_head.o := -DTEXT_OFFSET=$(TEXT_OFFSET)

© 2024 Realtek Semiconductor Corp. All rights reserved 170

Kbuild: compilation flags (3)
 ccflags-remove-y, asflags-remove-y
 These flags are used to remove particular flags for the compiler, assembler invocations.

ccflags-remove-$(CONFIG_MCOUNT) += -pg

© 2024 Realtek Semiconductor Corp. All rights reserved 171

Kbuild: clean infrastructure
 "make clean" deletes most generated files in the obj tree where the kernel is
compiled
 Additional files can be specified in kbuild makefiles by use of $(clean-files) and
$(clean-dirs)
 To exclude certain files from make clean, use the $(no-clean-files) variable

clean-files := crc32table.h
clean-dirs := $(objtree)/debian/

no-clean-files := $(bounds-file) $(offsets-file)

© 2024 Realtek Semiconductor Corp. All rights reserved 172

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 173

Filesystems
 Filesystems are used to organize data in directories and files on storage devices
or on the network. The directories and files are organized as a hierarchy
 In UNIX systems, applications and users see a single global hierarchy of files and
directories, which can be composed of several filesystems.
 Filesystems are mounted in a specific location in this hierarchy of directories
 When a filesystem is mounted in a directory (called mount point), the contents of this
directory reflect the contents of this filesystem.
 When the filesystem is unmounted, the mount point is empty again.
 This allows applications to access files and directories easily, regardless of their
exact storage location

© 2024 Realtek Semiconductor Corp. All rights reserved 174

mount / umount
 mount allows to mount filesystems
 mount -t type device mountpoint
 sudo mount -t vfat /dev/sda1 /mnt/usbkey
 mount with no arguments shows the currently mounted filesystems
 umount allows to unmount filesystems
 This is needed before rebooting, or before unplugging a USB key, because the Linux kernel
caches writes in memory to increase performance. umount makes sure that these writes
are committed to the storage.

© 2024 Realtek Semiconductor Corp. All rights reserved 175

Root filesystem
 A particular filesystem is mounted at the root of the hierarchy, identified by /
 This filesystem is called the root filesystem
 As the root filesystem is the first mounted filesystem, it cannot be mounted with
the normal mount command
 It is mounted directly by the kernel, according to the root= kernel option
 When no root filesystem is available, the kernel panics:

Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown block(0,0)

© 2024 Realtek Semiconductor Corp. All rights reserved 176

Location of the root filesystem
 It can be mounted from different locations
 From the partition of a hard disk
 From the partition of a USB key
 From the partition of an SD card
 From the partition of a flash chip or similar type of storage device
 From the network, using the NFS protocol
 From memory, using a pre-loaded filesystem (by the bootloader)
 etc.

© 2024 Realtek Semiconductor Corp. All rights reserved 177

Root filesystem organization
 The organization of a Linux root filesystem in terms of directories is well-defined
by the Filesystem Hierarchy Standard
 https://refspecs.linuxfoundation.org/fhs.shtml
 Most Linux systems conform to this specification
 Applications expect this organization
 It makes it easier for developers and users as the filesystem organization is similar in all
systems

© 2024 Realtek Semiconductor Corp. All rights reserved 178

Important directories (1)
/bin Basic programs

/boot Kernel images, configurations and initramfs (only when the kernel is
loaded from a filesystem, not common on non-x86 architectures)
/dev Device files

/etc System-wide configuration

/home Directory for the users home directories

/lib Basic libraries

/media Mount points for removable media

/mnt Mount point for a temporarily mounted filesystem

/proc Mount point for the proc virtual filesystem

© 2024 Realtek Semiconductor Corp. All rights reserved 179

Important directories (2)
/root Home directory of the root user

/run Run-time variable data (previously /var/run)

/sbin Basic system programs

/sys Mount point of the sysfs virtual filesystem

/tmp Temporary files

/usr /usr/bin Non-basic programs

/usr/lib Non-basic system programsNon-basic libraries

/usr/sbin Non-basic system programs

/var Variable data files, for system services. This includes spool directories and
files, administrative and logging data, and transient and temporary files

© 2024 Realtek Semiconductor Corp. All rights reserved 180

Minimal filesystem
 In order to work, a Linux system needs at least a few applications
 An init application, which is the first user space application started by the kernel after
mounting the root filesystem
 The kernel tries to run the command specified by the init= command line parameter if available.
 Otherwise, it tries to run /sbin/init, /etc/init, /bin/init and /bin/sh.
 If none of this works, the kernel panics and the boot process is stopped.
 The init application is responsible for starting all other user space applications and services, and for
acting as a universal parent for processes whose parent terminate before they do.
 A shell, to implement scripts, automate tasks, and allow a user to interact with the system
 Basic UNIX executables, for use in system scripts or in interactive shells: mv, cp, mkdir, cat,
modprobe, mount, ip, etc.
 These basic components have to be integrated into the root filesystem to make it usable

© 2024 Realtek Semiconductor Corp. All rights reserved 181

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 182

Concept of build system
 Each open-source software project comes with its own set of scripts/files to
control its configuration/compilation: its build system
 Detect if system requirements/dependencies are met
 Compile all source files, to generate applications/libraries, as well as documentation
 Installs build products
 Most common build systems:
 Hand-written Makefiles
 Autotools: autoconf, automake, libtool
https://en.wikipedia.org/wiki/GNU_Autotools
 Cmake
https://cmake.org/
 Meson
https://mesonbuild.com/
 Language specific build systems for Python, Perl, Go, Rust, NodeJS, etc.

© 2024 Realtek Semiconductor Corp. All rights reserved 183

Target and staging spaces
 When manually cross-compiling software, we will distinguish two “copies” of the
root filesystem
 The target root filesystem, which ends up on our embedded hardware, which contains only
what is needed for runtime
 The staging space, which has a similar layout, but contains a lot more files than the target
root filesystem: headers, static libraries, documentation, binaries with debugging symbols.
Contains what’s needed for building code.
 Indeed, we want the root filesystem on the target to be as minimal as possible.

© 2024 Realtek Semiconductor Corp. All rights reserved 184

Cross-compiling with hand-written Makefiles
 There is no general rule, as each project has a different set of Makefiles, that use a
different set of variables
 Though it is common to use make standard variables: CC (C compiler path), CXX
(C++ compiler path), LD (linker path), CFLAGS (C compiler flags), CXXFLAGS (C++
compiler flags), LDFLAGS (linker flags)
 DESTDIR for installation destination, sometimes PREFIX for execution location
 Common sequence
$ make CC=arm-linux-gcc CFLAGS=-I/path/to/headers \
LDFLAGS=-L/path/to/libraries
$ make DESTDIR=/installation/path install

 Need to read the documentation (if any), read the Makefiles, and adapt to their
behavior.

© 2024 Realtek Semiconductor Corp. All rights reserved 185

Autotools
 A family of tools, which associated together form a complete and extensible build
system
 autoconf is used to handle the configuration of the software package
 automake is used to generate the Makefiles needed to build the software package
 libtool is used to handle the generation of shared libraries in a system-independent way
 Most of these tools are old and relatively complicated to use
 But they are used by a large number of software components, even though
Meson is gaining significant traction as a replacement today

© 2024 Realtek Semiconductor Corp. All rights reserved 186

automake / autoconf / autoheader

© 2024 Realtek Semiconductor Corp. All rights reserved 187

autotools usage: four steps
1. Only if needed: generate configure and Makefile.in. Either using autoreconf tool,
or sometimes an autogen.sh script is provided by the package
2. Configuration: ./configure
 ./configure --help is very useful
 --prefix: execution location
 --host: target machine when cross-compiling, if not provided, auto-detected. Also used as
the cross-compiler prefix.
 Often --enable-<foo>, --disable-<foo>, --with-<foo>, --without-<foo> for optional
features.
 CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS and many more variables
3. Build: make
4. Installation: make install
 DESTDIR variable for diverted installation

© 2024 Realtek Semiconductor Corp. All rights reserved 188

Example: can-utils native compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 189

Example: can-utils cross-compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 190

CMake
 More modern build system, started in 1999, maintained by a company called
Kitware
 Used by Qt 6, KDE, and many projects which didn’t like autotools
 Perhaps losing traction these days in favor of Meson
 Needs cmake installed on your machine
 CMakeLists.txt files that describe what the dependencies are and what to build and install
 cmake, a tool that processes CMakeLists.txt to generate either Makefiles (default) or Ninja
files
 Typical sequence, when using the Makefile backend:
1. cmake .
2. make
3. make install

© 2024 Realtek Semiconductor Corp. All rights reserved 191

Example: cJSON native compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 192

Example: cJSON cross-compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 193

CMake toolchain file
 When cross-compiling with CMake, the number of arguments to pass to specify
the paths to all cross-compiler tools, libraries, headers, and flags can become
quite long.
 They can be grouped into a toolchain file, which defines CMake variables
 Can then be used with
cmake -DCMAKE_TOOLCHAIN_FILE=/path/to/toolchain-file.txt
 Such a toolchain file is commonly provided by embedded Linux build systems:
Buildroot, Yocto, etc.

© 2024 Realtek Semiconductor Corp. All rights reserved 194

Meson
 The most modern one, written in Python
 Gaining big traction in lots of major open-source projects
 Processes meson.build + meson_options.txt and generates Ninja files
 Ninja is an alternative to make, with much shorter build times
 Needs meson and ninja installed on your machine
 Ninja, a new build system
 Meson requires an out-of-tree build: the build directory must be distinct from the
source directory
1. mkdir build
2. cd build
3. meson ..
4. ninja
5. ninja install

© 2024 Realtek Semiconductor Corp. All rights reserved 195

Example: ipcalc native compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 196

Meson cross file
 In a similar manner to CMake’s
toolchain file, Meson has a concept of
cross file
 Small text file that contains variable
definitions telling Meson all details
needed for cross-compilation
 Can be created manually, or may be
provided by an embedded Linux build
systems such as Buildroot or Yocto.
 --cross-file option of Meson

© 2024 Realtek Semiconductor Corp. All rights reserved 197

Example: ipcalc cross-compilation

© 2024 Realtek Semiconductor Corp. All rights reserved 198

Distinction between prefix and DESTDIR
 There is often a confusion
between prefix and DESTDIR
 Distinction is very important in
cross-compilation context
 prefix: where the software will be
executed from on the target
 DESTDIR: where the software is
installed by the build system
installation procedure. Allows to
install in a different place than
prefix, when creating a root
filesystem for a different machine.

© 2024 Realtek Semiconductor Corp. All rights reserved 199

Build embedded Linux system
 Three main approaches to build your embedded Linux system:
1. Cross-compile everything manually from source
2. Use an embedded Linux build system that automates the cross-compilation process
3. Use a binary distribution such as Debian, Ubuntu or Fedora

© 2024 Realtek Semiconductor Corp. All rights reserved 200

Approaches pros and cons

© 2024 Realtek Semiconductor Corp. All rights reserved 201

Why using building tool?

© 2024 Realtek Semiconductor Corp. All rights reserved 202

Embedded Linux build system

 Building from source → lot of flexibility

 Cross-compilation → leveraging fast build machines
 Recipes for building components → easy

© 2024 Realtek Semiconductor Corp. All rights reserved 203

Build systems vs. Embedded Linux build systems
 Possible confusion between build system
(Makefiles, autotools, CMake, Meson) and
embedded Linux build systems (Buildroot,
Yocto/OpenEmbedded, OpenWrt, etc.)
 Build systems are used by individual software
components, to control the build process of
each source file into a library, executable,
documentation, etc.
 Embedded Linux build systems are tools that
orchestrate the build of all software
components one after the other. They invoke
the build system of each software component.

© 2024 Realtek Semiconductor Corp. All rights reserved 204

System building tools
 Buildroot
 Builds a root filesystem image, no binary packages. Much simpler to use, understand and
modify.
 OpenWRT
 Originally a fork of Buildroot for wireless routers, now a more generic project
 Yocto/OpenEmbedded
 Builds a complete Linux distribution with binary packages. Powerful, but somewhat
complex, and quite steep learning curve.

© 2024 Realtek Semiconductor Corp. All rights reserved 205

Buildroot
 Allows to build a toolchain, a root filesystem image with many
applications and libraries, a bootloader and a kernel
 Supports building uClibc, glibc and musl toolchains, either built by
Buildroot, or external
 Over 2800 applications or libraries integrated, from basic utilities to
more elaborate software stacks: Wayland, GStreamer, Qt, Gtk,
WebKit, Python, PHP, NodeJS, Go, Rust, etc.
 Good for small to medium size embedded systems, with a fixed set
of features
 No support for generating packages (.deb or .ipk)
 Needs complete rebuild for most configuration changes.
 Active community, releases published every 3 months. One LTS
release made every year (YYYY.02 so far).

© 2024 Realtek Semiconductor Corp. All rights reserved 206

Yocto Project / OpenEmbedded
 OpenEmbedded
 Started in 2003
 Goal is to build custom Linux distributions for embedded
devices
 Back then, no stable releases, limited/no documentation,
difficult to use for products
 Yocto Project
 Started in 2011
 By the Linux Foundation
 Goal is to industrialize OpenEmbedded
 Funds the development of OpenEmbedded, makes
regular stable releases, QA effort, extensive
documentation
 One Long Term Support release every 2 years, supported
for 4 years.

© 2024 Realtek Semiconductor Corp. All rights reserved 207

OpenWrt
 Another Embedded Linux build system
 Derived from Buildroot a very long time ago
 Now completely different, except for the use of Kconfig and make
 Targeted at building firmware for WiFi routers and other networking equipments
 Unlike Buildroot or Yocto that leave a lot of flexibility to the user in defining the
system architecture, OpenWrt makes a lot of set in stone decisions:
 musl is the C library
 an OpenWrt specific init system
 an OpenWrt specific inter-process communication bus
 a Web UI specific to OpenWrt
 The aim of OpenWrt is to build a final product out of the box, with support for
popular networking products and development boards

© 2024 Realtek Semiconductor Corp. All rights reserved 208

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 209

Free software vs. open-source
 Free software: term defined by the Free Software Foundation, grants 4 freedoms
 Freedom to use
 Freedom to study
 Freedom to copy
 Freedom to modify and distribute modified copies
 See https://www.gnu.org/philosophy/free-sw.html
 Open Source: term defined by the Open Source Initiative, with 10 criterias
 See https://www.opensource.org/docs/osd
 Free Software movement insists more on ethics, while Open Source insists more
on the technical advantages
 From a freedom standpoint, they are similar.

© 2024 Realtek Semiconductor Corp. All rights reserved 210

Open source licenses
 All free software/open-source licenses rely on copyright law
 Those licenses fall in two main categories
 The copyleft licenses
 The non-copyleft licenses, also called permissive licenses

© 2024 Realtek Semiconductor Corp. All rights reserved 211

Non-Copyleft VS Copyleft licenses
Non-Copyleft Copyleft
(BSD, MIT, Apache, X11…) (GPL, LGPL, AGPL…)

You can You can

Use Use
Modify Modify
Redistribute Redistribute

You must You must

Provide license text Provide license text
Attribution Attribution
Make source code available

© 2024 Realtek Semiconductor Corp. All rights reserved 212

Most common non-copyleft licenses
 MIT, BSD 2 CLAUSE
 Very simple
 Require to preserve the copyright notice
 BSD 3 CLAUSE
 Adds a non-endorsement clause
 Apache
 More complex
 Includes a patent grant, a mechanism to prevent users of the licensed project from suing
others based on patents related to the project

© 2024 Realtek Semiconductor Corp. All rights reserved 213

GPL: GNU General Public License
 The flagship license of the GNU project
 Used by Linux, BusyBox, U-Boot, Barebox, GRUB, many projects from GNU
 Is a copyleft license
 Requires derivative works to be released under the same license
 Source code must be redistributed, including modifications
 If GPL code is integrated in your code, your code must now be GPL-licensed modifications
 Only applies when redistribution takes place
 Also called strong copyleft license
 Programs linked with a library released under the GPL must also be released under the GPL
 Does not prevent GPL programs and non-GPL programs from co-existing in the same
system or to communicate

© 2024 Realtek Semiconductor Corp. All rights reserved 214

LGPL: GNU Lesser General Public License
 Used by glibc, uClibc, and many libraries
 Derived from the GPL license
 Also a copyleft license
 But a weaker copyleft license
 Programs linked against a library under the LGPL do not need to be released under the
LGPL and can be kept proprietary.
 However, the user must keep the ability to update the library independently from the
program.
 Requires using dynamic linking, or in the case of static linking, to provide the object files to
relink with the library.

© 2024 Realtek Semiconductor Corp. All rights reserved 215

GPL/LGPL: redistribution
 No obligation when the software is not distributed
 You can keep your modifications secret until the product delivery
 It is then authorized to distribute binary versions, if one of the following
conditions is met:
 Convey the binary with a copy of the source on a physical medium
 Convey the binary with a written offer valid for 3 years that indicates how to fetch the
source code
 Convey the binary with the network address of a location where the source code can be
found
 In all cases, the attribution and the license must be preserved

© 2024 Realtek Semiconductor Corp. All rights reserved 216

GPL/LGPL: version 2 vs version 3
 GPLv2/LGPLv2 published in 1991, widely used in the open-source world for major
projects
 GPLv3/LGPLv3 published in 2007, and adopted by some projects
 Anti-Tivoization
 Requirement that the user must be able to run the modified versions on the device
 Need to provide installation instructions

© 2024 Realtek Semiconductor Corp. All rights reserved 217

Licensing: examples

© 2024 Realtek Semiconductor Corp. All rights reserved 218

Respect free software licenses
 Free Software is not public domain software, the distributors have obligations due
to the licenses
 Before using a free software component, make sure the license matches your
project constraints
 Make sure to keep your modifications and adaptations well-separated from the
original version.
 Make sure to keep a complete list of the free software packages you use, and the
version in use
 Buildroot and Yocto Project can generate this list for you!
 Buildroot: make legal-info
 Yocto: see the project documentation
 Conform to the license requirements before shipping the product to the
customers.

© 2024 Realtek Semiconductor Corp. All rights reserved 219

Agenda
 Introduction to Embedded Linux
 Bootloaders and firmware
 Linux kernel introduction
 Linux kernel framework and device driver
 KBuild system
 Linux root filesystem
 System building and integration
 Open source licenses and compliance
 Useful resources

© 2024 Realtek Semiconductor Corp. All rights reserved 220

Bootlin
 https://bootlin.com/

© 2024 Realtek Semiconductor Corp. All rights reserved 221

LWN.net
 https://lwn.net/
 LWN.net is a reader-supported news site dedicated to producing the best
coverage from within the Linux and free software development communities

© 2024 Realtek Semiconductor Corp. All rights reserved 222

Thank You
And may the Source be with you

© 2024 Realtek Semiconductor Corp. All rights reserved

© 2024 Realtek Semiconductor Corp. All rights reserved 223
Homework
 Write a hello world module
 Character device
 You can echo a string to the device node
 You can cat the node and get the string
 You can clean the string buffer using ioctl
 Run the module in QEMU

© 2024 Realtek Semiconductor Corp. All rights reserved 224

Download SDK source
 Wiki: https://wiki.realtek.com/display/SIBMC/Downloading+BMC+SDK+Source
 mkdir bmc_sdk
 cd bmc_sdk
 repo init -u ssh://rssi-git.rtkbf.com:29418/bmc/sdk/manifest -b develop/fpga/rts4930
 repo sync
./
├── openbmc
│ ├── bitbake -> poky/bitbake/
│ ├── build
│ ├── LICENSE -> poky/LICENSE
│ ├── meta -> poky/meta
│ ├── meta-amd
│ ├── …
│ ├── oe-init-build-env -> poky/oe-init-build-env
│ ├── openbmc-env
│ ├── OWNERS
│ ├── poky
│ ├── README.md
│ ├── scripts -> poky/scripts/
│ └── setup
└── source
├── linux
└── uboot

© 2024 Realtek Semiconductor Corp. All rights reserved 225

Build for QEMU
 Build steps:
 cd openbmc
 . setup rts4930-vm
 (build/rts4930-vm) tar xvf /home/share/dl/obmc/rts4930/downloads.tar.gz
 (build/rts4930-vm) tar xvf /home/share/dl/obmc/rts4930/sstate-cache.tar.gz
 (build/rts4930-vm) bitbake -k core-image-minimal
 Image location:
 build/rts4930-vm/ tmp/deploy/images/rts4930-vm/

© 2024 Realtek Semiconductor Corp. All rights reserved 226

runqemu
 runqemu rts4930-vm nographic nonetwork
$ runqemu rts4930-vm nographic nonetwork
runqemu - INFO - Running MACHINE=rts4930-vm bitbake -e ...
runqemu - INFO - Continuing with the following parameters:
KERNEL: [/home/wei_wang/projects/bmc/sdk/rts4930_fpga/openbmc/build/rts4930-vm/tmp/deploy/images/rts4930-vm/Image]
MACHINE: [rts4930-vm]
FSTYPE: [ext4]
ROOTFS: [/home/wei_wang/projects/bmc/sdk/rts4930_fpga/openbmc/build/rts4930-vm/tmp/deploy/images/rts4930-vm/core-
image-minimal-rts4930-vm.rootfs-20240710031615.ext4]
CONFFILE: [/home/wei_wang/projects/bmc/sdk/rts4930_fpga/openbmc/build/rts4930-vm/tmp/deploy/images/rts4930-vm/core-
image-minimal-rts4930-vm.rootfs-20240710031615.qemuboot.conf]
…
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd471]
[ 0.000000] Linux version 6.6.35-yocto-standard (oe-user@oe-host)
…
Poky (Yocto Project Reference Distro) 5.0+snapshot-1d4c3b22f745f2625fcd1bca4177c9504116677a rts4930-vm /dev/ttyAMA0

rts4930-vm login: root

root@rts4930-vm:~#

© 2024 Realtek Semiconductor Corp. All rights reserved 227

Rights to copy
© Copyright 2004-2024, Bootlin
License: Creative Commons Attribution - Share Alike 3.0
https://creativecommons.org/licenses/by- sa/3.0/legalcode
You are free:
 to copy, distribute, display, and perform the work
 to make derivative works
 to make commercial use of the work

Under the following conditions:

 Attribution. You must give the original author credit.
 Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a
license identical to this one.
 For any reuse or distribution, you must make clear to others the license terms of this work.
 Any of these conditions can be waived if you get permission from the copyright holder.

Your fair use and other rights are in no way affected by the above.

Document sources: https://github.com/bootlin/training-materials/

© 2024 Realtek Semiconductor Corp. All rights reserved 229