EE514 Real-Time Computing
Real-Time Linux
Department of Electrical Engineering
The Hong Kong Polytechnic University
EE514 Real-Time Computing
Real Time Linux Operating Systems
Swaminathan Sivasubramanian
Dependable Computing & Networking Laboratory
Iowa State University
[email protected]EE514 Real-Time Computing
Linux as a Real-Time OS
Traditional
RT Systems used custom built
systems which were not extensible I.e.
tough to develop new applications
However, as technology improved, generic
real-time OS became acceptable
In OS suited for extensible development
Linux looks more appealing
EE514 Real-Time Computing
Why Linux?
Linux (and its Real Time versions) are
free!!
Linux (and its Real Time versions) are
Open Source!!
Easy for developing RT applications
EE514 Real-Time Computing
Why not Linux?
Linux
didnt have any corporate support
until now
Linux, is a very good general purpose
operating system, but not so for real-time
OS (Why??)
Because, the design motive of a
conventional OS and RTOS is different
EE514 Real-Time Computing
Linux as your real-time
solution?
Could increase priority for real-time tasks
and assume they get scheduled
Problem Linux optimizes average case
whereas an RTOS should work under worst
case assumptions
EE514 Real-Time Computing
Linux A Simplified View
EE514 Real-Time Computing
Linux conflicts with RT constraints
Coarse grained synchronization long intervals when a
task has exclusive use of data ( as fine grained leads to
lot of overhead reducing the average case performance)
Linux batch all operations for efficient use of H/W
E.g freeing a list of pages when memory is full
reducing the worst case performance
Linux doesn't preempt low-priority task during system call
Linux will make high priority tasks wait for low priority to
release resources
EE514 Real-Time Computing
Real Time Linux approaches
Modify
the current Linux kernel to
guarantee RT constraints
Used by KURT
Make
the standard Linux kernel run as a
task of the real-time kernel
Used by RT-Linux, RTAI
EE514 Real-Time Computing
Modifying Linux kernel
Advantages
Most problems, such as interrupt handling,
already solved
Less initial labor
Disadvantages
No guaranteed performance
RT tasks dont always have precedence over
non-RT tasks.
EE514 Real-Time Computing
10
Running Linux as a process of a second
RT kernel
Advantages
Can make hard real time guarantees
Easy to implement a new scheduler
Disadvantages
Initial port difficult, must know a tremendous
amount about underlying hardware
Running a small real-time executive is not a
substitute for a full fledged RTOS
EE514 Real-Time Computing
11
RTAI Overview
Based
on Real-Time Hardware Abstraction
Layer (HAL) (also used in Windows NT)
HAL exports Linux data and functions
related to hardware
HAL defines an interface between RTAI
and Linux
The Windows NT hardware abstraction layer (HAL) refers to a layer of software that
deals directly with your computer hardware.
EE514 Real-Time Computing
12
RTAI A Simplified View
EE514 Real-Time Computing
13
RTAI Overview (continued)
Software
architecture
Interface to Linux hardware management
(HAL)
3 basic components: dispatcher, scheduler,
fifos
1 interface used in user tasks to initialize and
start the components
RTAI
is basically an interrupt dispatcher
(reroutes interrupts to Linux if necessary)
e.g: Disk interrupt
EE514 Real-Time Computing
14
RT-Linux Overview
Open
source Linux project
Supports x86, PowerPC, Alpha
Patch of the regular Linux kernel (simply
install the patch and recompile the kernel)
Provides an RT API for developers
Runs Linux kernel as lowest priority
process
EE514 Real-Time Computing
15
RT-Linux Task Structure
EE514 Real-Time Computing
16
RT-Linux Interrupt Dispatcher
EE514 Real-Time Computing
17
RT-Linux Overview (continued)
RT
tasks are coded as modules
Modules are inserted and removed at
users discretion
Extremely good at handling periodic tasks
Communicates with non-RT kernel and
other RT tasks via fifo queues
Tools are provided for graphical analysis of
RT execution
EE514 Real-Time Computing
18
Problems with RT-Linux
Currently
no support for aperiodic tasks
Not very useful for complex RT systems
Currently limited to simple problems
EE514 Real-Time Computing
19
Montavista A Faster Response Linux
Kernel
Montavista
Inc. provides a linux solution for
embedded systems
The solutions aim is to make the Linux
kernel fully preemptable
It identifies the points where priority
inversion occurs in Linux and makes those
points fully preemptable
A good embedded solution not a complete
RT solution.
EE514 Real-Time Computing
20
Real-Time Linux
A class of Linux systems which could meet the real-time
requirements.
There are a number of such real-time Linux systems
available:
RTLinux hard real-time
RTAI hard real-time
MontaVista soft real-time
and many more
Many of them are developed as research projects and
were released to public freely.
Some of them are commercialized though.
EE514 Real-Time Computing
21
RTLinux or RTCore
The first real-time Linux system developed as a research
project over a decade ago.
Subsequently commercialized to become a real-time core
(RTCore) of a commercial Linux called Wind River Linux.
RTLinux or RTCore is a hard real-time microkernel that
runs the entire Linux operating system as a fully
preemptable process.
The technologies developed to enable the real-time
operation are patented and cant be used freely.
EE514 Real-Time Computing
22
RTAI
RTAI stands for Real-Time Application Interface and is a
Linux extension to hard real-time application.
It is initially developed as a research project in Italy for
enhancing the RTLinux functionalities at that time.
Subsequently, new technologies were invented to replace
all the RTLinux code adopted from the initial RTLinux
prototype to overcome any patent problems.
The new technologies is collectively referred as ADEOS.
Current version is 3.8.1 released in May 2010.
Still actively under development and maintained by the
original developers in Italy and the Internal community.
EE514 Real-Time Computing
23
Beginning of Linux as a UNIX clone
EE514 Real-Time Computing
24
Linux System
Linux kernel
In Sept 1991, Linus Torvalds, a second year student of Computer
Science at the University of Helsinki, developed the preliminary
kernel of Linux, known as Linux version 0.0.1, and released to the
Internet.
Version 1.0, 2.0, 2.2, 2.4, and 2.6 were released in 1994, 1996,
1999, 2001, and 2003, respectively.
Current version is 2.6.37.2 released on 24 Feb 2011.
Linux system
Software from the GNU project + Linux kernel = Linux system
Redhat, Fedora, CentOS, Ubuntu, etc. are popular variants of
Linux distributions.
EE514 Real-Time Computing
25
Evolution of Linux
Early Linux is Not Designed for Real-Time Processing
Early Linux (1.x Kernel) installations on retired Windows PCs
Old/Obsolete hardware useful under Linux due to efficiency of O/S
Linux outperformed Windows in reliability and uptime (still does)
Linux Design: Fairness, Throughput and Resource-Sharing
Basic Unix development design principles applied in Kernel
Heavily (over)-loaded systems continue to make progress
Does not drop network connections or starve users / applications
Fairness- and Resource-Sharing Design is Linux's Strength
contributed to make Linux competitive and popular in the enterpriseserver and development-application environments
Gave rise to RedHat and others.
Essential to the evolution of Linux, endemic of UNIX legacy
EE514 Real-Time Computing
26
Why Linux in Real-Time Systems?
Not because of the Kernels Real-Time Performance!
UNIX-legacy Operating Systems were designed with operating
principles focused on throughput and progress
User tasks should not stall under heavy load
System resources must be shared fairly between users
Fairness, progress and resource-sharing conflict with the
requirements of time-critical applications
VIP vs. General Admission
UNIX systems (and Linux) are historically not Real-Time OS
Linux has lagged many commercial Unix's in Real-Time
performance-enhancement and Real-Time capabilities
Solaris, LynxOS, QNX
SCO
EE514 Real-Time Computing
27
Real time computing requirements
Scheduling
Hardware Interface
Communication and
Synchronization
Memory Management
Task Management
Timing
Configuration
External
Communications
Error Reporting
Embedded Features
File system
Reliability
28
EE514 Real-Time Computing
Why Real-Time in Linux Systems (Embedded)
Most Important Factors Inhibiting Linux Adoption
Data from VDC, Linuxs Future in the Embedded Systems Market, June 2004
EE514 Real-Time Computing
29
Real-Time in Handheld & Embedded Systems
Cost / Performance / Power / Weight Compromise
Competitive, High-Volume, Low-margin Markets
Maximum Feature-set, Add-ons, Responsive UI feel
Device specs: minimal CPU & Memory & Battery Powered
Minimal CPU = High CPU utilization
High CPU load + Time-Critical functionality RT specs
Real-time Requirements will never be alleviated by
Improvements in Hardware Performance / Efficiency
Software utilizing latest hardware technologies easily keep up with,
and usually out-paces, advances in hardware technology
If you don't believe that, go shopping (for a mobile phone)
EE514 Real-Time Computing
30
Real-Time Linux 2.6 Enablers
Pro-Audio Performance Requirements
Audio Community Involved in Kernel-Preemption since 2.2
Audio Community strongly Endorsing RT technology
Embedded Application Domain
Single-Chip, Mobile Applications (Wireless / Cellular Handsets)
Predictable OS performance eliminates HW design uncertainty
Multimedia Carrier (QOS) Application Domain
Reliable Prototyping and Improved Product Scheduling
Telephony, Audio / Video / Multimedia / Home Entertainment
Fine-Granular Preemption improves SMP scalability
Mainstreaming of SMP Technology
Dual / Quad / Octa - Core Intel, AMD, PPC, Arm
EE514 Real-Time Computing
31
Real-Time and Linux Kernel Evolution
Gradual Kernel Optimizations over Time
SMP Critical sections (Linux 2.x)
Low-Latency Patches (Linux 2.2)
Preemption Points / Kernel Tuning (Linux 2.2 / 2.4)
Preemptible Kernel Patches (Linux 2.4)
Fixed-time O(1) Scheduler (Linux 2.6)
Voluntary Preemption (Linux 2.6)
In 2003-04 Linux 2.6 RT Technology Regressed
Early Linux 2.6 Real-Time Performance was worse than 2.4
Kernel Performance
Audio Community and others balked at moving to 2.6 Kernel
Base
EE514 Real-Time Computing
32
Real-Time Inhibitor: Critical Section Locking
Linux 2.6 Kernel Critical Sections are Non-Preemptible
Critical sections protect shared resources, e.g. hardware
registers, I/O ports, and data in RAM
Critical sections are shared by Processes, Interrupts and CPUs.
Effective protection is provided by the Spin-Lock Subsystem
Critical sections must be locked and unlocked
Locked critical sections are not preemptible
Linux 2.6 Kernel has 11,000 critical sections
Exhaustive Kernel testing to identify worst-case code paths
Labour-intensive cleanup of critical sections
No control over 3rd party drivers
Worst-case after cleanup still not acceptable
Maintenance, community education, policing / regression testing
EE514 Real-Time Computing
33
Real-Time Inhibitor: Interrupt Handlers
Linux 2.6 Kernel: Unbounded IRQ subsystem latencies
Task-Preemption latency increases with hardware-interrupt load
Interrupts cannot be preempted
No Priorities for Interrupts
IRQ Subsystem always preempts tasks unconditionally
Unbounded SoftIRQ subsystem (Bottom Half Processing)
Activated by HW IRQs (Timers, SCSI, Network)
SoftIRQs re-activate, iterate
Driver-level adaptations
Network Driver NAPI adaption reduces denial of service (D.o.S.)
effects of high packet loads
EE514 Real-Time Computing
34
Real-Time Inhibitor: Legacy Locking
Existing Locking Subsystems are not Priority-Aware
System semaphore
Counting semaphore used to wake multiple waiting tasks
No support for priority inheritance
No priority ordering of waiters
Big Kernel Lock (BKL)
Originally non-preemptible, now preemptible using system semaphore
Can be released by blocking tasks, re-acquired upon wake-up
No priority-awareness, or priority inheritance for contending tasks
RCU (Read-Copy-Update) Locks in Network subsystem
Read-optimized cached locking requiring race-free invalidation
Read Write Locks
Classical blocking / starvation issues with no priority awareness
EE514 Real-Time Computing
35
The Fully Preemptible Linux Kernel
Dramatic Reduction in 2.6 Preemption Latencies
Multiple Concurrent Tasks in Independent Critical Sections
Generally Fully Preemptible No Delays
Design Flexibility
Provides Full Access to Kernel Resources to RT Tasks
Supports existing driver and application code
User-space Real-Time
Optimization Flexibility
Non-preemptible: Interrupt off paths and lowest-level interrupt management
Non-preemptible: Scheduling and context switching code
RT Tasks designed to use Kernel-resources in managed ways can
reduce or eliminate Priority-Inheritance delays
Adequate Instrumentation
Latency timing, latency triggers & stack tracing, histograms
EE514 Real-Time Computing
36
Kernel Evolution: Preemptible Code
Kernel 2.0
Kernels
2.2-2.4
Preemptible
Kernel 2.4
Kernel 2.6
Real-Time
Kernel 2.6
Preemptible
EE514 Real-Time Computing
Non-Preemptible
37
Real-Time Linux 2.6 Performance
Real-Time Linux 2.6 Kernel Performance
Far exceeds most stringent Audio performance requirements
Enables sub-millisecond control-loop response
Enables Hard Real Time for qualified RT-aware Applications
SMP Kernel Performance
SMP-safe code is by definition preemptible
Any code that allows concurrent execution by multiple CPUs, also
allows context switching and therefore preemption
Increased preemptible code surface in the Kernel also increases
SMP throughput / efficiency
EE514 Real-Time Computing
38
Real-Time Linux 2.6 Performance
(Cont)
No Preemption
Preemptible
MontaVista Linux 4.0
Target machine:
CPU utilization during test:
Intel Celeron 800 MHz
Real-Time
Preemption
100% most of the time
Test Duration:
20 hours
EE514 Real-Time Computing
39
RTLinux / RTAI Architecture
Users task
System libraries
Real Time task
Linux kernel
Drivers
I/O
Real-time Scheduler
Software interrupts
Real-time plugin
I/O
Hardware interrupts
HARDWARE
Hardware abstraction layer
real-time interrupt dispatcher, real-time scheduler
Inter processes communication services
EE514 Real-Time Computing
40
Interrupt Latency
The time stamp is frozen when the interrupt is generated
The current time is read when the interrupt is
acknowledged
The difference gives the interrupt latency
Dual-Xeon
3.2GHz
EE514 Real-Time Computing
Dual-core
Opteron
2GHz
41
Freescale dual-kernel, real-time solutions
(BSP) for Linux (Case studies)
Solutions provide sub-20 microsecond precision with
hard, real-time scheduling guarantees but at the
expense of increased programming effort and
memory protection
These techniques offer less than 250 sec latency
performance for greater than 99.5 percent of all
scheduled task under demanding system test loads
Under real-world conditions, performance is near the
99.5 percent, 100 sec mark
BSP : board support service
EE514 Real-Time Computing
42
Linux kernel improvements for real time
Features and performance enhancement for
upgrading standard Linux kernel to challenge real
time products and applications:
Enhanced schedulers
Virtual memory
Shared memory
Portable operating system interface X (POSIX) timers
Real-time signals
POSIX asynchronous I/O
POSIX threads
Quality of service capabilities
Low latency/preemptable kernel modifications
EE514 Real-Time Computing
43
Quality of sercice
As for quality of service, Linux v2.4 offers a
sophisticated component for bandwidth
management called Traffic Control. The
component supports methods for classifying,
prioritizing and limiting both incoming and
outgoing traffic. Linux can perform:
Limit bandwidth for certain computers
Help fairly share bandwidth
Protect the Internet from abuses
Restrict access and do routing based on user ID, MAC
address, source IP address and so on
EE514 Real-Time Computing
44
Linux network subsystem (Traffic
control)
EE514 Real-Time Computing
45
Queueing discipline- traffic control
Queueing
discipline provides a method to
enqueue a packet
EE514 Real-Time Computing
46
Low-Latency patches for Linux
The kernel latency is a quantity used to measure
the difference between the theoretical schedule
and the actual one
In a standard Linux kernel, the maximum latency
is equal to the maximum length of a system call
plus the processing time of all the interrupts that
fire before returning to user mode. This value can
be as large as 100 ms
EE514 Real-Time Computing
47
Low-Latency Linux (Red-Linux, some
Real-time
Linux)
This approach corrects the monolithic structure by
inserting explicit rescheduling points inside the
kernel. In this approach, when a thread is executing
inside the kernel it can explicitly decide to yield the
CPU to some other thread
In this way, the size of non-preemptable sections is
reduced, thus decreasing the latency
The consistency of kernel data is enforced by suing
cooperative scheduling
Since the low-latency patch has been carefully handtuned for quite a long time, it performs surprisingly
well
EE514 Real-Time Computing
48
Preemptable Linux (used in most realtime
systems)
Removes the constraint of a single execution flow
inside the kernel. It is not necessary to disable
preemption when an execution flow enters the kernel
To support full kernel preemptability, kernel data must
be explicitly protected using mutexes or spinlocks
The Linux preemptable kernel pathc, maintained by
Robert Love and sponsored by MontaVista
EE514 Real-Time Computing
49
Evaluation
EE514 Real-Time Computing
50
Probability density function of the
latency in the low-latency kernel
EE514 Real-Time Computing
51
Summary of the Freescale case studies
Linux is rapidly evolving to include more and more
features that have explicit value to the embedded
developer
Key enhancements to the Linux kernel, which
perform identically for all practical purposes
The developers choice regarding which should be
implemented can be based on independent criteria
including the ability to update/modify existing device
drivers for compatibility with the real-time
implementation, and the availability of each real-time
technique for the selected architecture and kernel
version
EE514 Real-Time Computing
52
References
RT-Linux
: http://www.rtlinux.org
RTAI
:
http://www.aero.polimi.it/projects/rtai/contri
b.htm
Montavista: http://www.mvista.com
Linux as a real-time operating system
Freescale semiconductor, David Beal,
Nov/2005
EE514 Real-Time Computing
53
EE514 Real-Time Computing
Real Time Linux
54
EE514 Real-Time Computing
Real Time Linux
55
