Security in Operating Systems

• The operating system is the first line of defense against all sorts of unwanted
behavior. It protects one user from another, ensures that critical areas of memory
or storage are not overwritten by unauthorized processes, performs identification
and authentication of people and remote operations, and ensures fair sharing of
critical hardware resources.
• As the powerful traffic cop of a computing system, it is also a tempting target for
attack because the prize for successfully compromising the operating system is
complete control over the machine and all its components.
• The operating system is the fundamental controller of all system resources—
which makes it a primary target of attack, as well.

1
 OS Security: Overview
• What is protection in OS?
• Prevention of mischievous, intentional violation of an access restriction by a
user.
• Prevention of erroneous program from affecting the execution of other
programs.
• Improve reliability.
• OS protection includes these:
• Memory protection
• File protection
• General control of access to objects
• User authentication

2
 Background: Operating System Structure
• An operating system is an executive or supervisor for a piece of computing
machinery. Operating systems are not just for conventional computers. Some
form of operating system can be found on any of the following objects:
• a dedicated device such as a home thermostat or a heart pacemaker
• an automobile (especially the engine performance sensors and the
automated control functions such as antilock brakes); similarly, the avionics
components of an airplane or the control system of a streetcar
• a smartphone, tablet, or other web appliance
• a network appliance, such as a firewall or intrusion detection and prevention
system
• a (computer) network traffic management device
• computers—from microcomputers to laptops to huge mainframes—have
operating systems.
• The nature of an operating system varies according to the complexity of the
device on which it is installed, the degree of control it exercises, and the amount
of interaction it supports, both with humans and other devices.
3
 Security Features of Operating Systems
• A multiprogramming operating system performs several functions that relate to security. To see
how, examine Figure, which illustrates how an operating system interacts with users, provides
services, and allocates resources.

Operating System Functions 4

 Security Features of Operating Systems
• We can see that the system addresses several particular functions that involve computer
security:
• Enforced sharing. Resources should be made available to users as appropriate. Sharing brings
about the need to guarantee integrity and consistency. Table lookup, combined with integrity
controls such as monitors or transaction processors, is often used to support controlled
sharing.
• Interprocess communication and synchronization. Executing processes sometimes need to
communicate with other processes or to synchronize their accesses to shared resources.
Operating systems provide these services by acting as a bridge between processes,
responding to process requests for asynchronous communication with other processes or
synchronization. Interprocess communication is mediated by access control tables.
• Protection of critical operating system data. The operating system must maintain data by
which it can enforce security. Obviously, if these data are not protected against unauthorized
access (read, modify, and delete), the operating system cannot provide enforcement. Various
techniques (including encryption, hardware control, and isolation) support protection of
operating system security data.
• Guaranteed fair service. All users expect CPU usage and other service to be provided so that
no user is indefinitely starved from receiving service. Hardware clocks combine with 5
scheduling disciplines to provide fairness.
 Security Features of Ordinary Operating Systems
• Interface to hardware. All users access hardware functionality. Fair access and
controlled sharing are hallmarks of multitask operating systems (those running more
than one task concurrently), but a more elementary need is that users require access to
devices, communications lines, hardware clocks, and processors. Few users access
these hardware resources directly, but all users employ such things through programs
and utility functions. Hardware interface used to be more tightly bound into an
operating system’s design; now, however, operating systems are designed to run on a
range of hardware platforms, both to maximize the size of the potential market and to
position the operating system for hardware design enhancements.
• User authentication. The operating system must identify each user who requests
access and must ascertain that the user is actually who he or she purports to be. The
most common authentication mechanism is password comparison.
• Memory protection. Each user’s program must run in a portion of memory protected
against unauthorized accesses. The protection will certainly prevent outsiders’
accesses, and it may also control a user’s own access to restricted parts of the program
space. Differential security, such as read, write, and execute, may be applied to parts of
a user’s memory space. Memory protection is usually performed by hardware 6
mechanisms, such as paging or segmentation.
 Security Features of Ordinary Operating Systems
• File and I/O device access control. The operating system must protect user and
system files from access by unauthorized users. Similarly, I/O device use must be
protected. Data protection is usually achieved by table lookup, as with an access
control matrix.
• Allocation and access control to general objects. Users need general objects, such as
constructs to permit concurrency and allow synchronization. However, access to
these objects must be controlled so that one user does not have a negative effect on
other users. Again, table lookup is the common means by which this protection is
provided.
• Operating systems show several faces: traffic director, police agent, preschool
teacher, umpire, timekeeper, clerk, and housekeeper, to name a few.
• The primitive functions of an operating system are called kernel functions, because
they are basic to enforcing security as well as the other higher-level operations an
operating system provides. Indeed, the operating system kernel, is the basic block
that supports all higher-level operating system functions.

7
 Operating System Design to Protect Objects
• A well-structured operating system also implements several levels of function and
protection, from critical to cosmetic. This ordering is fine conceptually, but in
practice, specific functions span these layers. One way to visualize an operating
system is in layers, as shown in Figure.
• The figure shows functions arranged from most critical (at the bottom) to least
critical (at the top). When we say “critical,” we mean important to security. So, in
this figure, the functions are grouped in three categories: security kernel (to
enforce security), operating system kernel (to allocate primitive resources such
as time or access to hardware devices), and other operating system functions (to
implement the user’s interface to hardware).
• Above the operating system come system utility functions and then the user’s
applications. In this figure the layering is vertical; other designers think of
layering as concentric circles. The critical functions of controlling hardware and
enforcing security are said to be in lower or inner layers, and the less critical
functions in the upper or outer layers.

8
Operating System Design to Protect Objects

Layered Operating System 9

 Operating System Design to Protect Objects
• Consider password authentication as an example of a security-relevant operating system activity.
• This activity includes several different operations, I
→ displaying the box in which the user enters a password,
→ receiving password characters but echoing a character such as *,
→ comparing what the user enters to the stored password,
→ checking that a user’s identity has been authenticated,
→ modifying a user’s password in the system table.
• Changing the system password table is certainly more critical to security than displaying a box
for password entry, because changing the table could allow an unauthorized user access but
displaying the box is merely an interface task. The functions listed would occur at different levels
of the operating system. Thus, the user authentication functions are implemented in several
places, as shown in Figure.

10
Operating System Design to Protect Objects

Authentication Functions Spanning Layers in an Operating

11
System
 Operating System Modules
• A modern operating system has many different modules, as depicted in Figure. Not all this code
comes from one source. Hardware device drivers may come from the device manufacturer or a
third party, and users can install add-ons to implement a different file system or user interface, for
example.
• As you can guess, replacing the file system or user interface requires integration with several
levels of the operating system. System tools, such as antivirus code, are said to “hook” or be
incorporated into the operating system; those tools are loaded along with the operating system
so as to be active by the time user programs execute.
• Even though they come from different sources, all these modules, drivers, and add-ons may be
collectively thought of as the operating system because they perform critical functions and run
with enhanced privileges.

12
Operating System Design to Protect Objects

• From a security standpoint these modules come from

different sources, not all trustworthy, and must all integrate
successfully.
• Operating system designers and testers have a nightmarish
job to ensure correct functioning with all combinations of
hundreds of different add-ons from different sources.
• All these pieces are maintained separately, so any module
can change at any time, but such changes risk
incompatibility.
• Also, the operating system must protect itself in order to
protect its users and resources.

Operating System Modules

13
 Operating system is loaded in stages
• An operating system is loaded in stages, as shown
in Figure. The process starts with basic I/O support
for access to the boot device, the hardware device
from which the next stages are loaded.
• Next the operating system loads something called
a bootstrap loader, software to fetch and install the
next pieces of the operating system, pulling itself in
by its bootstraps, hence the name.
• The loader instantiates a primitive kernel, which
builds support for low-level functions of the
operating system, such as support for
synchronization, interprocess communication,
access control and security, and process
dispatching.
• Those functions in turn help develop advanced
functions, such as a file system, directory
structure, and third-party add-ons to the operating
system. At the end, support for users, such as a
graphical user interface, is activated. Operating System Loaded in Stages

14
 Separation and Sharing
• The basis of protection is separation: keeping one user’s objects separate from other users.
• John Rushby and Brian Randell [RUS83 ] note that separation in an operating system can occur in several
ways:
• physical separation, by which different processes use different physical objects, such as separate printers for
output requiring different levels of security
• temporal separation, by which processes having different security requirements are executed at different
times
• logical separation, by which users operate under the illusion that no other processes exist, as when an
operating system constrains a program’s accesses so that the program cannot access objects outside its
permitted domain
• cryptographic separation, by which processes conceal their data and computations in such a way that they
are unintelligible to outside processes. Separation occurs by space, time, access control, or cryptography.
• Of course, combinations of two or more of these forms of separation are also possible. The categories of
separation are listed roughly in increasing order of complexity to implement, and, for the first three, in
decreasing order of the security provided. However, the first two approaches are very stringent and can lead
to poor resource utilization. Therefore, we would like to shift the burden of protection to the operating
system to allow concurrent execution of processes having different security needs.
15
 Separation and Sharing
• But separation is only half the answer. We generally want to separate one user from another user’s
objects, but we also want to be able to provide sharing for some of those objects. For example, two
users with two bodies of sensitive data may want to invoke the same search algorithm or function
call. We would like the users to be able to share the algorithms and functions without compromising
their individual data. An operating system can support separation and sharing in several ways,
offering protection at any of several levels.
• Do not protect. Operating systems with no protection are appropriate when sensitive procedures
are being run at separate times.
• Isolate. When an operating system provides isolation, different processes running concurrently are
unaware of the presence of each other. Each process has its own address space, files, and other
objects. The operating system must confine each process somehow so that the objects of the other
processes are completely concealed.
• Share all or share nothing. With this form of protection, the owner of an object declares it to be
public or private. A public object is available to all users, whereas a private object is available only
to its owner.

16
 Separation and Sharing
• Share but limit access. With protection by access limitation, the operating system checks the
allowability of each user’s potential access to an object. That is, access control is implemented for
a specific user and a specific object. Lists of acceptable actions guide the operating system in
determining whether a particular user should have access to a particular object. In some sense,
the operating system acts as a guard between users and objects, ensuring that only authorized
accesses occur.
• Limit use of an object. This form of protection limits not just the access to an object but the use
made of that object after it has been accessed. For example, a user may be allowed to view a
sensitive document but not to print a copy of it. More powerfully, a user may be allowed access
to data in a database to derive statistical summaries (such as average salary at a particular grade
level), but not to determine specific data values (salaries of individuals).
• Again, these modes of sharing are arranged in increasing order of difficulty to implement, but also
in increasing order of fineness (which we also describe as granularity) of protection they provide.
A given operating system may provide different levels of protection for different objects, users, or
situations.

17
 Hardware Protection of Memory
• Here, we describe several ways of protecting a memory space. We want a program to be
able to share selected parts of memory with other programs and even other users, and
especially we want the operating system and a user to coexist in memory without the
user’s being able to interfere with the operating system.
• Even in single-user systems, as you have seen, it may be desirable to protect a user from
potentially compromisable system utilities and applications. Although the mechanisms
for achieving this kind of sharing are somewhat complicated, much of the
implementation can be reduced to hardware, thus making sharing efficient and highly
resistant to tampering.
• Memory protection implements both separation and sharing.

18
 Hardware Protection of Memory: Fence
• The simplest form of memory protection was
introduced in single-user operating systems, to
prevent a faulty user program from destroying
part of the resident portion of the operating
system. As its name implies, a fence is a method to
confine users to one side of a boundary.
• In one implementation, the fence was a
predefined memory address, enabling the
operating system to reside on one side and the
user to stay on the other. An example of this
situation is shown in Figure.
• Unfortunately, this kind of implementation was
very restrictive because a predefined amount of
space was always reserved for the operating
system, whether the space was needed or not. If
less than the predefined space was required, the
excess space was wasted. Conversely, if the
operating system needed more space, it could not
grow beyond the fence boundary.

Fence Protection
19
 Hardware Protection of Memory: Fence
• Another implementation used a hardware register,
often called a fence register, containing the
address of the end of the operating system.
• In contrast to a fixed fence, in this scheme the
location of the fence could be changed. Each time
a user program generated an address for data
modification, the address was automatically
compared with the fence address.
• If the address was greater than the fence address
(that is, in the user area), the instruction was
executed; if it was less than the fence address (that
is, in the operating system area), an error condition
was raised. The use of fence registers is shown in
Figure.
• A fence register protects in only one direction. In
other words, an operating system can be protected
from a single user, but the fence cannot protect
one user from another user. Similarly, a user
cannot identify certain areas of the program as
inviolable (such as the code of the program itself Fence Registers
or a read-only data area). 20
 Security in the Design of Operating Systems
• Operating systems are complex pieces of software. The components come from many sources, some pieces
are legacy code to support old functions; other pieces date back literally decades, with long-forgotten design
characteristics. May be some pieces were written just yesterday.
• Old and new pieces must interact and interface successfully, and new designers must ensure that their code
work correctly with all existing previous versions.
• Not all complex software is vulnerable to attack. The point we are making is that the more complex the
software, the more possibilities for unwanted software introduction.
• A house with no windows leaves no chance for someone to break in through a window, but each additional
window in a house design increases the potential for this harm and requires the homeowner to apply more
security.
• Now extend this metaphor to modern operating systems that typically include millions of lines of code:
What is the likelihood that every line is perfect for its use and fits perfectly with every other line?

21
Simplified Design
• OS performs many duties.
• It must minimize overhead and should not slow user computations
and interactions.
• Adding the responsibility of security enforcement to the operating
system increases the difficulty of design. So, design of operating
system for a high degree of security should start at the beginning of
design of OS
Layered Design
• An operating system consists of at least four levels: hardware, kernel,
operating system, and user.
• Each of these layers can include sublayers.
• most sensitive operations are performed by the innermost layers.
• inner layer (fundamental part) of operating system must control the
accesses of all outer or higher layers (to enforce separation).
• Each layer provides a certain level of functionality to the layers farther
out. In this way, we can “peel off” each layer and still have a logically
complete system with less functionality
• In a conventional,
nonhierarchically designed OS
(shown in Table 5-1), any
problem—hardware failure,
software flaw, or unexpected
condition, can cause disaster
because the effect of the
problem is unbounded
• as shown in Table 5-2, hierarchical
structuring has two benefits:
• Hierarchical structuring permits
identification of the most critical
parts, which can then be analyzed
intensely for correctness, so the
number of problems should be
smaller.
• Isolation limits effects of problems
to the hierarchical levels at and
above the point of the problem, so
the harmful effects of many
problems should be confined.
Kernelized Design
• Kernal, also called a nucleus or core, is fundamental part of OS. It performs basic
operations such as synchronization, interprocess communication, message
passing, and interrupt handling.
• A security kernel is responsible for enforcing the security mechanisms of the
entire operating system.
• It provides the security interfaces among the hardware, operating system, and
other parts of the computing system.
Note: Security kernel is designed either as an addition to the operating system or
as the basis of the entire operating system. Usually, the security kernel is contained
within the operating system kernel.
Note: Implementing a security kernel may degrade system performance because
the kernel adds another layer of interface between user programs and operating
system resources.
Why are security functions isolated in a
security kernel?
• Coverage: Every access to a protected object must pass through the
security kernel to ensure that every access is checked.
• Separation: Isolating security mechanisms from operating system
and user space, makes it easier to protect those mechanisms from
penetration by the operating system or the users.
• Unity: All security functions are performed by a single set of code, so
it is easier to trace the cause of any problems that arise with these
functions.
• Modifiability: Changes to the security mechanisms are easier to make
and easier to test.
• Compactness: Because it performs only security functions, the
security kernel is likely to be relatively small.
• Verifiability: Being relatively small, the security kernel can be
analyzed rigorously.