18/02/2025

 

Lecturer: Nguyễn Thị Thanh Vân – FIT - HCMUTE

 Introduction To OS and OS Security

 System Security Planning
 The Components of an OS Security Environment
 Vulnerabilities of OS
 Secure an operating system
 Operating Systems Hardening
o Linux/Unix Security
o Windows Security
 Virtualization Security

18/02/2025 2

1
 18/02/2025

 Operating system: collection of

programs that allows user to
operate computer hardware

 Three layers:
o Inner layer, computer hardware
o Middle layer, operating system
o Outer layer, different software

2
 18/02/2025

 Key functions of an operating system:

o Multitasking, multisharing
o Computer resource management
o Controls the flow of activities
o Provides a user interface
o Administers user actions and accounts
o Runs software utilities and programs
o Enforce security measures
o Schedule jobs
o Provide tools to configure the operating system and hardware

3
 18/02/2025

 User mode
 Kernel mode
o Kernel mode drive
o Microkernel

• User space

• Kernel space

• System call Interface

• Kernel

• Dependent kernel code

18/02/2025 8

4
 18/02/2025

 Comparison Windows & Linux.

 Version:
• Linux
0.01. 17 September 1991;
Initial release
22 years ago (1991-09-17))
Latest release 4.15-rc8 (14 January 2018)

• Windows NT:

3.1. July 27, 1993 (1993-07-27)

Initial release
(as Windows NT 3.1)
1709 (10.0.16299.192)
Latest release
(January 3, 2018)

 Much variation
o Early OSes in assembly language
o Then system programming languages like Algol, PL/1
o Now C, C++
 Actually, usually a mix of languages
o Lowest levels in assembly
o Main body in C
o Systems programs in C, C++, scripting languages like PERL,
Python, shell scripts
 More high-level language easier to port to other hardware
o But slower
 Emulation can allow an OS to run on non-native hardware

18/02/2025 10

5
 18/02/2025

18/02/2025 11

 https://www.cvedetails.com/

18/02/2025 12

6
 18/02/2025

Why do we need to trust the operating system?

(a trusted computing base or TCB)
What requirements must it meet to be trusted?
• HĐH đáng tin cậy thường đề cập đến một HĐH cung cấp đủ hỗ trợ
cho bảo mật đa cấp và bằng chứng về tính chính xác để đáp ứng một
bộ yêu cầu cụ thể
• HĐH đó tồn tại một mức độ tin cậy (dựa trên phân tích và thử nghiệm
nghiêm ngặt) rằng các nguyên tắc và cơ chế bảo mật (ví dụ: tách biệt, cô
lập, đặc quyền tối thiểu, kiểm soát truy cập, đường dẫn đáng tin cậy, xác
thực và thực thi chính sách bảo mật) được triển khai chính xác và hoạt
động như mong muốn ngay cả khi có hoạt động đối đầu.
• HĐH đáng tin cậy dựa trên cơ sở tính toán đáng tin cậy (Trusted
computing base - TCB)
• Nó phải đáp ứng những yêu cầu nào để được tin cậy

TCB Requirements:

1. Complete mediation
• The reference validation mechanism must always be invoked
(before executing security-sensitive operations)
2. Tamperproof
• The reference validation mechanism must be tamperproof
3. Verifiable
• The reference validation mechanism must be small enough
to be subject to analysis and tests, the completeness of
which can be assured

7
 18/02/2025

1. Bảo vệ bộ nhớ: kiểm soát quyền truy cập vào các phần bị hạn chế của không gian c.trình.
2. Bảo vệ tệp: ngăn các chương trình thay thế các tệp hệ điều hành quan trọng.
3. Kiểm soát truy cập đối tượng chung: không gây ảnh hưởng tiêu cực đến những user khác
4. Xác thực người dùng: password, sinh trắc học…
5. Kiểm soát truy cập thiết bị I/O
6. Dịch vụ công bằng được đảm bảo
7. Chính sách: Yêu cầu bảo mật, được xác định rõ ràng, nhất quán
8. Mô hình: Biểu diễn chính sách, chính thức. Không được làm giảm chức năng.
9. Thiết kế: Bao gồm chức năng, tùy chọn triển khai
10. Tin cậy: Xem xét các tính năng, đảm bảo khiến hệ điều hành trở nên đáng tin cậy.

15

TCB Controls access to protected resources

●Must establish the source of a

request for a resource
(authentication is how we do it)

●Authorization or access control

●Mechanisms that allow various

policies to be supported

8
 18/02/2025

 Least privilege for users and programs

 Economy
o Trusted code small
o Easier to analyze & test
 Open design
 Complete mediation
o Access check,
o Prevent the bypass
 Fail-safe default
o Deny: default
 Ease of use

18/02/2025 17

Standard Architecture: Trusted Architecture: separate

Open - after the user passes initial components from each other and make
authentication access between areas more difficult

18

9
 18/02/2025

 Hardware Support: Protected Mode - Protected Virtual

Address Mode, is an operating mode of x86-compatible
central processing units (CPUs).
o Virtual Memory,
o Paging
o Safe Multitasking is designed to increase the operating system's
control over application software

18/02/2025 19

 Processor execution modes: ensure the safety of the

kernel, avoid user program intrusion into the kernel
o kernel mode: high level of privilege. It has direct access to the
computer hardware that all programs running in Kernel mode,
including the operating system, share the same address space
o user mode: applications have less privileges, do not have direct
access to hardware resources and cannot write to the address
space of other applications
 OS only allows some processes to run in Kernel mode.
o ensure that there are no problems when running in Kernel mode
- which can cause the entire operating system to crash

18/02/2025 20

10
 18/02/2025

 More trusted processes operate

within lower numbers ring
o Inner Ring: more Privileges
o Outer Rings: less
 Rings:
o Define Access Level to resource.
 Protect system integrity
o Protect kernel from services
o Protect services from apps
o So on..

18/02/2025 21

 A compromised OS can
compromise a database
environment
 Physically protect the
computer running the OS
(padlocks, chain locks,
guards, cameras)
 Model:
o Bank building (operating system)
o Safe (database)
o Money (data)

22

11
 18/02/2025

 Used as access
points to the
database
 Three components:
o Services
o Files
o Memory

23

 Common threats:
o File permission
o File sharing
 Files must be protected from unauthorized reading and
writing actions
 Data resides in files; protecting files; protects data

24

12
 18/02/2025

 Read, write, and execute privileges

 In Windows:
o Change permission on the Security tab on a file’s Properties box
o Allow indicates grant; Deny indicates revoke
 In UNIX/Linux
o Three permission settings: owner; group to which owner belongs;
all other users. Each setting consist of rwx
o CHMOD command used to change file permissions

o Ex: chmod 644 test

25

$ chmod 644 mail_list

26

13
 18/02/2025

 FTP (File Transfer Protocol):

o Internet service for transferring files from one computer to another
o Transmits usernames and passwords in plaintext
o Root account cannot be used with FTP
o Anonymous FTP: ability to log on to the FTP server without being
authenticated
 Best practices:
o Use Secure FTP utility if possible
o Make two FTP directories:
• One for uploads with write permissions only
• One for downloads with read permissions only
o Use specific accounts with limited permissions
o Log and scan FTP activities
o Allow only authorized operations

27

 Naturally leads to security risks and threats

 Peer-to-peer programs: allow users to share
files over the Internet
 Reasons for blocking file sharing:
o Malicious code
o Adware and spyware
o Privacy and confidentiality
o Pornography
o Copyright issues

28

14
 18/02/2025

 Hardware memory available on the system can be

corrupted by badly written software
 Can harm data integrity
 Two options:
o Stop using the program
o Apply a patch (service pack) to fix it
 Safe and efficient memory usage:
o Memory division User space & Kernel space

 Prevent one program or user from interfering with another

user program's memory space:
o Segmentation
o Paging
29

 Services
o Main component of operating system security
environment
o Used to gain access to the OS and its features
 Include
o User authentication
o Remote access
o Administration tasks
o Password policies

30

15
 18/02/2025

 Authentication: Verifies user access to the operating system

 Physical authentication:
o Allows physical entrance to company property
o Magnetic cards and biometric measures
 Digital authentication: verifies user identity by digital means
 Digital certificates: identifies and verifies holder of certificate
 Digital token (security token):
o Small electronic device
o Displays a number unique to the token holder;
o Uses a different password each time
 Digital card: Also known as a security card or smart card
o Similar to a credit card; uses an electronic circuit instead of a magnetic strip
o Stores user identification information
 Kerberos:
o Developed by MIT
o Uses tickets for authentication purposes
31

 Process that decides whether users are permitted to

perform the functions they request
 Authorization is not performed until the user is
authenticated
 Deals with privileges and rights

32

16
 18/02/2025

 Create user accounts

 Set password policies
 Grant privileges to users
 Best practices:
o Use a consistent naming convention
o Always provide a password to an account and force the user to
change it at the first logon
o Protect passwords
o Do not use default passwords

33

 Best practices (continued):

o Create a specific file system for users
o Educate users on how to select a password
o Lock non-used accounts
o Grant privileges on a per host basis
o Do not grant privileges to all machines
o Use ssh, scp, and Secure FTP
o Isolate a system after a compromise
o Perform random auditing procedures

34

17
 18/02/2025

 Tool must widely used by public

 May be the tool must frequently used by hackers:
o Viruses; Worms; Spam; Others
 Used to send private and confidential data as well as offensive
material
 Used by employees to communicate with:
o Clients
o Colleagues
o Friends
 Recommendations:
o Do not configure e-mail server on the same machine where sensitive
data resides
o Do not disclose technical details about the e-mail server

35

 Top vulnerabilities to Windows systems:

o Internet Information Services (IIS)
o Microsoft SQL Server (MSSQL)
o Windows Authentication
o Internet Explorer (IE)
o Windows Remote Access Services National Vulnerability
o Microsoft Data Access Components (MDAC) Database:
o Windows Scripting Host (WSH) http://nvd.nist.gov/
o Microsoft Outlook and Outlook Express
o Windows Peer-to-Peer File Sharing (P2P)
o Simple Network Management Protocol (SNMP)

36

18
 18/02/2025

 Top vulnerabilities to UNIX systems:

o BIND Domain Name System National Vulnerability
o Remote Procedure Calls (RPC)
Database:
o Apache Web Server http://nvd.nist.gov/
o General UNIX authentication accounts with no passwords or
weak passwords
o Clear text services
o Sendmail
o Simple Network Management Protocol (SNMP)
o Secure Shell (SSH)
o Misconfiguration of Enterprise Services NIS/NFS
o Open Secure Sockets Layer (SSL)

37

 Basic steps
o Install and patch the operating system.
o Harden and configure the OS to adequately address the
identified security needs of the system by:
• Removing unnecessary services, applications, and protocols.
• Configuring users, groups, and permissions.
• Configuring resource controls.
o Install and configure additional security controls, such as anti-
virus, hostbased firewalls, and IDS, if needed.
o Test the security of the basic OS to ensure that the steps taken
adequately address its security needs.

18/02/2025 38

19
 18/02/2025

 Hardening:
o attempting to make OS bulletproof.
o Ideally - leave OS exposed to the general public on the Internet
without any other form of protection.
o A hardened system should serve only one purpose--it's a Web
server or DNS or Exchange server, and nothing else. These
systems need too many functions to be properly hardened.

18/02/2025 39

 Disable all unnecessary services.

o determine which services can be disabled.
• Remote Procedure Call (RPC) service.
• little documentation exists to identify what services a given purpose will
require.
• knowing which services are required and which can be disabled is
largely a matter of trial and error.
 Remove all unnecessary executables and registry
entries.
o Forgetting to remove unneeded executables and registry entries
might allow an attacker to invoke something that had previously
been disabled.
 Apply appropriately restrictive permissions to files,
services, end points and registry entries.
o Inappropriate permissions could give an attacker an opening.
o The ability to launch CMD.EXE as "LocalSystem," for example, is a
classic backdoor.

18/02/2025 40

20
 18/02/2025

 Adjusting retransmission of SYN-ACKS. This makes

connection responses time out more quickly during a SYN flood .
 Determining how many times TCP retransmits an
unacknowledged data segment on an existing connection. TCP
retransmits data segments until they are acknowledged or until this
value expires.
 Disabling ICMP Router Discovery Protocol (IRDP) where an
attacker may remotely add default route entries on a remote system.
 Disabling these services: Telnet, Universal Plug and Play Device
Host, IIS, Disable Guest accounts
 Use the Local Security Policy
 Disable File and Print Sharing.
 Disable Remote Assistance and Remote Desktop
 Use NTFS File system.
 Disable auto-logins.

18/02/2025 41

 Encrypt Data Communication

 Avoid Using FTP, Telnet, And Rlogin / Rsh Services
 Minimize Software to Minimize Vulnerability
 One Network Service Per System or VM Instance
 Keep Linux Kernel and Software Up to Date
 Use Linux Security Extensions
 SELinux
 Password: Policy, Aging, Empty
 Login:
o Locking User Accounts After Login Failures
o Make Sure No Non-Root Accounts Have UID Set To 0
o Disable root Login

18/02/2025 42

21
 18/02/2025

 Disable Unwanted Services

 Find Listening Network Ports
 Configure Iptables and TCPWrappers
 Linux Kernel /etc/sysctl.conf Hardening
 Separate Disk Partitions
 Disk Quotas
 Turn Off IPv6
 Disable Unwanted SUID and SGID Binaries
 Logging and Auditing
 Secure OpenSSH Server
 Install And Use Intrusion Detection System
 Disable USB/firewire/thunderbolt devices

18/02/2025 43

 Privilege escalation :
o the act of exploiting a bug,
o design flaw or configuration oversight in an OS or software
application to gain elevated access to resources that are normally
protected from an application or user.
o => The result is that an application with more privileges than
intended by the application developer or system administrator can
perform unauthorized actions.
 2 types:
o Vertical privilege escalation, also known as privilege elevation,
where a lower privilege user or application accesses functions or
content reserved for higher privilege users or applications
(e.g. Internet Banking users can access site administrative functions or the
password for a smartphone can be bypassed.)
o Horizontal privilege escalation, where a normal user accesses
functions or content reserved for other normal users
(e.g. Internet Banking User A accesses the Internet bank account of User B)

18/02/2025 44

22
 18/02/2025

 a rootkit gaining access to the kernel, and the little gate

represents normal privilege elevation

18/02/2025 45

 Privilege rings for the x86 available in protected mode

18/02/2025 46

23
 18/02/2025

• A VM is a software implementation of a machine that execute programs like a

physical machine
• A VM can support individual processes or a complete system depending on
the abstraction level where virtualization occurs.
• Virtualization – a technology that allows running two or more OS side by side
on one PC or embedded controller

•q

24
 18/02/2025

More security implications because of the VM is installed that communicates

reliance on the underlying OS, used in directly with system hardware rather
VMware and MS Virtual PC than relying on a host OS

Hosted Bare - Metal

• Thin Virtualization – reduced size, independence

=> Get Strong Security in a Small Package
• the attack surface much smaller, and reduces the potential for vulnerabilities
• far fewer interfaces to exploit and less malware threats

25
 18/02/2025

• Security Concepts in Architecture

 Extended computing stack
 Guest isolation
 Host Visibility from the Guest
 Virtualized interfaces
 Management interfaces
 Greater co-location of data and assets on one box

 Abstraction and Isolation

 Better Forensics and Faster Recovery After an Attack
 Patching is Safer and More Effective
 More Cost Effective Security Devices
 Future: Leveraging Virtualization to Provide Better
Security

26
 18/02/2025

 VM Sprawl
 Mobility
 Hypervisor Intrusion
 Hypervisor Modification
 Communication
 Denial of Service

Issue Hosted Bare-Metal

susceptible to all the
Vulnerability of the
vulnerabilities and attacks that a much smaller attack surface
underlying OS
are prevalent on such systems.

Sharing of files
there is no mechanism share user
and data between vulnerable to data leakage and
information between virtual machines
the guest and the malicious code intrusion.
and their host.
host

Resource They are at the mercy of the host No single virtual machine can use all
allocation OS and other applications. the resources or crash the system.

- is targeted for environments can potentially be exposed to

where the guest virtual machines malicious users and network traffic.
can be trusted. (software Strong isolation and strict separation
Target Usage
development, testing, of management greatly reduce any
demonstration, and trouble- risk of harmful activity going beyond
shooting.) the boundaries of the virtual machine.

27
 18/02/2025

 Managing oversight and responsibility

 Patching and maintenance
 Visibility and compliance
 VM sprawl
 Managing Virtual Appliances

 Introduction To OS and OS Security

 System Security Planning
 The Components of an OS Security Environment
 Vulnerabilities of OS
 Secure an operating system
 Operating Systems Hardening
o Linux/Unix Security
o Windows Security
 Virtualization Security

18/02/2025 56

28
 18/02/2025

18/02/2025 57

29