Course File (Theory)

Course Title: Parallel And Distributed Computing

Course Code: CS-428
Semester: Spring 2024(final year)
Batch: 2019-20 (Computer Science)
Course Teacher: Mehar Fatima
Department: Computer Science and Information Technology
 NCEAC Checklist (Theory)
Documents included in Course file Source

a. Course Objectives Course Profile

b. Course Contents Course Profile
c. Weekly plan of contents of lectures delivered Course Plan
d. Attendance Record NED Portal -> Teachers Facility
e. Copy of lecture notes Instructor

f. List of Reference Material Instructor

g. Copy of assignments, quizzes, midterms and final examinations Instructor

h. Model solutions of all assessments tests given in (g) above Instructor

i. Three sample graded assignments, quizzes, midterms and final

Instructor
examination securing max, min and average marks
University Grading Scheme + Sessional
j. Marks distribution and Grading Model
Distribution (from Course Plan)
NED Exam Portal,
k. Complete result of the course
CLO-wise marks (sessional and final)
l. Outcomes Assessment *
m. Detail of technology involved Instructor
n. Design skills/techniques practiced Instructor
o. Complete analysis of effectiveness of course and level of skills
Instructor
ensured
* Batches 2019 and 2020: A summary generated by the course instructor.
Batch 2021 and onwards: CLO Report from OBE Portal.
 Lecture1
Introduction to Parallel and Distributed Computing (PDC)

CS482
Computing History
• Introduction to Early Computers:
□ ENIAC and UNIVAC in the 1940s and 1950s.
□ Massive room sized structures with vacuum tubes.
□ Limited processing capabilities compared to contemporary technology.
• Evolution to Mainframes:
• Transition in the mid20th century.
• IBM System/360 as a milestone in the 1960s.
• Room sized mainframe computers with enhanced processing power.
• Personal Computers Era:
• Late 20thcentury rise of personal computers.
• Apple and Microsoft pivotal in popularizing personal computing.
• Introduction of user-friendly interfaces and affordable hardware.
• Apple II (1977) and IBM PC (1981) marked the beginning
• Moore's Law:
• Formulated by Gordon Moore in 1965.
• Predicts doubling of transistors on a microchip every two years.
• Drives exponential growth in processing power.
• Facilitates development of smaller, faster, and more efficient computers.
• Influences technological innovation across various devices.
• Revolutionizes modern life, impacting smartphones to supercomputers..
Serial Computation:

Traditionally software has been written for serial computations:

❑ To be run on a single computer having a single Central Processing Unit (CPU)

❑ A problem is broken into a discrete set of instructions
❑ Instructions are executed one after another
❑ Only one instruction can be executed at any moment in time
Introduction to Parallel Computing
In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve
a computational problem:

□ To be run using multiple CPUs

□ A problem is broken into discrete parts that can be solved concurrently
□ Each part is further broken down to a series of instructions
□ Instructions from each part execute simultaneously on different CPUs
Parallel Computing Cont..
□ Definition:
□ Execution: Simultaneous execution of the same task on multiple processors.
□ Memory: Shared memory architecture or tightly coupled processors.
□ Communication: High bandwidth, low-latency communication within processors.
□ Focus: Solving a single complex problem faster by dividing it into smaller tasks.
□ Characteristics:
□ Speedup:
□ Achieves speedup by breaking a task into parallel subtasks.
□ Synchronization:
□ Requires synchronization to coordinate parallel tasks.
□ Programming Model:
□ Often uses parallel programming models like OpenMP or CUDA.
□ Example:
□ Symmetric Multiprocessing (SMP) systems.
 Flynn’s Taxonomy

□ Flynn’s Taxonomy uses two basic concepts: Parallelism in

instruction stream, and parallelism in data stream.
□ A n CPU system has n program counter, so there are n
“instruction stream” that can execute in parallel.
□ A data stream can be used as a sequence of data, and there
exist 4 possible combinations.
Flynn’s Classification
 Flynn’s Classification

Insrtuction Set Data Name Examples

Streams
1 1 SISD Von Neumann Machine
1 Multiple SIMD Vector Super Computer

Multiple 1 MISD Arguably None

Multiple Multiple MIMD Multiprocessor, Multicomputer
SISD(Single Instruction Single Data)

□ A processor that can only do one job at a time from start to

finish.
 SIMD(Single Instruction Multiple Data)
□ Single CU and multiple PEs
□ CU fetches an instruction from memory and after decoding,
broadcasts control signals to all PEs.
□ That is, at any given time, all PEs are
Synchronously executing the same.
□ Instruction but on different sets of data; hence the name
SIMD
 MISD(Multiple instructions single data)
□ A rare type, since data throughput is limited.
 MIMD(Multiple instruction Multiple Data)
□ A MIMD is a true multiprocessor
□ In contrast to SIMD, a MIMD is a general purpose machine.
□ When all the processor in MIMD are running the same program, we
call it Single Program Multiple Data(SPMD) computation.
□ The SPMD model is widely used by many parallel platforms.
Introduction to Distributed Computing
□ Definition:
□ Distributed computing involves the use of multiple interconnected computers
to solve a single problem or execute a task. It emphasizes collaboration,
communication, and shared resources.
□ Key Concepts:
□ Decentralization:
□ Tasks are divided among multiple machines, reducing reliance on a single central
processing unit.
□ Interconnected Nodes:
□ Computers are linked through a network, enabling communication and data
exchange.
□ Resource Sharing:
□ Resources such as processing power, memory, and storage are shared among the
interconnected nodes.
Distributed Computing Cont..
□ Definition:
□ Execution: Tasks run on geographically dispersed computers.
□ Memory: Each computer has its own memory, not shared.
□ Communication: Relies on message passing over a network.
□ Focus: Solving multiple tasks simultaneously, often involving large datasets.
□ Characteristics:
□ Scalability:
□ Scales horizontally by adding more machines to the network.
□ Autonomy:
□ Each node operates independently with its own memory.
□ Programming Model:
□ Often uses distributed programming models like MapReduce or MPI.
□ Example:
□ Cluster computing, Cloud computing.
Designing parallel programs communication:
Most parallel applications require tasks to share data with each other.

Cost of communication: Computational resources are used to package and transmit data. Requires frequently
synchronization – some tasks will wait instead of doing work. Could saturate network bandwidth.

Latency vs. Bandwidth: Latency is the time it takes to send a minimal message between two tasks. Bandwidth is the
amount of data that can be communicated per unit of time. Sending many small messages can cause latency to dominate
communication overhead.

Synchronous vs. Asynchronous communication: Synchronous communication is referred to as blocking

communication – other work stops until the communication is completed.
Asynchronous communication is referred to as nonblocking since other work can be done while communication is
taking place.

Scope of communication: Pointtopoint communication – data transmission between tasks.

Collective communication – involves all tasks (in a communication group)

This is only partial list of things to consider!

33
Comparison
Parallel Computing Distributed Computing
Memory Architecture Shared memory architecture Each node has its own memory

Communication Highbandwidth, lowlatency within Relies on message passing over a

processors network
Task Focus Solves a single complex problem by Handles multiple tasks
dividing it into smaller tasks simultaneously, often involving large
datasets
Scalability Limited by the number of processors Scales horizontally by adding more
and shared memory capacity machines to the network
Coordination Requires synchronization for Each node operates independently,
coordination coordination is through message
passing
 Parallel Computer Memory Architectures:

Advantages:
❑ Global address space provides a userfriendly programming perspective to memory
❑ Fast and uniform data sharing due to proximity of memory to CPUs

Disadvantages:
❑ Lack of scalability between memory and CPUs. Adding more CPUs increases traffic on the shared
memoryCPU path
❑ Programmer responsibility for “correct” access to global memory
Parallel Computer Memory Architectures:

Distributed Memory:
❑ Requires a communication network to connect interprocessor
memory
❑ Processors have their own local memory. Changes made by one CPU
have no effect on others
❑ Requires communication to exchange data among processors

Advantages:
❑ Memory is scalable with the number of CPUs
❑ Each CPU can rapidly access its own memory without overhead incurred with trying to maintain global cache
coherency

Disadvantages:
❑ Programmer is responsible for many of the details associated with data communication between processors
❑ It is usually difficult to map existing data structures to this memory organization, based on global memory
Parallel Computer Memory Architectures:
Hybrid DistributedShared Memory:

The largest and fastest computers in the world today employ both shared and distributed memory
architectures.

❑ Shared memory component can be a shared memory machine and/or GPU

❑ Processors on a compute node share same memory space
❑ Requires communication to exchange data between compute nodes

Advantages and Disadvantages:

❑ Whatever is common to both shared and distributed memory architectures
❑ Increased scalability is an important advantage
❑ Increased programming complexity is a major disadvantage
□ Thank you
 Lecture 2
Lecture3
Concurrent systems:

CS482
Parallelism in microprocessor
□ Definition: Parallelism in microprocessors refers to the
simultaneous execution of multiple tasks to enhance overall
processing speed and efficiency.
□ Types of Parallelism:
1. InstructionLevel Parallelism (ILP):
Description: Executes multiple instructions in parallel within a single instruction
stream.
Example: Pipelining allows the CPU to process different stages of multiple
instructions simultaneously.
2. DataLevel Parallelism (DLP):
Description: Processes multiple data elements simultaneously.
Example: SIMD (Single Instruction, Multiple Data) instructions enable operations on
multiple data items concurrently.
3. TaskLevel Parallelism (TLP):
Description: Involves the parallel execution of multiple independent tasks.
Example: Multicore processors, where each core handles a distinct task concurrently.
Parallelism in microprocessor (Cont..)
□ Benefits of Parallelism:
1. Increased Throughput:
Parallel execution of tasks results in higher overall processing speed.
2. Improved Performance:
Reduces the time taken to complete complex computations and tasks.
3. Enhanced Efficiency:
Allows for optimal resource utilization, maximizing computational power.
Parallelism in microprocessor (Cont..)
□ Challenges and Considerations:
1. Synchronization:
Ensuring coordinated execution to maintain data integrity and prevent conflicts.
2. Dependency Management:
Handling dependencies between parallel tasks to avoid errors and maintain accuracy.
3. Scalability:
Ensuring effective parallelism with the addition of more processing units.
□ Examples in Modern Processors:
1. Multithreading:
Description: Simultaneous execution of multiple threads within a single processor.
Example: HyperThreading Technology in Intel processors.
2. Multicore Processors:
Description: Integration of multiple processing cores on a single chip.
Example: Intel Core i9, AMD Ryzen processors.
□ Conclusion:
Parallelism in microprocessors significantly enhances computing capabilities by
executing tasks concurrently, leading to improved performance, efficiency, and
throughput.
Architectural classification schemes
Architectural classification schemes in the context of computing refer to the ways in which
computer architectures are categorized based on their design principles, features, and
structures. Here are some common architectural classification schemes:

□ Instruction Set Architecture (ISA):

Classifies architectures based on the type and complexity of instructions the processor can execute.
Examples: CISC (Complex Instruction Set Computing) and RISC (Reduced Instruction Set
Computing).

□ Memory Hierarchy:
Classifies architectures based on the organization and hierarchy of memory components.
Examples: Von Neumann architecture, Harvard architecture, and Cache Memory Hierarchy.

□ Pipelined Architecture:
Classifies architectures based on the use of pipelines for instruction execution.
Examples: Instruction Pipelining and Superscalar Architecture.

□ Parallelism:
Classifies architectures based on the degree of parallel processing employed.
Examples: SIMD (Single Instruction, Multiple Data) and MIMD (Multiple Instruction, Multiple Data)
architectures.
Architectural classification schemes
□ Data Flow Architecture:
Classifies architectures based on the flow of data through the system rather than the control flow.
Examples: Dataflow computers.
□ Von Neumann vs. Harvard Architecture:
Classifies architectures based on how they handle instructions and data storage.
Examples: Von Neumann (single memory space for data and instructions) and Harvard (separate spaces for data and
instructions) architectures.
□ Microarchitecture:
Classifies architectures based on the internal organization and design decisions within a processor.
Examples: Superscalar, VLIW (Very Long Instruction Word), and SIMD microarchitectures.
□ System Organization:
Classifies architectures based on the organization of components within a computing system.
Examples: Single processor systems, Multiprocessor systems, and Multicore systems.
□ Memory Addressing Modes:
Classifies architectures based on how they access and manipulate data in memory.
Examples: Register addressing, Immediate addressing, and Indirect addressing.
□ Fault Tolerance:
Classifies architectures based on their ability to handle faults and errors.
Examples: SISD (Single Instruction, Single Data) vs. MIMD (Multiple Instruction, Multiple Data) fault tolerant
architectures.
These classification schemes help in understanding the design principles, capabilities, and characteristics of
different computer architectures, aiding in the selection and analysis of appropriate systems for specific
applications.
Principles of pipelining and vector processing
□ Dividing Tasks into Stages:
Principle: Pipelining involves breaking down the execution of instructions into discrete stages. Each stage
represents a specific task in the instruction execution process.
□ Parallel Processing of Instructions:
Principle: Different stages of the pipeline operate in parallel, allowing multiple instructions to be in
various stages of execution simultaneously. This increases throughput and overall processing speed.
□ Continuous Flow of Instructions:
Principle: Instructions move through the pipeline continuously. As one instruction completes a stage, the
next instruction enters the pipeline, ensuring a steady flow of operations.
□ Overlap of Execution:
Principle: Pipelining aims to overlap the execution of multiple instructions. While one instruction is in
the execution stage, another can be in the decoding stage, maximizing processor utilization.
□ Stall and Hazard Handling:
Principle: Pipelining may face hazards such as data dependencies or branch instructions. Techniques like
instruction forwarding and branch prediction are employed to handle these hazards and prevent pipeline
stalls.
□ Optimizing Resource Utilization:
Principle: Pipelining optimizes the use of processor resources by allowing different stages to work
concurrently. This reduces idle time and improves overall efficiency.
Principles of pipelining and vector processing
Principles of Vector Processing:
□ Simultaneous Processing of Data Elements:
Principle: Vector processing involves the simultaneous execution of the same operation on multiple data elements. This is achieved
through specialized vector instructions.
□ Vector Registers:
Principle: Vector processors use vector registers to store and manipulate multiple data elements. These registers allow efficient access to
and processing of contiguous data.
□ Vectorization of Code:
Principle: Vector processing requires code to be written or compiled in a way that exploits the capabilities of vector instructions. Loops
and operations are structured to take advantage of parallelism.
□ Parallelism with a Single Instruction:
Principle: Vector processors achieve parallelism by executing a single instruction on multiple data elements concurrently. This contrasts
with scalar processors that operate on individual data items.
□ Enhanced Throughput for Regular Data:
Principle: Vector processing is particularly effective for regular and repetitive data structures, where the same operation is performed on
a large set of data elements.
□ Reduced Instruction Overhead:
Principle: Vector processing minimizes instruction overhead by expressing operations on entire vectors with a single instruction, reducing
the need for individual instructions for each data element.
□ Efficient Memory Access:
Principle: Vector processors often implement techniques like vector prefetching and caching to optimize memory access patterns,
ensuring efficient retrieval of vector data from memory.
Both pipelining and vector processing aim to improve processing speed and efficiency by introducing parallelism. Pipelining
focuses on breaking down instruction execution into stages and overlapping them, while vector processing emphasizes the
simultaneous processing of multiple data elements using specialized vector instructions and registers.
Array Processors
□ Definition:
Array processors are specialized computing units designed for efficiently processing arrays or matrices of data. These processors excel at
performing parallel computations on large sets of data elements simultaneously.
□ Key Characteristics:
1. Parallel Processing:
Description: Array processors leverage parallelism to process multiple elements of an array simultaneously. Each processing element in the array
processor handles a different element of the array concurrently.
2. Specialized Instructions:
Description: Array processors typically come with a set of specialized instructions optimized for array operations. These instructions facilitate efficient
parallel computation on data arrays.
3. Vector and Matrix Operations:
Description: Array processors excel at performing vector and matrix operations. Common operations include addition, multiplication, and other
mathematical transformations applied concurrently to multiple elements.
4. Memory Architecture:
Description: The memory architecture of array processors is designed to support rapid access to array elements. This may involve vector registers or
specialized memory banks to facilitate efficient data retrieval.
5. High Throughput:
Description: Array processors are known for their high throughput when dealing with regular and repetitive data structures. This makes them suitable
for scientific and engineering applications involving large datasets.
6. Scientific and Engineering Applications:
Description: Array processors find extensive use in scientific and engineering computations, such as simulations, signal processing, and numerical
simulations where large arrays of data need to be processed simultaneously.
7. Data Parallelism:
Description: The architecture of array processors emphasizes data parallelism, where the same operation is performed on multiple data elements
concurrently. This aligns with the nature of array based computations.
□ Examples:
Description: Notable examples of array processors include the Connection Machine and the Cray T90 series. Graphics processing units
(GPUs) can also function as array processors, particularly in the context of parallel processing for graphics rendering and general purpose
computing (GPGPU).
Array Processors
□ Advantages:
• Efficiency in Parallel Operations: Array processors are highly efficient in
parallelizing operations on arrays, leading to faster computation times.
• Optimized for Mathematical Operations: The specialized instructions and
architecture make array processors well suited for mathematical computations
common in scientific and engineering applications.
• High Throughput: The parallel processing capabilities contribute to high
throughput, making array processors suitable for handling large datasets.
□ Challenges:
• Limited Applicability: Array processors are specialized and may not be as
versatile as general purpose processors for all types of computations.
• Programming Complexity: Developing software for array processors can be
more complex than traditional programming due to the need to explicitly handle
parallelism.
□ Conclusion: Array processors play a crucial role in accelerating
computations involving large datasets, particularly in scientific and
engineering domains. Their architecture is tailored for efficient parallel
processing of arrays, making them valuable in specific applications where
such parallelism is essential.
Multiprocessor Architecture and Parallel algorithms
□ Multiprocessor Architecture:
□ Definition:
Multiprocessor architecture involves the use of multiple processors or central processing units (CPUs) working
together to execute tasks concurrently. It aims to improve overall system performance and throughput by
parallelizing computations.
□ Types of Multiprocessor Architectures:
1. Shared Memory Multiprocessor (SMP):
□ Multiple processors share a common memory space.
□ Communication occurs through shared memory.
2. Distributed Memory Multiprocessor:
□ Processors have their own local memory.
□ Communication happens via message passing.
□ Advantages:
• Increased Throughput: Multiprocessor systems can execute multiple tasks simultaneously, improving
overall throughput.
• Scalability: Additional processors can be added to enhance system performance as workloads
increase.
• Fault Tolerance: Redundancy in processors allows for continued operation in the presence of
failures.
□ Challenges:
• Synchronization: Coordinating tasks among processors without conflicts.
• Data Sharing and Consistency: Ensuring consistency in shared data across processors.
• Programming Complexity: Developing parallel algorithms for multiprocessor systems can be
complex.
Multiprocessor Architecture and Parallel algorithms
Parallel Algorithms:
□ Definition:
Parallel algorithms are designed to solve computational problems by dividing them into smaller tasks that can be executed simultaneously.
They exploit the parallel processing capabilities of multiprocessor architectures.
□ Types of Parallelism in Algorithms:
Task Parallelism: Divides the overall task into subtasks, each processed concurrently by different processors.
Data Parallelism: Involves processing multiple data elements simultaneously using parallel operations.
Pipeline Parallelism: Breaks down a task into stages, allowing different stages to be executed concurrently.
□ Examples of Parallel Algorithms:
Matrix Multiplication: Divide matrices into submatrices, and perform multiplications concurrently.
Sorting Algorithms: Divide the data into subsets for parallel sorting.
Graph Algorithms: Parallelize graph traversal or search algorithms for faster processing.
□ Advantages:
• Improved Speedup: Parallel algorithms can significantly reduce the time needed to solve problems.
• Efficient Resource Utilization: Multiprocessor systems can concurrently execute different parts of an algorithm, optimizing resource
usage.
• Scalability: Parallel algorithms can scale with the addition of more processors.
□ Challenges:
• Load Balancing: Distributing the workload evenly among processors.
• Communication Overhead: Efficient communication between processors is crucial for parallel algorithm performance.
• Dependency Management: Handling dependencies between parallel tasks.
□ Conclusion: Multiprocessor architecture and parallel algorithms together form a powerful combination to address the
increasing demand for computational power. The efficient utilization of multiple processors in solving complex problems
provides a scalable and high performance computing solution. However, effective design and implementation require
addressing synchronization, data sharing, and communication challenges inherent in parallel systems.
RISC (Reduced Instruction Set Computing) and CISC (Complex
Instruction Set Computing)
□ RISC Architecture:
□ RISC processors are characterized by a simplified set of instructions,
aiming for streamlined execution.
□ The focus is on a smaller, more optimized instruction set, each taking a
single clock cycle to execute.
□ Examples of RISC architectures include ARM processors, widely used in
mobile devices and embedded systems.
□ CISC Architecture:
□ CISC processors have a more extensive and complex set of instructions,
allowing for more powerful and versatile operations.
□ Instructions can vary in length, and a single instruction can perform
multiple low level operations.
□ x86 processors, found in many desktop and server environments, are
prominent examples of CISC architecture.
□ Both RISC and CISC architectures have distinct advantages and use
cases, and the choice between them depends on the specific
requirements of the computing tasks at hand.
RISC Architecture
□ Principles of RISC:
□ RISC, which stands for Reduced Instruction Set Computing, is a processor architecture
that emphasizes simplicity and efficiency in its design.
□ The core principle of RISC is to use a small, highly optimized set of instructions, each
taking a single clock cycle to execute.
□ The goal is to streamline instruction execution, making it faster and more predictable.
□ Advantages of RISC Architecture:
□ Simplicity: A reduced instruction set leads to simpler processor design, making it easier
to optimize and manufacture. This simplicity also facilitates faster instruction execution.
□ Efficiency: With a focus on basic instructions that execute quickly, RISC architectures
are often more efficient in terms of power consumption and overall performance.
□ CompilerFriendly: RISC architectures are typically more compilerfriendly, allowing
compilers to optimize code more effectively.
□ RealWorld Applications and Use Cases:
□ Mobile Devices: RISC architectures, such as ARM processors, are prevalent in mobile
devices due to their efficiency and low power consumption.
□ Embedded Systems: RISC architectures are commonly used in embedded systems,
where compact size and power efficiency are crucial.
□ Networking Equipment: RISC processors find applications in networking equipment,
where fast and predictable execution is essential for routing and packet processing.
CISC Architecture
□ Complex Instruction Set Computing (CISC) architecture is characterized by a diverse and
extensive set of instructions, each capable of performing complex operations.
□ Principles of CISC:
□ CISC processors aim to reduce the number of instructions per program by providing instructions
that can perform multiple lowlevel operations in a single instruction.
□ This design philosophy is based on the idea that more complex instructions can lead to more
efficient programs.
□ Advantages of CISC:
□ Versatility: CISC instructions can perform intricate tasks, reducing the number of instructions
needed for a given operation.
□ Efficiency for Complex Operations: Wellsuited for tasks that require multiple operations, as a
single CISC instruction can handle them.
□ Disadvantages of CISC:
□ Complexity: The extensive instruction set can make the processor architecture more complex,
potentially leading to longer development cycles.
□ Power Consumption: In some cases, CISC architectures may consume more power compared to
RISC due to the complexity of instructions.
□ Examples of CISC Instructions:
□ x86 processors, such as those manufactured by Intel and AMD, are classic examples of CISC
architecture.
 Lecture# #3 5
Lecture
Concurrency Controls
CS-482
Concurrency Control
Concurrency refers to the ability of a system to handle
multiple tasks or processes simultaneously.
Concurrency can be achieved in various ways:
□ Multithreading
□ Multiprocessing
□ Asynchronous Programming
Concurrency introduces challenges such as race conditions,
deadlocks, and resource sharing, which need to be carefully
managed to ensure the correctness and reliability of
concurrent software.
Conflicts of serializabity of transactions
Concurrency in databases refers to the ability of multiple transactions
or operations to be executed simultaneously without causing conflicts
or data inconsistency.
In the context of concurrency control in databases, conflicts can arise
when multiple transactions concurrently access and modify the same
data. There are three main types of conflicts that can occur:
□ Reading uncommitted data(Write-Read (WR) Conflict):
Occurs when one transaction reads uncommitted data that another
transaction writes.
□ Unrepeatable read ( Read-Write (RW) Conflict)
Occurs when one transaction reads data that another transaction writes.
□ Lost update (Write-Write (WW) Conflict)
Occurs when two transactions both write to the same data item.(blind
write)
Why concurrency control in database?
□ Isolation of Transactions:
□ Preventing Lost Updates:
□ Avoiding Dirty Reads:
□ Preventing Inconsistent Reads:
□ Deadlock Prevention:
□ Improving Concurrency:
Synchronization mechanism
These 4 requirements/ condition are crucial for preventing
race conditions and ensuring the correctness of concurrent
programs.
□ Primary condition (Mutual Exclusion, Progress),
□ Secondary condition( Bounded Waiting and no
assumption related to hardware speed)
Process synchronization
□ Process synchronization refers to the coordination of
activities or ordering of operations among multiple
concurrent processes or threads to ensure correct and
predictable behavior. Synchronization mechanisms are
used to prevent race conditions, deadlock, and other
concurrency-related issues.
Shared =y Shared =x
Y— X++
Sleep Sleep
Abort(return) Abort(return)
Process Types
In the context of process synchronization,
processes can be categorized into various
types based on their interaction and
synchronization requirements. Here are
some common types of processes:
□ Independent Processes:
□ Cooperating Processes:
□ Producer-Consumer Processes:
□ Readers-Writers Processes:
□ Client-Server Processes:
□ Real-Time Processes:
Cooperating Processes:
Cooperative processes can share various resources such as variables, memory, code, and other system
resources in a coordinated manner. Let's discuss how each of these resources can be shared among
cooperative processes:
□ Variables Cooperative processes
□ Memory Cooperative processes
□ Code Cooperative processes
□ Resources Cooperative processes
Race condition
A race condition is a situation that occurs in a concurrent system when the outcome of the system depends on
the timing or interleaving of multiple threads or processes. Race conditions typically occur when multiple
threads or processes access shared resources concurrently and at least one of them performs a write
operation. Without proper synchronization mechanisms in place, the order of execution of these
threads/processes becomes unpredictable, leading to unexpected and incorrect behavior. Common
manifestations of race conditions include:
□ Lost Updates
□ Inconsistent Reads
□ Deadlocks
□ Livelocks
 Peterson’s Solution algorithm
Peterson's Solution is a classic algorithm for solving the critical section problem, which ensures mutual exclusion
between two processes without requiring hardware support for synchronization. The algorithm was proposed
by Gary L. Peterson in 1981. Here's a simplified version of Peterson's Solution for two processes:
In this algorithm:
□ Each process has its flag indicating its intent to enter the critical section.
□ The `turn` variable indicates whose turn it is to enter the critical section. If `turn` is 0, it's Process P0's turn;
if `turn` is 1, it's Process P1's turn.
□ When a process wants to enter the critical section, it sets its flag to true and gives priority to the other
process by setting `turn` accordingly.
□ Processes busy wait until it's their turn to enter the critical section. They spin in a loop until the other
process has finished its critical section and set its `flag` to false or changed `turn` to indicate that it's now
their turn.
□ Once a process exits the critical section, it sets its flag to false.
Shared variables:
int turn; // Indicates whose turn it is to enter the critical section
bool flag[2]; // Indicates whether a process wants to enter the critical section

Process P0:
flag[0] = true; // P0 wants to enter the critical section
turn = 1; // P0 gives priority to P1
while (flag[1] && turn == 1) {} // Busy waiting until it's P0's turn
// Critical section
flag[0] = false; // P0 exits the critical section
// Remainder section

Process P1:
flag[1] = true; // P1 wants to enter the critical section
turn = 0; // P1 gives priority to P0
while (flag[0] && turn == 0) {} // Busy waiting until it's P1's turn
 Lecture
Lecture##46
System APIs for concurrency control
CS-482
System APIs for concurrency control
□ System APIs typically refer to platform-specific
mechanisms provided by the operating system
for managing concurrency, such as
□ Thread creation,
□ Synchronization primitives (e.g., mutexes,
semaphores), and
□ Inter-process communication facilities.
Threads
A thread is a basic unit of execution within a process, representing a single sequence of instructions that can
be independently scheduled and executed by the operating system's scheduler.

#include <iostream>
#include <omp.h>

int main() {
// OpenMP directive to create a parallel region with 4 threads
#pragma omp parallel num_threads(4)
{
// Get the unique identifier of the current thread
int thread_id = omp_get_thread_num();

// Print the thread ID and a message

std::cout << "Hello from thread " << thread_id << std::endl;
}

return 0;
}
What is mutex?
□ A mutex, short for "mutual exclusion," is a
synchronization primitive used to control access to
shared resources in concurrent programming. It ensures
that only one thread can access a shared resource at a
time, preventing data races and ensuring data integrity.

□ Mutexes are essential for preventing race conditions and

ensuring thread-safe access to shared resources in
multi-threaded programs.
Here's a simple example demonstrating the use of `mutex`:
#include <iostream>
#include <omp.h>

int main() {
int shared_variable = 0;
omp_lock_t lock;
omp_init_lock(&lock);

#pragma omp parallel num_threads(4)

{
int thread_id = omp_get_thread_num();

// Each thread tries to increment the shared variable

// while ensuring mutual exclusion using a lock
omp_set_lock(&lock);
shared_variable++;
std::cout << "Thread " << thread_id << " incremented shared_variable to: " << shared_variable << std::endl;
omp_unset_lock(&lock);
}

omp_destroy_lock(&lock);

std::cout << "Final value of shared_variable: " << shared_variable << std::endl;

return 0;
}
Semaphore
A semaphore is a synchronization primitive used in concurrent programming to control access to a shared resource by multiple threads or
processes. Semaphores maintain a count or value, which can be incremented or decremented by threads. Depending on the value of the
semaphore, threads may either be allowed to proceed (if the count is positive) or be blocked until the count becomes positive.
Here's a conceptual overview of how semaphores work:
□ Initialization: A semaphore is initialized with an integer value, often referred to as the semaphore's "count" or "resource count."
□ Acquiring (Wait): When a thread wants to access the shared resource, it attempts to acquire the semaphore. If the semaphore's count is
greater than zero, indicating that resources are available, the thread decrements the count and continues execution. If the count is zero,
the thread is blocked until the count becomes positive.
□ Releasing (Signal): When a thread finishes using the shared resource, it releases the semaphore by incrementing its count. This allows
other threads waiting on the semaphore to proceed if resources become available.
There are two main types of semaphores:
□ Binary Semaphore: Also known as mutexes, binary semaphores have a count of either 0 or 1. They are typically used to control access
to a single resource, ensuring that only one thread can access it at a time.
□ Counting Semaphore: Counting semaphores can have a count greater than 1, allowing multiple threads to access a finite pool of
resources concurrently. They are useful for scenarios where multiple instances of a resource can be accessed simultaneously, up to a
certain limit.
Binary Semaphore
#include <iostream>
#include <omp.h>
int main() {
int shared_resource = 0;
omp_lock_t semaphore;
omp_init_lock(&semaphore);
#pragma omp parallel num_threads(2)
{
int thread_id = omp_get_thread_num();
if (thread_id == 0) { // Thread 0
omp_set_lock(&semaphore); // Acquire the semaphore
std::cout << "Thread " << thread_id << " has acquired the semaphore" << std::endl;
shared_resource = 1; // Modify the shared resource
std::cout << "Thread " << thread_id << " has modified the shared resource to: " << shared_resource << std::endl;
omp_unset_lock(&semaphore); // Release the semaphore
std::cout << "Thread " << thread_id << " has released the semaphore" << std::endl;
} else { // Thread 1
omp_set_lock(&semaphore); // Acquire the semaphore
std::cout << "Thread " << thread_id << " has acquired the semaphore" << std::endl;
shared_resource = 2; // Modify the shared resource
std::cout << "Thread " << thread_id << " has modified the shared resource to: " << shared_resource << std::endl;
omp_unset_lock(&semaphore); // Release the semaphore
std::cout << "Thread " << thread_id << " has released the semaphore" << std::endl;
}
}
omp_destroy_lock(&semaphore);

return 0;
}
 Amdahl's Law
□
Example
□
Practice
□
Distributed computing

Lecture## 75
Lecture
Types of Computer System
□ Multiprocessors
□ A computer-system in which 2 or more CPUs share full-access to a
common RAM.
□ Characterized by tight coupling of CPUs
□ Multicomputers
□ An interconnected collection of nodes such that each node
generally has a CPU, RAM, a network interface and perhaps a hard
disk for paging.
□ Characterized by loose coupling of CPUs that do not share
memory.
□ All nodes are in a single room and communicate by a high-speed
dedicated network.
□ All nodes run the same OS, share a single file system and are under
a common administration.
□ A typical example of a multicomputer is 512 nodes in a single room
at a company, working on, say, pharmaceutical modeling.
Distributed systems
A collection of independent computers appearing to its users as a single
coherent system in which hardware or software components communicate
and coordinate their actions only by passing messages.
□ Each node is a complete computer with a full complement of peripherals.
□ Nodes of a distributed system may each run a different OS, each has its
own file system and be under a different administration.
□ A typical distributed system consists of thousands of machines loosely
cooperating over the internet.
□ Distributed systems are even more loosely coupled than multicomputers.
□ Loose coupling of distributed systems is both
□ A strength
□ And a weakness
□ Strength: Computers can be used for a variety of different applications
□ Weakness: Programming these applications is difficult due to lack of any
common underlying model.
Significant consequences for definition of
distributed systems
□ Concurrency
□ Different computers in a network can concurrently execute programs sharing
resources such as web pages or files when necessary.
□ Coordination of concurrently executing programs is an important and recurring
topic.
□ No global clock
□ Computers in a network can’t synchronize their clocks accurately.
□ Programs coordinate actions by exchanging messages.
□ Independent Failures
□ Distributed systems can fail in new ways:-
□ Faults in network result in isolation of computers but the later don’t stop working.
◻ Programs may not be able to detect whether network failed or becomes unusually slow.
□ Failures of a computer or unexpected termination of a program (a crash) is not
immediately made known to other components.
◻ Each component can fail independently.
□ Motivation
□ Motivation for constructing and using distributed systems stems from desire to
share resources.
Two types of distribution systems
Two opposing extreme positions provide a pair of models
□ The first has a strong assumption of time and
□ The second makes no assumptions about time.
Synchronous distributed systems:
□ Synchronous distributed systems: Hadzilacos and Toueg define
such systems as one in which following bounds are defined:-
□ The time to execute each step of a process has known lower and
upper bounds.
□ Each message transmitted over a channel is received within a known
bounded time.
□ Each process has a local clock whose drift rate from real time has a
known bound.
□ Advantage: It is possible to use timeouts to detect the failure
of a process.
□ Synchronous distributed systems can be built.
□ provided that processes’ resource requirements are known
□ so that sufficient processor cycles and network capacity can be
guaranteed, and
□ clocks provided with bounded drift rates.
Synchronous distributed computing
#include <iostream>
#include <vector>
#include <omp.h>
#include <chrono>
#include <thread>

using namespace std;

int main() {
// Define the size of the data array
const int size = 10;

// Initialize data array

vector<int> data(size);
for (int i = 0; i < size; ++i) {
data[i] = i;
}

// Define upper and lower bounds for execution time and message transmission delay
const int lower_bound = 100; // in milliseconds
const int upper_bound = 500; // in milliseconds

// Parallel computation using OpenMP with bounded execution time

#pragma omp parallel
{
// Get the thread ID
int thread_id = omp_get_thread_num();

// Get the total number of threads

int num_threads = omp_get_num_threads();

// Simulate bounded execution time for each thread

An asynchronous distributed system
□ An asynchronous distributed system (such as Internet) is one in which no
bounds exist on Process execution speeds, Message transmission delays
and Clock drift rates.
□ In case of Internet, there is no intrinsic bound on server or network load and
therefore on how long it takes, for example, to transfer a file using FTP.
□ Sometimes an email message can take days to arrive.
□ Some design problems can be solved even with these assumptions.
□ e.g., although Web cannot always provide a response within a reasonable time
limit, browsers designed to allow users to do other things while they are waiting.
□ Actual distributed systems are very often asynchronous because of the
need for processes to share the processors and for communication
channels to share the network.
□ Many design problems cannot be solved for an asynchronous system.
□ The need for each element of a multimedia data stream to be delivered before a
deadline is such a problem.
□ For problems such as these, a synchronous model is required.
Example
#include <iostream>
#include <vector>
#include <omp.h>

using namespace std;

int main() {
// Define the size of the data array
const int size = 10;

// Initialize data array

vector<int> data(size);
for (int i = 0; i < size; ++i) {
data[i] = i;
}

// Parallel computation using OpenMP

#pragma omp parallel
{
// Get the thread ID
int thread_id = omp_get_thread_num();

// Get the total number of threads

int num_threads = omp_get_num_threads();
Difference
□ The difference between the asynchronous and
synchronous versions lies in how execution time is
managed. In the asynchronous version, there were no
constraints on execution time, whereas in the
synchronous version, we introduced constraints by
simulating bounded execution times for each thread. This
simulates the behavior of processes in synchronous
distributed systems, where the time to execute each step
of a process has known lower and upper bounds.
ARCHITECTURAL STYLES
□ In a DS, processes interact with each other and take on
given roles.
□ We examine two architectural styles stemming from the
role of individual processes:-
1. Client-server and
2. Peer-to-peer
Client-Server
□ In Fig 1 the client processes interact with server processes in
potentially separate host computers in order to access the
shared resources they manage.
□ Servers may in turn be clients of other servers.
□ For example, a web server is often a client of a local file server that
manages the files in which the web pages are stored.
□ Web servers and most other Internet services are clients of the DNS
service, which translates Internet domain names to network addresses.
□ Fig 1: Clients invoke individual servers
Client-Server
□ Another web-example: Search engines – enable users to look
up summaries of information on web pages throughout the
Internet.
□ Summaries are made by programs called web crawlers, which run in
the background at a search engine site using HTTP requests to
access web servers throughout Internet.
□ Thus a search engine is both a server and a client:
□ it responds to queries from browser clients and it runs web
crawlers that act as clients of other web servers.
□ The server tasks (responding to user queries) and crawler tasks
(requesting other web servers) are entirely independent;
□ There is little need to synchronize them.
□ In fact, a typical search engine would normally include many
concurrent threads of execution, some serving its clients and others
running web crawlers.
Peer-to-peer
□ All of the processes involved in a task play similar roles,
□ interacting cooperatively as peers
□ without any distinction between client & server processes.
□ In practical terms, all participating processes run the same program and
offer the same set of interfaces to each other.
□ Problem with the client-server model
□ the centralization of service provision and management does not scale well
□ beyond the capacity of the computer that hosts the service and the bandwidth of
its network connections.
□ The aim of the P2P architecture is to exploit the resources (both
data and hardware) in a large number of computers for the fulfillment
of a given task.
□ The useful consequence: the resources available to run the service grow
with the number of computers.
□ A more recent and widely used instance is the BitTorrent file-sharing
system.
Architecture
□ A large number of data objects are shared,
□ an individual computer holds only a small part of
the application database,
□ and the storage, processing and communication
loads distributed across many computers &
network links.
□ Each object replicated in several computers
□ to further distribute the load and to
□ provide resilience in the event of disconnection
of individual computers.
□ The above two characteristics rendered this
architecture substantially more complex than
the client-server architecture.
 Lecture #6
Lecture 2
Distributed Systems: Time And Global States
CS-482
Time And Global States
▪ Time is an important issue but why?
o Computers around the world required
timaersetamp e-commerce transactions consistteontly.
o Time is also an important theoretical construct in
understanding how distributed executions unfold.
▪ Problem with physical time in distributed systems.
oEach computer’s physical clock typically deviates, and
cannot be synchronized perfectly.
▪ This part of the course covers
o algorithms for synchronizing physical
clocks
approximately and then
o explain logical clocks, including vector clocks, which
are a tool for ordering events in distributed systems.
 Computers physical clocks
Electronic devices that count oscillations occurring in a crystal
at a definite frequency, and typically divide this count and store
the result in a counter register.
▪ Clock devices can be programmed to generate interrupts at
regular intervals so that, for instance, timeslicing can be
implemented.
▪ The operating system reads the node i hardware clock
value, Hi(t) ,
o scales it and adds an offset so as to produce a software clock
Ci(t) = αHi(t) + β
o that approximately measures real, physical time t for node i.
▪ For example, C i (t) could be the 64-bit value of the number of
nanoseconds that have elapsed at time t since a convenient
reference time.
▪ dIn
iffgeernferoraml, t.he clock is not completely accurate, so Ci(t) will
 Computers physical clocks

Fig 1: Skew between computer clocks in a distributed system

▪ Clock Skew The instantaneous difference between the readings
of any two clocks.
▪ Also, the crystal-based clocks used in computers are subject to
clock drift, which means that they count time at different rates,
and so diverge.
o The underlying crystal oscillators are subject to physical
variations (their kind or the way they are cut), so their
frequencies of oscillation differ.
o Even the same clock’s frequency varies with temperature.
▪ Designs exist that attempt to compensate for this variation,
but they cannot eliminate it.
 Computers physical clocks
▪ The extremely small difference in the oscillation period
o accumulated over many oscillations
o leads to an observable difference in the counters
registered by two clocks.
▪ A clock’s drift rate is the change in the offset (difference
in reading) between the clock and anominal perfect
reference clock per unit of time measured by the reference
clock.
▪ For ordinary clocks based on a quartz crystal this is about
10–6 seconds/second,
o giving a difference of 1 second every 1,000,000 seconds,
or 11.6 days.
▪ Drift rate of high-precision quartz clocks is about 10–7 or
10–8.
International Atomic Time
▪ Computer clocks can be synchronized to external sources
of highly accurate time.
▪ The most accurate physical clocks use atomic oscillators,
whose drift rate is about one part in 10 13.
▪ International Atomic Time TAI (from the French
equivalent) : the standard for elapsed real time which is
the output of atomic clocks.
▪ Since 1967, the standard second has been defined as
9,192,631,770 periods of transition between the two
hyperfine levels of the ground state of Caesium-133
(Cs133).
▪ Seconds and years and other time units are rooted in
astronomical time
o originally defined in terms of the rotation of the Earth
about its axis and its revolution around the Sun.
 Coordinated Universal Time
▪ However, the period of Earth’s rotation about its axis gradually
getting longer, primarily because of tidal friction, atmospheric drag
and other reasons.
▪ So astronomical time and atomic time have a tendency to get out
of step.
▪ Coordinated Universal Time – UTC (from the French equivalent) –
an international standard for timekeeping.
▪ Based on atomic time, but a so-called leap second inserted – or,
more rarely, deleted – occasionally to keep it in step with
astronomical time.

▪ UTC signals are synchronized and broadcast regularly from land-

based radio stations (0.1–10 msecs accuracy) and satellites (1 µsecs
accuracy) covering many parts of the world.
▪ Computers with receivers attached can synchronize their clocks
with these timing signals.
Synchronizing physical clocks
□ External synchronization
o Synchronizing the nodes’ clocks, Ci , with an
authoritative, external source of time.
o Allows knowing the time of day, events occur, at
the processes in our distributed systemP – e.g., for
accountancy purposes.
□ Internal synchronization
o Synchronizing clocks Ci with one another to a
known degree of accuracy,
o then we can measure the interval between two events
occurring at different computers by appealing to their
local clocks.
 Synchronizing physical clocks
▪ Defining these two modes of synchronization more
closely, over an interval of real timeI:-
I. External synchronization: For a synchronization
bound D > 0 , and for a source S of UTC time, |S(t) –
Ci(t) | < D, for i = 1, 2,…N and for all real timest in I.
o Another way of saying: the clocksCi are accurate to
within the bound D.
II. Internal synchronization: For a synchronization
bound D > 0 , |Ci(t) – Cj(t)| < D for i, j = 1, 2,…N ,
and for all real timest in I.
o Another way of saying: the clocksCi agree within the
bound D.
Synchronizing physical clocks
▪ Clocks that are internally synchronized are notnecessarily
externally synchronized, since they may drift collectively
from an external source of time.
▪ However, if the system P externally synchronized with a
bound D, then the same system is internally synchronized
with a bound of 2D.
Correctness of physical clocks
▪ Various notions of correctness for clocks
have been suggested.
▪ A hardware clock H is said to be correct if its drift rate falls
within a known bound ρ > 0 (e.g. 10–6 seconds/second).
▪ This means that the error in measuring the interval between
real times t and t’ (t’ > t) is bounded:
(1 – ρ)(t’ – t) ≤ H(t’) – H(t) ≤ (1 + ρ)(t’ – t)

▪ This condition forbids jumps in the value of hardware clocks

(during normal operation).
▪ Sometimes we also require our software clocks to obey this
condition but a weaker condition of monotonicity may suffice.
 Synchronizing physical clockss
• Monotonicity is the condition that a clock C only ever
advances:
t’ > t ⇒ C(t’) > C(t)
▪ For example, in the UNIX make facility ,
o the modification dates of each corresponding pair of source
and object files are compared to determine the condition for
recompiling.
❖ If a computer whose clock was running fast set its clock
back after compiling a source file but before the file was
changed,
❖ the source file might appear to have been modified prior to
the compilation.
▪ Erroneously, make will not recompile the source file.
 Synchronizing physical clocks
▪ A clock’s crash failure occurs when it stops ticking
altogether; any other clock failure is an arbitrary failure.
▪ A historical example of an arbitrary failure
o a clock with the ‘Y2K bug’, which broke the
monotonicity condition by registering the date after 31
December 1999 as 1 January 1900 instead of 2000;
o 31-12-99
o another example is a clock whose batteries are very low
and whose drift rate suddenly becomes very large.
• Note that clocks do not have to be accurate to be correct.
• Since the goal may be internal rather than external
synchronization.
Synchronization in a synchronous system
▪ The simplest possible case: internal
synchronization between two processes in a synchronous
distributed system.
▪ In a synchronous system, bounds are known for
o the drift rate of clocks,
o the message transmission delay, and
o the time required to execute each step of a process.
▪ One process P1 sends the time t on its local clock to P2 in a
message m.
▪ In principle, the receiving process P2 could set its clock to the
time t + Ttrans , where Ttrans is transmission time for m.
▪ The two clocks would then agree.
▪ Unfortunately, Ttrans is subject to variation and unknown.

▪ Other messages compete with m for the network resources.

Synchronization in a synchronous system
▪ Nonetheless, there is always a minimum transmission time,
min, if no other processes executed and no network traffic.
▪ In a synchronous system, there is also an upper bound max
on the time taken to transmit any message.
▪ The uncertainty u in message transmission time is
u = (max – min)
▪ If P2 sets its clock to the halfway point, t + (max + min)/2 ,
then the skew is at most u/2 .
▪ In general, for a synchronous system, the optimum bound on
clock skew when synchronizing N clocks is u(1 – 1/N).
▪ Most distributed systems (e.g. Internet) are asynchronous: no
upper bound max on message sending time.
▪ For such system, Ttrans = min + x , where x ≥ 0.
Cristian’s method for synchronizing clocks
▪ Cristian suggested the use of a time server, connected to a
device that receives signals from a source of UTC, to synchronize
computers externally.
▪ Upon request, the server process S supplies the time according
to its clock, as shown in Fig 2.

Fig 2: Clock synchronization using a time server

▪ t is inserted in m t at the last possible point before

transmission from S’s computer.

▪ The algorithm achieves synchronization only
o if the observed round-trip times between client and server
are sufficiently short
Cristian’s method for synchronizing clocks

o compared with the required accuracy.

 Cristian’s method for synchronizing clocks
▪ Process p records the total round-trip time Tround taken to
send the request mr and receive the reply mt .

▪ It can measure this time with reasonable accuracy if its rate of

clock drift is small.
▪ For example, the round-trip time should be on the order of 1–10
milliseconds on a LAN, over which time a clock with a drift rate
of 10–6 seconds/second varies by at most 10–5 milliseconds.
1 sec → 10–6 seconds
10 millisecs → 10–8 seconds
▪ A simple estimate of the time to which p should set its clock is t
+ Tround/2 , which assumes that the elapsed time is split equally
before and after S placed t in mt .
▪ This is a reasonably accurate assumption, unless the two
messages are transmitted over different networks.
 Cristian’s method for synchronizing clocks
▪ If the value of the minimum transmission time min is
known or can be conservatively estimated, then we can
determine the accuracy of this result as follows.
▪ The earliest point at which S could have placed the time in
mt was min after p dispatched mr .
▪ The latest point at which it could have done this was min
before mt arrived at p.

▪ The time by S’s clock when the reply message arrives is

therefore in the range [t + min, t + Tround – min].
▪ The width of this range is T round – 2min , so the accuracy is
±(Tround / 2 – min).
 Discussion of Cristian’s algorithm
▪ Cristian’s method suffers from the problem of single point of
failure.
▪ Cristian suggested, for this reason, that time should be
provided by a group of synchronized time servers, each
with a receiver for UTC time signals.
▪ For example, a client could multicast its request to all
servers and use only the first reply obtained.
▪ Note that a faulty time server or an imposter time server
could wreak havoc in a computer system.
▪ The problem of dealing with faulty clocks is partially
addressed by the Berkeley algorithm.
▪ The problem of malicious interference with time
synchronization can be dealt with by authentication
techniques.
21
 The Berkeley algorithm
▪ Gusella and Zatti [1989] developed an algorithm for internal
synchronization for collections of computers running Berkeley
UNIX.
▪ A coordinator computer chosen to act as the master.
▪ Unlike in Cristian’s protocol, this computer periodically polls
the other computers whose clocks are to be synchronized,
called slaves.
▪ The slaves send back their clock values to it.
▪ The master estimates their local clock times by
o observing the round-trip times (similarly to Cristian’s
technique), and
o it averages the values obtained (including its own clock’s
reading).
▪ The balance of probabilities is that this average cancels out
the individual clocks’ tendencies to run fast or slow.
 The Berkeley algorithm
▪ That is, a subset is chosen of clocks that do not differ from
one another by more than a specified amount, and the
average is taken of readings from only these clocks.
▪ Gusella and Zatti describe an experiment involving 15
computers whose clocks were synchronized to within
about 20–25 milliseconds using their protocol.
▪ The local clocks’ drift rates were measured to be less than
2 x 10–5, and the maximum round-trip time was taken to
be 10 milliseconds.
▪ Should the master fail, then another can be elected to
take over and function exactly as its predecessor.
 The Network Time Protocol
▪ Cristian’s method and Berkeley algorithm are intended
primarily for use within intranets.
▪ The Network Time Protocol (NTP) [Mills 1995] distributes
time information over the Internet.
□ NTP’s Chief Design Aims and Features
1. To enable clients across the Internet to synchronize accurately
to
UTC:
o Although large and variable message delays are
encountered in Internet communication,
o NTP employs statistical techniques for the filtering of
timing data and it discriminates between the quality of
timing data from different servers.
25
 The Network Time Protocol
2. To provide a reliable service that can survive lengthy losses of
connectivity:
o There are redundant servers and redundant paths
between the servers.
3. To enable clients to resynchronize sufficiently frequently to offset
the rates of drift found in most computers:
o The service is designed to scale to large numbers of clients
and servers.
4. To provide protection against interference with the time service,
whether malicious or accidental:
o The time service uses authentication techniques to check
that timing data originate from the claimed trusted sources.

26
 The Network Time Protocol
▪ The NTP service is provided by a network of servers located
across the Internet.
▪ Primary servers are connected directly to a time source such as
a radio clock receiving UTC;
▪ Secondary servers are synchronized, ultimately, with primary
servers.
▪ The servers are connected in a logical hierarchy called a
synchronization subnet, whose levels are called stra ta.

Fig 3: An example of synchronization subnet in an NTP implementation

▪ Primary servers occupy stratum 1: they are at the root. 27
 The Network Time Protocol
▪ The four times T i – 3 , Ti – 2 , Ti – 1 and Ti shown in Fig 4 for the

messages m and m' sent between servers A and B.

▪ Also, messages may be lost, but the three timestamps carried
by each message are nonetheless valid.
▪ For each pair of messages sent between two servers, the NTP
calculates
o an offset oi , which is an estimate of the actual offset
between the two clocks, and
o a delay di, which is the total transmission time for the two
messages.
 The Network Time Protocol
▪ If the true offset of the clock at B relative to that at A is o, and if the
actual transmission times for m and m' are t and t', respectively,
then we have:-
Ti – 2 = Ti – 3 + t + o
and
Ti = Ti – 1 + t’ – o
▪ This leads to:
di = t + t’ = Ti – Ti – 1 + Ti – 2 – Ti – 3
▪ and:

o = oi + (t’ – t) / 2 , where oi = (–Ti + Ti – 1 + Ti – 2 – Ti – 3) / 2

▪ Using the fact that t, t’ ≥ 0 , it can be shown that o – d /2 ≤ o ≤ o +
i i i
di/2.
The Network Time Protocol

▪ Thus o i is an estimate of the offset, and di is a measure of the

accuracy of this estimate.

 Lecture
Lecture##74
Time And Global States
CS-482
 A model of distributed executions
▪ A distributed application consists of a set of n asynchronous
processes p , p , …, p , …, p that communicate by message
1 2 i n
passing over the communication network.
▪ Let Cijdenote the channel from process pi to pj and let m ij
denote a message sent by pi to pj .
▪ The communication delay is finite and unpredictable.
▪ The process actions are modeled as three types of events, namely,
o internal events,
o message send events, and
o message receive events.
▪ Let e x denote the xth event at process p .
i i
▪ Subscripts and/or superscripts will be dropped when
they are irrelevant or are clear from the context.
 Causal precedence relation
▪ The causal ordering of events is based on two simple and intuitively
obvious points:-
o If two events occurred at the same process pi (i = 1, 2, …, N), then
they occurred in the order in which pi observes them i.e. the order
→.
i
o Whenever a message is sent between processes, the event of
sending the message occurs before the event of receiving the
message.
▪ Lamport called the partial ordering obtained by generalizing these
two relationships the happened-before relation.
▪ Also sometimes known as the relation of causal ordering or
potential causal ordering.
 Causal precedence relation
▪ The happened-before relation, denoted by →, defined as follows:
1. HB1: If ∃process p i : e → i e', then e → e’.

2. HB2: For any message m, send(m) → receive(m).

3. HB3: If e, e’ and e’’ are events such that e → e’ & e’ → e’’, then e → e’’
i.e. e’’ is transitively dependent on e.
▪ The relation → is illustrated for the case of three processes,
p1, p2 and p3 , in Fig 1.

Fig 1: Events occurring at three

processes

▪ Not all events (Fig 1) are related by the relation →.

▪ For example, a ↛ e and e ↛ a, since there is no chain of messages
intervening between them.
Causal precedence relation
▪ Event a doesn’t causally effect e and event e doesn’t causally effect
a.
▪ We say that events a and e are concurrent i.e. a ║ e .
▪ Another point to note: if the happened-before relation holds
between two events, then the first might or might not actually have
caused the second.
▪ A process might, for example, receive a message and subsequently
issue another message, but one that it issues every five minutes
anyway and that bears no specific relation to the firstmessage.

▪ No actual causality has been involved, but the relation →

would order these events.
 Causal precedence
▪ In Fig 2, e 3
↛ e 3
and e 4 relation
↛ e 1.
1 3 2 3
▪ Also, e ∥ e and e ∥e .
3 3 4 1

1 3 2 3

Fig 2

▪ Note that relation → denotes flow of information in a distributed computation

and ei → ej dictates that all the information available at ei is potentially
accessible at ej .
▪ E.g., in Fig 2, event e 26 has the knowledge of all other events.
▪ Note the following two rules:
o for any two events ei and ej , ei ↛ ej ⇏ ej ↛ ei
Causal precedence relation
o for any two events ei and ej , ei → ej ⟹ ej ↛ ei
 Logical vs. physical concurrency
▪ Physical concurrency has a connotation that the events occur at the
same instant in physical time.
▪ In a distributed computation, two events are logically concurrent iff
o they do not causally affect each other
o even though they don’t occur at the same physical time.

▪ For example, in Fig 2, events in the set {e13, e 4, e 3} are 2 3

logically concurrent, but they occurred at different instants in
physical time.
Logical time and logical clocks
Logical clocks

▪ Lamport [1978] invented a simple mechanism by which the

happened before ordering can be captured numerically.
▪ A Lamport logical clock is a monotonically increasing
software counter.
▪ Each process p i keeps its own logical clock, L i to apply
Lamport timestamps to events.
▪ Li(e): denotes the timestamp of event e at pi
▪ L(e): denotes the timestamp of event e at any process.
 Logical time and logical clocks
I. LC1: Li is incremented before each event is issued at process pi : Li
:= Li + 1.
II. LC2:
a) When a process p i sends a message m, it piggybacks on

m the value t = Li .
b) On receiving (m, t), a process p j computes L j := max(Lj, t) and
then applies LC1 before time-stamping the event receive(m).
▪ Although we increment clocks by 1, we could have chosen any
positive value.
▪ It can easily be shown, by induction on the length of any sequence
of events relating two events e and e’ , that e → e’
⇒ L(e) < L(e’).
 Logical time and logical clocks
▪ Note that the converse is not true.
▪ If L(e) < L(e’), then we cannot infer that e → e’ .
▪ Each of the processes p1 , p2 and p3 has its logical clock initialized to
0.

Fig 3: Lamport timestamps for the events in fig 1

▪ The clock values given are those immediately after the event
to which they are adjacent.
▪ Note that, for example, L(b) > L(e) but b ║ e .
Totally ordered logical clocks
▪ Distinct events, generated by different processes may have
numerically identical Lamport timestamps.
▪ However, we can create a total order on the set of events.
▪ If e is an event occurring at p i with local timestamp T i , and
e’ is an event occurring at pj with local timestamp T j ,
o we define the global logical timestamps for these events
to be (Ti, i) and (Tj, j) , respectively.
▪ And we define (T , i) < (T , j) if and only if either T < T , or
i j i j
Ti = T j and i < j.
▪ This ordering has no general physical significance (because process
identifiers are arbitrary), but it is sometimes useful.
▪ Lamport used it, for example, to order the entry of processes to a
critical section.
 Vector clocks
▪ Mattern [1989] and Fidge [1991] developed vector clocks to
overcome the shortcoming of Lamport’s clocks:
o the fact that from L(e) < L(e’) we cannot conclude e → e’.
▪ A vector clock for a system of N processes is an array of N
integers.
▪ Each process keeps its own vector clock, Vi , which it uses to
timestamp local events.
▪ Like Lamport timestamps, processes piggyback vector timestamps on
the messages they send to one another, and there are simple rules for
updating the clocks:
o VC1: Initially, Vi[j] = 0 , for i, j = 1, 2,… N.
o VC2: Just before pi timestamps an event, it sets Vi[i] :=Vi[i] + 1.
o VC3: pi includes the value t = Vi in every message it sends.
o VC4: When pi receives a timestamp t in a message, it sets Vi[j] :=
max(Vi[j], t[j]) , for j = 1, 2, … N .
 Vector clocks
▪ Taking the component-wise maximum of two vector timestamps in
this way is known as a merge operation.
▪ For a vector clock V , V [i] is the number of events that p has
i i i
timestamped, and Vi[j] (j ≠ i) is the number of events that have
occurred at pj that have potentially affected pi.

Fig 4: Vector timestamps for the events shown in Fig 1

Vector clocks
▪ We may compare vector timestamps as follows:-
o V = V’ iff V[j] = V’[j] for j = 1, 2,… N
o V ≤ V’ iff V[j] ≤ V’[j] for j = 1, 2,… N
o V < V’ iff V ≤ V’ ^ V ≠ V’
V = (3, 4, 5) and V’ = (3, 6, 5)
V < V’
▪ Let V(e) be the vector timestamp of event e at a process.
▪ It is straightforward to show, by induction, on the length of any
sequence of events relating two events e and e’ , that e → e’⇒V(e)
< V(e’) .
Vector clocks
▪ It can be seen from fig 4, for example, that V(a) < V(f) , which
reflects the fact that a→f.

▪ Similarly, we can tell when two events are concurrent by comparing

their timestamps.
▪ For example, that c║e can be seen from the facts that neither
V(c) ≤ V(e) nor V(e) ≤ V(c) .
Vector clocks
▪ Disadvantage of Vector timestamps: taking up an amount of
storage and message payload that is proportional to N, the number
of processes.
▪ Charron-Bost [1991] showed that, in order to tell whether or not two
events are concurrent by inspecting their timestamps, thedimension
N is unavoidable.
▪ However, techniques exist for storing and transmitting smaller
amounts of data, at the expense of the processing required to
reconstruct complete vectors.
Models of communication networks
▪ Several models of the service provided by communication
networks, namely, FIFO, non-FIFO, and causal ordering.
▪ FIFO model: each channel acts as a FIFO message queue and
thus, message ordering is preserved by a channel.
▪ Non-FIFO model: the sender process adds and the receiver
process removes messages from channel in random order.
▪ Causal ordering model: based on Lamport’s happens before
relation satisfying the following property:-
CO: For any two messages m and m kj, if send(m ) → send(m kj),
ij

then rec(m ) → rec(m ) ij

ij kj
▪ Causally ordered delivery of messages implies FIFO message
delivery.
Models of communication networks
▪ Note that CO ⊂ FIFO ⊂ Non-FIFO.
▪ Causal ordering model is useful as it considerably
simplifies the design of distributed algorithms because
it provides a built-in synchronization.
▪ For example, in replicated database systems, it is
important
□ o every process responsible for updating a replica
receives the updates in the same order to maintain
database consistency.
o Without causal ordering, each update must be checked
to ensure that database consistency is not being
violated.
o Causal ordering eliminates the need for such checks.
Global state of a distributed system
▪ The global state of a distributed system
□ o a collection of the local states of processes and the
communication channels.
▪ The state of a process at any time is defined by
□ o the contents of registers, stacks, local memory, etc.
▪ The state of a channel is given by the set of messages in
□ transit in the channel.
▪ The occurrence of events changes the states of respective
processes and channels, thus causing transitions in global
system state.
▪ For example, an internal event changes the state of the process
at which it occurs.
 Global state of a distributed system
▪ A send event (or a receive event) changes
o the state of the process that sends (or receives) the
message and
o the state of the channel on which the message is sent
(or received).
▪ Let LSix denote the state of process pi after the occurrence of
event e x and before the event e x+1.
▪ LSi 0denoteis thi e initial state of process pi.
▪ LS ix is a result of the execution of all the events on process
pi till e xi .
▪ The state of a channel C denoted as SCijx,y denotes all
ij x
messages that pi sent up to event e anid wj hich process p
had not received until event e y. j
Global state of a distributed system
▪ The essential problem in recording the global state is the
□ absence of global time.
▪ If all processes had perfectly synchronized clocks then we could
record the global state of the system at any particular instant.
▪ Since we cannot achieve perfect clock synchronization, this
method is not available.
▪ Design of efficient methods for recording the global state of a
distributed system is an important problem.
▪ From the global state it is possible to
o Detect deadlock between a set of process
o Determine that an object has become garbage
o Detect that a distributed algorithm has terminated
o Debug a distributed program
Global state of a distributed system
▪ Mathematically, we can take any set of states of the individual
processes to form a global state S = (s1, s2, …, sN ).
▪ But which global states are meaningful – that is, which
□ process states could have occurred at the same time?
▪ For a global snapshot to be meaningful, it should be
consistent.
▪ Consistent global states
o Every message recorded as received is also recorded as
sent.
o Basic idea is that an effect should not be present without
its cause.
▪ Inconsistent global states
o A message that is recorded as received is not recorded as
sent.
 Global state of a distributed system
▪ In Fig 5, a global state GS 1 = {LS 1, L1S 3,2LS 33 , LS4 2}
o inconsistent because p2 has recorded receipt of message m12,
however, p1 has not recorded its send.

Fig 5: The space–time

diagram of a distributed
execution.
▪ GS2 ={LS 21, LS24, LS
3 , 4LS }
4 2

o consistent; all channels empty except C21 (contains m21).

▪ A global state GS = {∪i LSixi, ∪ j,k SC yj,zk } ijsk transitless if all the

channels are recorded as empty.

▪ A global state is strongly consistent iff it is transitless as well as
consistent.
▪ Note that in Fig 5, the global state GS = {LS 2, LS 31, LS23 , LS
3
2}
4
is strongly consistent.
Cuts of a distributed computation
□ What is a cut of distributed computation?
▪ A zigzag line joining one arbitrary point on each process line
in the space–time diagram of a distributed computation.

▪ Slices the space–time diagram and the set of events in the

distributed computation into a PAST and a FUTURE.
▪ For a cut C, let PAST(C) and FUTURE(C) denote the set of events
in the PAST and FUTURE of C, respectively.
▪ Every cut corresponds to a global state.
▪ Consistent cut: A consistent global state corresponds to acut
in which every message received in the PAST of the cut was
sent in the PAST of that cut.
Cuts of a distributed computation
▪ All messages that cross the cut from the PAST to the FUTURE are in
transit in the corresponding consistent
□ global state.
▪ Inconsistent cut: message crosses the cut from the FUTURE to the
PAST.

□ Fig 6: Illustration of cuts in a distributed execution

▪ In the space–time diagram of Fig 6, C1 is an inconsistent cut
□ whereas C2 is a consistent cut.
▪ Cuts provide a powerful graphical aid in representing and reasoning
about global states of a computation.
Issues in recording a consistent global state
□ I1: How to distinguish between the messages to be recorded
in the snapshot from those not to be recorded.
▪ The answer comes from conditions C1 and C2 as follows:-
a. C1: Any message, sent by a process before recording its
snapshot, must be recorded in global snapshot.
b. C2: Any message, sent by a process after recording its
snapshot, must not be recorded in the global snapshot.
□ I2: How to determine the instant when a process takes its
snapshot.
□ The answer to this comes from condition C2 as follows:
▪ A process pj must record its snapshot before processing a
message mij that was sent by process pi after recording its
snapshot.
Chandy–Lamport Snapshot algorithm for FIFO channels
▪ The first algorithm to record the global snapshot.
▪ Uses a control message, called a marker.
▪ After a site records its snapshot, it sends a marker along all of its
outgoing channels before sending any more messages.
i

j
k

▪ Since channels are FIFO, a marker separates messages to be

included in the snapshot from those not to be included.
o This addresses issue I1.
▪ Since all messages that follow a marker on C have been sent by p
ij i
after it has taken its snapshot, pj must record its snapshot no
ateTrhth
lo isaandw
drheesnseitsriescsu
eieveI2s .a marker on C ij .
Snapshot algorithm
1. A process initiates snapshot collection by executing the
□ marker sending rule.

2. A process executes the marker receiving rule on receiving a marker.

▪ The algorithm can be initiated by any process by executing the marker sending
rule.
Snapshot algorithm
▪ The algorithm terminates after each process has received a
marker on all of its incoming channels.
▪ The recorded local snapshots can be put together to create
the global snapshot in several ways.
▪ One policy is to have each process send its local snapshot to
the initiator of the algorithm.
▪ Another policy is to have each process send the information it
records along all outgoing channels.
□ o all the processes can determine the global state.
▪ Multiple processes can initiate the algorithm concurrently.
o Each initiation needs to be distinguished by using unique
markers.
o Different initiations by a process are identified by a
sequence number.
 Properties of the recorded global state
▪ Two possible executions of the snapshot algorithm for the
money transfer example (Fig 7):
1. (Markers shown using dashed-and-dotted
arrows.)
▪ Let site S1 initiate the algorithm just after t1.
▪ S1 records its local state (account A=$550)
and sends a marker to site S2.
▪ The marker is received by site S2 after t4.
▪ Then it records its local state (account
B=$170), the state of channel C12 as $0, and
sends a marker along C21.
▪ When site S1 receives this marker, it Fig 7: Timing diagram of
records the state of C21 as $80. two possible executions of
▪ The $800 amount in the system is the banking example

conserved in the recorded global state,

A = $550, B = $170, C12 = $0, C21 = $80.
 Properties of the recorded global state
2. (Markers shown using dotted arrows)
▪ Let S1 initiate the algorithm just after t0 and
before sending the $50 for S2.
▪ S1 records its local state (account A = $600)
and sends a marker to S2.
▪ The marker is received by S2 between t2 and t3.
▪ When S2 receives the marker, it records its local
state (account B = $120), the state of C12 as $0,
and sends a marker along C21.
▪ When S1 receives this marker, it records the
state of C21 as $80.
▪ The $800 amount in the system is conserved in
the recorded global state,
Fig 7: Timing diagram of
A = $600, B = $120, C12 = $0, C21 = $80 two possible executionsof
the banking example
Complexity of Snapshot algorithm

▪ The recording part of a single instance of the

algorithm requires O(e) messages and O(d) time,
where
□ o e is the number of edges in the network and
□ o d is the diameter of the network.
Distributed computing(challenges)

Lecture # 8
Challenges facing Distributed Systems
Significant challenges are encountered in the design and use of distributed systems.
1. Heterogeneity
Heterogeneity (i.e., variety and difference) applies to all of the following:-
□ networks;
□ computer hardware;
□ operating systems;
□ programming languages;
Although Internet consists of many different sorts of network, their differences are masked as all computers attached
to them use the Internet protocols (IPs) for communication. e.g., a computer attached to an Ethernet has an
implementation of the IPs over the Ethernet, whereas a computer on a different sort of network will need an
implementation of the IPs for that network.
□ Data types such as integers may be represented in different ways on different hardware e.g., two
alternatives for the byte ordering of integers.
□ These differences must be dealt with if messages are to be exchanged between programs running on
different hardware.
□ Although the operating systems of all computers on the Internet need to include an implementation of
the Internet Protocols, they do not necessarily all provide the same API to these protocols. e.g., the calls
for exchanging messages in UNIX are different from the calls in Windows.
□ Different programming languages use different representations for characters and data structures such as
arrays and records.
□ These differences must be addressed if programs in different languages need to communicate with one
another.
Middleware
□ Heterogeneity and mobile code mobile code is one that can be
transferred from one computer to another and run at the destination –
Java applets are an example.
□ Code suitable for running on one computer is not necessarily suitable for running
on another because
□ executable programs are normally specific both to the instruction set and to the host OS.
□ e.g., executable files sent as e-mail attachments by Windows/x86 users will not
run on Macintosh computer running Mac OS X.
□ The virtual machine (VM) approach provides a way of making code executable on
a variety of host computers:
□ e.g., the Java compiler produces code for a Java VM, which executes it by interpretation.
□ The Java VM needs to be implemented once for each type of computer to
enable Java programs to run.
□ Today, the most commonly used form of mobile code is the inclusion of
Javascript programs in some web pages loaded into client browsers.
Openness
□ An open distributed system is a system that may be extended
□ by the introduction of new services and
□ the reimplementation of old ones,
□ enabling application programs to share resources.
□ For example, in an extensible system, it should be relatively easy to add parts that run on a different OS or even to
replace an entire file system.
□ It also allows two independent parties to build completely different implementations of those interfaces, leading to two
separate distributed systems that operate in exactly the same way.
□ Openness cannot be achieved unless the specification of key software interfaces of the components of a system are
published so that they are available to software developers.
□ In distributed systems, services are generally specified through interfaces in an Interface Definition Language (IDL).
□ Interface definitions written in an IDL always capture only the syntax of services.
□ i.e. they specify precisely the names of the functions together with types of parameters, return values, possible exceptions that can
be raised, and so on.
□ The hard part is specifying precisely what those services do, that is, the semantics of interfaces.
□ In practice, such specifications are always given in an informal way by means of natural language.
□ Open distributed systems can be constructed from heterogeneous hardware and software, possibly from different
vendors.
□ In the case of Web caching, for example,
□ a browser should ideally provide facilities for only storing documents, and
□ at the same time allow users to decide about
□ the size of the cache,
□ about which documents are stored and for how long,
□ whether a cached document should always be checked for consistency.
□ In practice, a user can implement his own policy in the form of a component that can be plugged into the browser.
Security
□ Security for information resources has three components:-
□ Confidentiality (protection against disclosure to unauthorized individuals),
□ Integrity (protection against alteration or corruption), and
□ Availability (protection against interference with the means to access the resources).
□ Although firewall can be used to form barrier around an intranet,
□ this does not deal with ensuring the appropriate use of resources by users within an
intranet, or in the Internet.
□ In a distributed system, clients send requests to access data managed by servers.
□ For example:
□ A doctor might request access to hospital patient data or send additions to that data.
□ In electronic commerce and banking, users send their credit card numbers across the
Internet.
□ In both examples, the challenge is to send sensitive information in a message over a network
in a secure manner.
□ Solution: the use of encryption techniques to send sensitive information in a message.
□ But security is not just a matter of concealing contents of messages
□ it also involves knowing for sure the identity of the user on whose behalf a message was
sent.
□ Solution: use of biometric techniques or verification code on the cell phone to authenticate
the user.
Scalability
□ A system is scalable if it remains effective even with significant
increase in the number of resources and users.
□ Distributed systems operate effectively and efficiently at many
different scales, ranging from a small intranet to Internet.
□ System scalability measured along at least 3 different
dimensions.
□ With respect to size, - we can easily add more users and
resources to the system.
□ A geographically scalable system - the users and resources
may lie far apart.
□ An administratively scalable system - still be easy to manage
even if it spans many independent administrative organizations.
Scalability Problems
□ Scalability problems in DSs appear as performance problems caused by limited capacity of servers and network.
□ With respect to size
□ Obvious problem with centralized services: the server can become a bottleneck as number of users and applications
grows.
□ Using only a single server is sometimes unavoidable.
□ Imagine a service for managing highly confidential information such as medical records, bank accounts and so on.
□ Copying the server to several locations to enhance performance would otherwise make the service less secure.
□ With geographical scalability
□ Earlier distributed systems were designed for LANs that are based on synchronous communication.
□ In LANs communication between two machines is generally at worst a few hundred microseconds.
□ However, in a WAN, IPC may be hundreds of millisecs, three orders of magnitude slower.
□ Building interactive applications using synchronous communication in WAN systems requires a great deal of care.
□ Communication in wide-area networks is inherently unreliable, and virtually always point-to-point.
□ In contrast, LANs generally provide highly reliable communication facilities based on broadcasting, making it much easier
to develop distributed systems.
□ For example, consider the problem of locating a service.
□ Only those machines that have that service respond, each providing its network address in the reply message.
□ Such a location scheme is unthinkable in a WAN system.
□ Special location services needed which may scale worldwide.
□ In a system with many centralized components, geographical scalability (like size one) is limited due to performance and
reliability problems from wide-area communication.
Problems with administrative scalability
□ Conflicting policies with respect to resource usage, management,
and security.
□ If a distributed system expands into another domain, two types of
security measures need to be taken.
□ First of all, the distributed system has to protect itself against
malicious attacks from the new domain.
□ e.g., users from the new domain may have only read access to the
file system in its original domain.
□ Second, the new domain has to protect itself against malicious
attacks from the distributed system.
□ A typical example is that of downloading programs such as applets
in Web browsers.
□ Administrative scalability seems to be the most difficult one, partly
also because we need to solve nontechnical problems (e.g., politics
of organizations and human collaboration).
Scaling Techniques
Basically three techniques for scaling: hiding communication latencies, distribution, and replication.
□ Hiding communication latencies: important to achieve geographical scalability.
□ The basic idea: when a service has been requested at a remote machine, an alternative to waiting for a reply is to do other useful
work at the requester's side.
□ constructing the requesting application for using only asynchronous communication.
□ When a reply comes in, the application interrupted and a special handler called to complete previously-issued request.
□ Alternatively, a new thread of control can be started to perform the request.
□ Distribution: involves taking a component, splitting it into smaller parts, and subsequently spreading those parts across the
system.
□ An excellent example of distribution is the Internet DNS.
□ DNS – the Domain Name System comprises of a table with the correspondence between the domain names of computers (e.g.
www.amazon.com) and their Internet addresses.
□ Algorithms that use hierarchic structures scale better than those that use linear structures.
□ The time taken to access hierarchically structured data is O(log n), where n is the size of the set of data.
□ The DNS name space is hierarchically organized into a tree of domains divided into nonoverlapping zones (Fig 1).
□ The names in each zone are handled by a single name server.
□ One can think of each path name, being the name of a host in the Internet, and thus associated with a network address of that host.
□ Basically, resolving a name means returning the network address of the associated host.
□ Consider, for example, the name nl. vu.cs.flits.
□ To resolve this name, it is first passed to the server of zone z1 (Fig. 1) which returns the address of the server for zone
z2, to which the rest of name, vu.cs.flits, can be handed.
□ The server for z2 will return the address of the server for zone z3, which is capable of handling the last part of the name
and will return the address of the associated host.
Replication
□ Replication of components across a distributed system.
□ It not only increases availability, but also helps to balance the load between components leading to better
performance.
□ Also, in geographically-dispersed systems, having a copy nearby can hide much of the communication
latency problems mentioned before.
□ In general, for a system with n users to be scalable, the quantity of physical resources required to
support them should be at most O(n).
□ For example, if a single file server can support 20 users, then two such servers should be able to support
40 users.
□ One serious drawback to replication may badly affect scalability.
□ modifying one copy makes that copy different from others.

□ Fig 1: An example of dividing the DNS name space into zones.

Failure handling
□ Failure handling
□ The handling of failures in DSs is particularly difficult.
□ Failures are partial i.e. some components fail while others continue to function.
□ Detecting failures: Some failures can be detected.
□ E.g., checksums can be used to detect corrupted data in a message or a file.
□ It is difficult or even impossible to detect some other failures, such as a remote crashed server in the
Internet.
□ The challenge is to manage in the presence of failures that cannot be detected but may be
suspected.
□ Masking failures: Some failures that have been detected can be hidden or made less severe.
□ Two examples of hiding failures:
□ Messages can be retransmitted when they fail to arrive.
□ File data can be written to a pair of disks so that if one is corrupted, the other may still be correct.
□ Techniques for hiding failures not guaranteed to work in worst cases; e.g., the data on the second
disk may be corrupted too.
□ Tolerating failures: When a web browser cannot contact a web server – it informs user
about the problem, leaving them free to try again later.
□ Recovery from failures: Recovery involves the design of software so that the state of permanent data
can be recovered or ‘rolled back’ after a server has crashed.
□ Redundancy: Services can be made to tolerate failures by the use of redundant components.
Consider the following examples:
□ At least 2 different routes between any two routers in the
Internet.
□ In DNS, every name table replicated in at least 2 different
servers.
□ A database may be replicated in several servers;
□ when a fault detected in one server, clients redirected to
remaining servers.
□ Availability of a system: measure of the proportion of time
that a system is available for use.
□ Distributed systems provide a high degree of availability in the
face of hardware faults.
□ A server process can be started on another computer.
□ When one of the components fails, only the work that was
using the failed component is affected.
Concurrency
□ Suppose that each resource is encapsulated as an object in a
server process and that client invocations are executed in
concurrent threads.
□ For example a data structure that records bids for an auction
may be accessed very frequently when it gets close to the
deadline time.
□ If two concurrent bids at an auction are Smith: $122 and
Jones: $111, and
□ the corresponding operations are interleaved without any
control,
□ then they might get stored as Smith: $111 and Jones: $122.
□ For the object to be safe, its operations must be synchronized
so that data remains consistent.
□ This can be achieved by mutual exclusion techniques such as
semaphores, used in most operating systems.
Transparency
□ The concealment from the user and application programmer of separation
of components in a distributed system, so that the system is perceived as a
whole rather than as a collection of independent components.
□ An important goal of a distributed system.
□ Types of Transparency
□ The ANSA Reference Manual [ANSA 1989] and ISO’s Reference Model
for Open Distributed Processing (RM-ODP) [ISO 1992] identify eight
forms of transparency.
1. Access transparency: Deals with hiding differences in data representation and
the way resources can be accessed by users.
□ It enables local and remote resources to be accessed using identical operations.
□ Differences in file naming conventions in different OSs, as well as how files can be
manipulated, should all be hidden from users and applications.
□ As another illustration, consider a GUI with folders, which is the same whether the files
inside the folder are local or remote.
□ Example of lack of access transparency: A DS allows access to files on a remote computer
using ftp program only.
Transparency
2.Location transparency enables resources to be accessed without knowledge of their physical or network
location.
□ Can be achieved by assigning logical names to resources.
□ Web resource names or URLs are location-transparent because the part of URL that identifies a web server
domain name refers to a computer name in a domain, rather than to an Internet address.
□ e.g. the URL http://www.prenhall.com/index.html which gives no clue about location of Prentice Hall's main Web
server.
3.Concurrency transparency enables several processes to operate concurrently using shared resources
without interference between them.
□ e.g., two independent users may be accessing the same tables in a shared database.
□ An important issue is that concurrent access to a shared resource leaves that resource in a consistent state.
□ Consistency can be achieved through the use of locking mechanisms and transactions but they are quite difficult to
implement in distributed systems.
4. Replication transparency enables multiple instances of resources to be used to increase reliability and
performance without knowledge of the replicas by users or application programmers.
□ To hide replication from users, it is necessary that all replicas have the same name.
□ Consequently, a system that supports replication transparency should generally support location transparency as
well,
□ because it would otherwise be impossible to refer to replicas at different locations.
□ In general, identifiers such as URLs that include the domain names of computers prevent replication transparency.
□ Although the domain name may refer to several computers, DNS picks just one of them at look up.
□ Since a replication scheme generally needs to be able to access all of the participating computers, it would need to
access each of the DNS entries by name.
Transparency
5. Failure transparency enables concealment of faults, allowing users and application programs to complete their tasks despite
failure of hardware or software components.
□ The main difficulty in masking failures lies in the inability to distinguish between a dead and a painfully slow resource.
□ e.g., when contacting a busy Web server, a browser will eventually time out and report that Web page is unavailable.
□ Failure transparency can also be illustrated in the context of electronic mail, which is eventually delivered, even when servers or
communication links fail.
□ The faults are masked by attempting to retransmit messages until they are successfully delivered, even if it takes several days.
6. Migration transparency is provided by allowing the resources to be moved without affecting how those resources (e.g. file,
web page) can be accessed.
□ A stronger form of migration transparency is mobility (or relocation) transparency.
□ The two phone users making the call are unaware of the mobility of the phones between cells.
□ However, URLs are not mobility-transparent, because
□ someone’s personal web page cannot move to their new place of work in a different domain –
□ all of the links in other pages will still point to the original page.
□ Performance transparency allows the system to be reconfigured to improve performance as loads vary.
7. Scaling transparency allows the system and applications to expand in scale without change to the system structure or the
application algorithms. e.g. video server.
□ Two most important transparencies: access & location transparency;
□ their presence most strongly affects utilization of distributed resources.
□ They are sometimes referred together as network transparency.
□ As an illustration, consider an email address such as [email protected].
□ It consists of a user’s name and a domain name.
□ Sending mail to such a user does not involve knowing their physical or network location.
□ Nor does the procedure to send an email message depend upon the location of the recipient.
□ Thus electronic mail within the Internet provides both location and access transparency (that is, network transparency).
Inter-process Communication
(Message Passing)

Lecture # 9
Middleware Layers
□ Consider the components shown in the shaded layer in fig 1.

Fig 1: Middleware Layers

□ Inter-process communication in the Internet provides both
datagram and stream communication.
□ Two Transport-Level Protocols
□ UDP (Universal Datagram Protocol)
□ TCP (Transmission Control Protocol)
UDP (Universal Datagram Protocol)
UDP (Universal Datagram Protocol) features
□ Datagrams: independent packets containing messages.
□ The essential feature of datagram networks
□ the delivery of each packet is a one-shot process;
□ no setup is required and once the packet is delivered the network retains no information about it.
□ a sequence of packets transmitted by a single host to a single destination may follow different routes.
□ A UDP datagram is encapsulated inside an IP packet.
□ It has a short header that includes the source and destination port numbers, a length field
and a checksum.
□ Offers no guarantee of delivery.
□ UDP adds no additional reliability mechanism except the checksum which is optional.
□ If the checksum field is non-zero,
□ the receiving host computes a check value from the packet contents and compares it with the
received checksum;
□ packets for which they do not match are dropped.
□ The application program interface to UDP provides a message passing abstraction – the
simplest form of IPC.
□ This enables a sending process to send a single message to a receiving process.
Sockets
□ A software interface, in a process, for sending and receiving
messages from the network.
□ Both forms of communication (UDP and TCP) use the socket
abstraction, which provides an endpoint for communication
between processes.
□ Inter-process communication consists of transmitting a message
between a socket in one process and a socket in another process,
as illustrated in Fig 2

□ Fig 2: Sockets and ports

Sockets
□ For a process to receive messages, its socket must be bound to a local
port and one of the Internet addresses of the computer on which it runs.
□ Messages sent to a particular Internet address and port number can be
received only by a process whose socket is associated with that Internet
address and port number.
□ Processes may use the same socket for sending and receiving messages.
□ Each computer has a large number (216) of possible port numbers for use
by local processes for receiving messages.
□ Any process may make use of multiple ports to receive messages, but a
process cannot share ports with other processes on the same computer.
□ Processes using IP multicast are an exception in that they do share ports.
□ However, any number of processes may send messages to the same port.
□ Each socket is associated with a particular protocol – either UDP or TCP.
Failure Model
□ Failure Model of Communication Channels
□ Reliable communication defined in terms of two properties:
□ Integrity and
□ Validity
□ The integrity property requires that messages should not be corrupted or
duplicated.
□ The use of a checksum ensures that there is a negligible probability that any
message received is corrupted.
□ The validity property requires that any message in the outgoing message
buffer is eventually delivered to the incoming message buffer.

□ Fig 3: Processes and channels

Ordering
□ Some applications require that messages be delivered in
sender order – i.e., the order in which they were
transmitted by the sender.
□ The delivery of messages out of sender order is regarded
as a failure by such applications.
UDP datagram communication
□ To send or receive messages, a process must first create
a socket bound to an Internet address of the local host
and a local port.
□ A server will bind its socket to a server port – one that it
makes known to clients so that they can send messages
to it.
□ A client binds its socket to any free local port.
□ The receive method returns
□ the Internet address and port of the sender,
□ in addition to the message, allowing the recipient to send
a reply.
Some issues relating to datagram communication
□ Message size
□ The underlying IP protocol allows packet size up to 216 (64k)
bytes, which includes the headers as well as message.
□ However, most environments impose a size restriction of 8
kilobytes.
□ Any application requiring messages larger than maximum must
fragment them into chunks of that size.
□ Generally, an application, e.g. DNS, will use size not
excessively large.
Blocking
□ Sockets normally provide non-blocking sends and blocking receives
for datagram communication.
□ The send operation returns when it has handed the message to the
underlying UDP and IP protocols.
□ On arrival, the message is placed in a queue for the socket that is
bound to the destination port.
□ The message can be collected from queue by an outstanding or
future invocation of receive on that socket.
□ Messages are discarded at the destination if no process already has
a socket bound to the destination port.
□ The method receive blocks until a datagram is received, unless a
timeout has been set on the socket.
□ If the process that invokes the receive method has other work to
do while waiting for the message, it should arrange to use a
separate thread.
Timeouts
□ In some programs, it is not suitable that a process
invoking a receive operation should wait indefinitely.
□ To allow for such requirements, timeouts can be set on
sockets.
□ Choosing an appropriate timeout interval is difficult, but it
should be fairly large in comparison with the time
required to transmit a message.
Receive from any
□ The receive method does not specify an origin for
messages.
□ The receive method returns the Internet address and
local port of the sender.
□ It is possible to connect a datagram socket to a particular
remote port and Internet address so that the socket is
only able to send and receive messages from that address.
Use of UDP
□ For some applications, it is acceptable to use a service
that is liable to occasional omission failures.
□ UDP datagrams are sometimes an attractive choice because
they do not suffer from the overheads associated with
guaranteed message delivery.
□ Three main sources of overhead:-
□ Need to store state information at the source and
destination;
□ Transmission of extra messages;
□ latency for the sender.
Client-server architecture
□ #include <iostream>
□ #include <mpi.h>
□ // Define the number of tasks
□ const int NUM_TASKS = 10;
□ // Function to perform computation on task
□ int computeTask(int task) {
□ // Simulated computation
□ return task * task;
□ }
□ int main(int argc, char** argv) {
□ // Initialize MPI
□ MPI_Init(&argc, &argv);
□ // Get the rank of the process and the total number of processes
□ int rank, size;
□ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &size);
□ if (rank == 0) { // Server
□ // Generate tasks and distribute them to clients
□ for (int task = 0; task < NUM_TASKS; ++task) {
□ // Determine which client to send the task to
□ int client = (task % (size - 1)) + 1; // Exclude rank 0 (server)
□ // Send task to client
□ MPI_Send(&task, 1, MPI_INT, client, 0, MPI_COMM_WORLD);
□ }
□ // Receive results from clients
□ int results[NUM_TASKS];
□ for (int task = 0; task < NUM_TASKS; ++task) {
□ int result;
□ MPI_Recv(&result, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
□ results[task] = result;
□ }
□ // Process and output results
□ std::cout << "Results:" << std::endl;
□ for (int task = 0; task < NUM_TASKS; ++task) {
□ std::cout << "Task " << task << ": " << results[task] << std::endl;
□ }
Peer to peer architecture
□ #include <iostream>
□ #include <mpi.h>
□ int main(int argc, char** argv) {
□ // Initialize MPI
□ MPI_Init(&argc, &argv);
□ // Get the rank of the process and the total number of processes
□ int rank, size;
□ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &size);
□ // Each process computes a value based on its rank
□ int myValue = rank * rank;
□ // Buffer to store received values from other processes
□ int receivedValues[size];
□ // Share my value with all other processes
□ MPI_Allgather(&myValue, 1, MPI_INT, receivedValues, 1, MPI_INT,
MPI_COMM_WORLD);
□ // Output received values from all processes
□ std::cout << "Process " << rank << " received values: ";
□ for (int i = 0; i < size; ++i) {
□ std::cout << receivedValues[i] << " ";
□ }
□ std::cout << std::endl;
 Inter-process Communication
(Message Passing)

Lecture # 10 Lecture # 9
Interprocess Communication (IPC)
□ Interprocess communication (IPC) mechanisms provide
ways for processes to communicate and synchronize with
each other in a computing system.
□ Shared memory model
□ Message passing
Message Passing
□ Message Passing Interprocess Communication (IPC) is a
mechanism that allows different processes to
communicate and exchange data with each other. In
message passing IPC, processes communicate by sending
and receiving messages rather than sharing a common
address space.
Operation in message passing
□ There are typically two main components involved in
message passing IPC:
□ Sender: The process that sends the message.
□ Receiver: The process that receives the message.
Operation in message passing
□ #include <iostream>
□ #include <mpi.h>
□ #include <omp.h>
□ int main(int argc, char** argv) {
□ MPI_Init(&argc, &argv);
□ int world_rank, world_size;
□ MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &world_size);
□ if (world_size != 2) {
□ std::cerr << "This example requires exactly 2 MPI processes." << std::endl;
□ MPI_Abort(MPI_COMM_WORLD, 1);
□ }
□ // OpenMP parallel region
□ #pragma omp parallel num_threads(2)
□ {
□ // Get thread number
□ int thread_id = omp_get_thread_num();
□ // Only first thread in each process will perform send or receive
□ if (thread_id == 0) {
□ if (world_rank == 0) { // Process P
□ int data = 42;
□ MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
□ std::cout << "Process P sent data: " << data << std::endl;
□ } else if (world_rank == 1) { // Process Q
□ int received_data;
□ MPI_Recv(&received_data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
□ std::cout << "Process Q received data: " << received_data << std::endl;
□ }
□ }
□ }
□ MPI_Finalize();
□ return 0;
□ }
Concepts in the MPI standard for building
distributed-memory parallel applications
□ MPI_Init: Initializes MPI environment.
□ MPI_Finalize: Finalizes MPI environment.
□ MPI_Comm_rank: Retrieves the rank of the calling process within the
communicator.
□ MPI_Comm_size: Retrieves the size of the communicator.
□ MPI_Send: Sends a message from one process to another.
□ MPI_Recv: Receives a message sent by another process.
□ MPI_Bcast: Broadcasts a message from one process to all other
processes in the communicator.
□ MPI_Reduce: Combines values from all processes and returns a result to
one process.
□ MPI_Wait: Waits for an MPI request to complete.
□ MPI_Isend: Starts a non-blocking send operation.
□ MPI_Irecv: Starts a non-blocking receive operation.
□ MPI_Test: Tests if a request has completed.
Blocking and non-blocking communication
□ Blocking Communication:
In blocking communication, a process that initiates a
communication operation (send or receive) is blocked until the
operation completes. The blocking nature of this communication
means that the sender will wait until the receiver has received
the message, and the receiver will wait until the sender has sent
the message.
□ Characteristics of blocking Communication:
□ Synchronization:
□ Simple Programming Model:
□ Potential Deadlocks:
□ Resource Utilization:
Blocking Communication:
□ # Blocking Send and Receive in MPI (Python)
□ from mpi4py import MPI

□ comm = MPI.COMM_WORLD
□ rank = comm.Get_rank()

□ if rank == 0:
□ data = {'message': 'Hello, World!'}
□ comm.send(data, dest=1) # Blocking send
□ else:
□ data = comm.recv(source=0) # Blocking receive
□ print(f"Process {rank} received: {data}")
Blocking and non-blocking communication
□ Non-blocking Communication:
In non-blocking communication, a process initiates a
communication operation and proceeds with its execution
without waiting for the operation to complete. This allows the
sender or receiver to perform other tasks concurrently with the
communication operation.
□ Characteristics of Non-blocking Communication:
□ Asynchronous:
□ Overlap of Computation and Communication:
□ Complex Programming Model:
□ Avoidance of Deadlocks:
Non-blocking Communication:
□ # Non-blocking Send and Receive in MPI (Python)
□ from mpi4py import MPI

□ comm = MPI.COMM_WORLD
□ rank = comm.Get_rank()

□ req = MPI.Request()

□ if rank == 0:
□ data = {'message': 'Hello, World!'}
□ req = comm.isend(data, dest=1) # Non-blocking send
□ # Continue with other computation
□ req.Wait() # Wait for completion if necessary
□ else:
□ data = {}
□ req = comm.irecv(source=0) # Non-blocking receive
□ # Continue with other computation
□ req.Wait() # Wait for completion if necessary
□ data = req.recv()

□ print(f"Process {rank} received: {data}")

Blocking and non-blocking communication
□ Blocking communication ensures synchronization at the
cost of potential inefficiency and deadlocks, while
non-blocking communication allows asynchronous
progress and potential performance improvements but
introduces complexity in programming. The choice
between blocking and non-blocking communication
depends on the specific requirements of the application
and the desired trade-offs between simplicity and
performance.
Types of communication channel
□ Message passing IPC can be implemented using various
techniques:
□ Direct / Indirect Communication:
□ Synchronous / Asynchronous Communication:
□ Explicit/ Automatic Communication:
Symmetric/ asymmetric address
□ In MPI (Message Passing Interface), a "symmetric address"
refers to an address space where each process in a
parallel program sees the memory of all other processes
as if it were its own. This implies that any process can
directly access the memory of any other process in a
symmetric manner.
□ In contrast, an "asymmetric address" refers to an address
space where processes do not necessarily see the
memory of other processes in the same way. Access to
remote memory may require specific communication
operations, such as sending or receiving messages, rather
than direct memory access.
Scheduling of message passing
□ In MPI (Message Passing Interface), scheduling refers to
the way in which communication operations (like sending
and receiving messages) are managed and executed
among processes. MPI supports both static and dynamic
scheduling approaches:
□ Static Scheduling:
In static scheduling, communication operations are
predetermined and fixed before the program starts executing.
□ Dynamic Scheduling:
In dynamic scheduling, communication operations are determined
and adjusted dynamically during program execution based on
runtime conditions.
Static scheduling in MPI
□ #include <iostream>
□ #include <mpi.h>

□ int main(int argc, char *argv[]) {

□ int rank, size;
□ MPI_Init(&argc, &argv);
□ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &size);

□ if (rank == 0) {
□ // Process 0 sends a message to Process 1
□ int data = 42;
□ MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
□ } else if (rank == 1) {
□ // Process 1 receives the message from Process 0
□ int received_data;
□ MPI_Recv(&received_data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
□ std::cout << "Process 1 received data: " << received_data << std::endl;
□ }

□ MPI_Finalize();
□ return 0;
□ }
Dynamic scheduling in MPI
□ #include <iostream>
□ #include <mpi.h>

□ int main(int argc, char *argv[]) {

□ int rank, size;
□ MPI_Init(&argc, &argv);
□ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &size);

□ if (rank == 0) {
□ // Process 0 dynamically determines whether to send a message
□ if (some_condition) {
□ int data = 42;
□ MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
□ }
□ } else if (rank == 1) {
□ // Process 1 dynamically determines whether to receive a message
□ if (some_condition) {
□ int received_data;
□ MPI_Recv(&received_data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
□ std::cout << "Process 1 received data: " << received_data << std::endl;
□ }
□ }

□ MPI_Finalize();
□ return 0;
□ }
Advantages of IPC
□ However, message passing IPC also introduces overhead due
to message copying and context switching between processes.
Careful design and optimization are necessary to minimize this
overhead and maximize performance. Message passing IPC
offers several advantages, including:
□ Isolation: Processes have separate address spaces, providing
better isolation and security.
□ Modularity: Processes can be developed and maintained
independently, promoting modularity and code reuse.
□ Scalability: Message passing can be scaled across multiple
systems, making it suitable for distributed computing
environments.
□ Flexibility: Different communication patterns can be
implemented, such as one-to-one, one-to-many, or
many-to-many communication.
Remote Procedure call

Lecture # 11
Remote Procedure call
□ A remote procedure call (RPC) is a protocol that allows
a computer program to cause a subroutine or procedure
to execute in another address space (commonly on
another computer on a shared network) without the
programmer explicitly coding the details for this remote
interaction.
RPC Architecture
□ RPC architecture has mainly five components of the
program:
□ Client
□ Client Stub
□ RPC Runtime
□ Server Stub
□ Server
How RPC Works?
□ Following steps take place during the RPC process:
□ Step 1) The client, the client stub, and one instance of RPC run time execute on
the client machine.
□ Step 2) A client starts a client stub process by passing parameters in the usual
way. The client stub stores within the client’s own address space. It also asks the
local RPC Runtime to send back to the server stub.
□ Step 3) In this stage, RPC accessed by the user by making regular Local
Procedural Call. RPC Runtime manages the transmission of messages between the
network across client and server. It also performs the job of retransmission(if the
message got lost), acknowledgment, routing, and encryption.
□ Step 4) After completing the server procedure, it returns to the server stub,
which packs (marshalls) the return values into a message. The server stub then
sends a message back to the transport layer.
□ Step 5) In this step, the transport layer sends back the result message to the
client transport layer, which returns back a message to the client stub.
□ Step 6) In this stage, the client stub demarshalls (unpack) the return parameters,
in the resulting packet, and the execution process returns to the caller.
Parallel MPI and OpenMP Program with Reduction
□ #include <iostream>
□ #include <mpi.h>
□ #include <omp.h>
□ using namespace std;
□ int main(int argc, char *argv[]) {
□ int rank, size;
□
□ MPI_Init(&argc, &argv);
□ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
□ MPI_Comm_size(MPI_COMM_WORLD, &size);
□
□ int total_threads = 4; // Total threads per MPI process
□
□ // Parallel region with OpenMP
□ #pragma omp parallel num_threads(total_threads)
□ {
□ int thread_id = omp_get_thread_num();
□ int num_threads = omp_get_num_threads();
□
□ // Compute some parallel task
□ int local_result = thread_id + rank * num_threads;
□
□ // Synchronize threads within MPI process
□ #pragma omp barrier
□
□ // Reduce local results to get global result
□ int global_result;
□ MPI_Reduce(&local_result, &global_result, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
□
□ // Print global result from root process
□ if (rank == 0 && thread_id == 0) {
□ cout << "Global result: " << global_result << endl;
□ }
□ }
□
□ MPI_Finalize();
□
□ return 0;
□ }
Characteristics of RPC
□ Here are the essential characteristics of RPC:
□ The called procedure is in another process, which is likely
to reside in another machine.
□ The processes do not share address space.
□ Parameters are passed only by values.
□ RPC executes within the environment of the server
process.
□ It doesn’t offer access to the calling procedure’s
environment.
Features of RPC
□ Here are the important features of RPC:
□ Simple call syntax
□ Offers known semantics
□ Provide a well-defined interface
□ It can communicate between processes on the same or
different machines
Communication Protocols For RPCs
□ The following are the communication protocols that are
used:
□ Request Protocol
□ Request/Reply Protocol
□ The Request/Reply/Acknowledgement-Reply Protocol
Request Protocol:
□ The Request Protocol is also known as the R protocol.
□ It is used in Remote Procedure Call (RPC) when a request is made from the calling procedure to the called procedure.
After execution of the request, a called procedure has nothing to return and there is no confirmation required of the
execution of a procedure.
□ Because there is no acknowledgement or reply message, only one message is sent from client to server.
□ A reply is not required so after sending the request message the client can further proceed with the next request.

□ May-be call semantics are provided by this protocol, which eliminates the requirement for retransmission of request
packets.
□ Asynchronous Remote Procedure Call (RPC) employs the R protocol for enhancing the combined performance of the
client and server. By using this protocol, the client need not wait for a reply from the server and the server does not
need to send that.
□ In an Asynchronous Remote Procedure Call (RPC) in case communication fails, the RPC Runtime does not retry the
request. TCP is a better option than UDP since it does not require retransmission and is connection-oriented.
□ In most cases, asynchronous RPC with an unstable transport protocol is utilized to implement periodic update services.
One of its applications is the Distributed System Window.
Request/Reply Protocol:
□ The Request-Reply Protocol is also known as the RR protocol.
□ It works well for systems that involve simple RPCs.
□ The parameters and result values are enclosed in a single packet
buffer in simple RPCs. The duration of the call and the time
between calls are both briefs.
□ This protocol has a concept base of using
□ Here, a reply from the server is treated as the acknowledgement
(ACK) for the client’s request message, and a client’s following call
is considered as an acknowledgement (ACK) of the server’s reply
message to the previous call made by the client.
□ To deal with failure handling e.g. lost messages, the timeout
transmission technique is used with RR protocol.
□ If a client does not get a response message within the
predetermined timeout period, it retransmits the request message.
□ Exactly-once semantics is provided by servers as responses get
held in reply cache that helps in filtering the duplicated request
messages and reply messages are retransmitted without processing
the request again.
□ If there is no mechanism for filtering duplicate messages then at
least-call semantics is used by RR protocol in combination with
timeout transmission.
The Request/Reply/Acknowledgement-Reply
Protocol:
□ This protocol is also known as the RRA protocol
(request/reply/acknowledge-reply).
□ Exactly-once semantics is provided by RR protocol
which refers to the responses getting held in reply cache
of servers resulting in loss of replies that have not been
delivered.
□ The RRA (Request/Reply/Acknowledgement-Reply )
Protocol is used to get rid of the drawbacks of the RR
(Request/Reply) Protocol.
□ In this protocol, the client acknowledges the receiving of
reply messages and when the server gets back the
acknowledgement from the client then only deletes the
information from its cache.
□ Because the reply acknowledgement message may be
lost at times, the RRA protocol requires unique ordered
message identities. This keeps track of the
acknowledgement series that has been sent.
Complicated RPCs
□ RPCs that involve long-duration calls or large gaps
between calls.
□ RPCs that involve parameters(arguments) and/or result in
values that are too large to fit in a single datagram packet.
RPCs that involve long-duration calls or large gaps
between calls:
□ The client probes the server regularly: After the submission of
a request message from a client to the server, the client
continuously sends a probe packet which a server needs to
acknowledge.The exception status is communicated to the
corresponding user if a communication failure occurs. Each probe
packet contains the message identifier from the initial request
message.
□ The server generates an acknowledgement regularly: If the
generation of the next packet by the server gets delayed then the
predicted retransmission time interval, then it generates an
acknowledgement on its own. Hence, during a long-duration call,
many acknowledgements may be generated from the server as
several acknowledgements directly correspond to the call duration.
If within the set interval of time the response or acknowledgement
from the server has not been received by the client then it comes
to the conclusion that either server has crashed or the failure
occurs on the client-side or in case of communication failure user is
alerted about the exception condition.
RPCs that involve parameters/arguments and/or result in
values that are too large to fit in a single
□ Datagram packet:
□ RPCs with Long Messages: To handle such an RPC, employ
many physical RPCs for a single logical RPC. The sending of
data in each physical RPC is made in the size of a single
datagram packet. This technique is inefficient since each RPC
incurs a set amount of overhead regardless of the quantity of
data transmitted.
□ Multidatagram Messages: Multidatagram messages are
another approach for dealing with sophisticated RPCs in this
category. It involves the division of long RPC
parameters(arguments) or result into many packets and then
sent in multiples. All the packets in a multi datagram message
utilize a single acknowledgement packet for enhancing
communication performance.
Advantages of RPC
□ Here are Pros/benefits of RPC:
□ RPC method helps clients to communicate with servers by the conventional use of
procedure calls in high-level languages.
□ RPC method is modeled on the local procedure call, but the called procedure is
most likely to be executed in a different process and usually a different computer.
□ RPC supports process and thread-oriented models.
□ RPC makes the internal message passing mechanism hidden from the user.
□ The effort needs to re-write and re-develop the code is minimum.
□ Remote procedure calls can be used for the purpose of distributed and the local
environment.
□ It commits many of the protocol layers to improve performance.
□ RPC provides abstraction. For example, the message-passing nature of network
communication remains hidden from the user.
□ RPC allows the usage of the applications in a distributed environment that is not
only in the local environment.
□ With RPC code, re-writing and re-developing effort is minimized.
□ Process-oriented and thread-oriented models support by RPC.
Disadvantages of RPC
□ Here are the cons/drawbacks of using RPC:
□ Remote Procedure Call Passes Parameters by values only and
pointer values, which is not allowed.
□ Remote procedure calling (and return) time (i.e., overheads)
can be significantly lower than that for a local procedure.
□ This mechanism is highly vulnerable to failure as it involves a
communication system, another machine, and another
process.
□ RPC concept can be implemented in different ways, which is
can’t standard.
□ Not offers any flexibility in RPC for hardware architecture as
It is mostly interaction-based.
□ The cost of the process is increased because of a remote
procedure call.
External Data Representation (IPC)
CORBA’s CDR
Lecture # 12
External data representation
▪ Information stored in running programs
o represented as data structures – for example, by sets of
interconnected objects.
▪ The information in messages consists of sequences of
bytes.
▪ The data structures must be flattened (converted to a
sequence of bytes) before transmission and rebuilt on
arrival.
External data representation
Why?
1. Not all computers store primitive values such as
integers in the same order.
□ There are two variants for the ordering of integers:
o the big-endian order, in which the most significant byte comes
first;
o and little-endian order, in which it comes last.

2. The representation of floating-point numbers also

differs between architectures.
External data representation
3. Another issue is the set of codes for representing
characters:
o e.g., the majority of applications on systems like UNIX use
ASCII character coding, taking one byte/character,
o whereas the Unicode standard takes 2 bytes/character and thus
allows for representation of texts in many different languages.
 External data representation
▪ Two possibilities for exchanging binary data values
between any two computers :-
I. The values converted to an agreed external format
before transmission and converted to the local form
on receipt;
o if the two computers are of the same type, the conversion to
external format can be omitted.
II. The values transmitted in the sender’s format,
together with an indication of the format used, and
the recipient converts the values if necessary.
External data representation and marshaling
▪ To support RMI/RPC, any data type that can be passed as
argument or returned as result must be able to be flattened.
□ External data representation: An agreed standard for the
representation of data structures and primitive values.
□ Marshalling: assembling or translation of structured data
and primitive values into an external data representation.
▪ Unmarshalling: disassembling or rebuilding of the data
structures and primitive values from their external data
representation.
 Three alternative approaches to External data
representation and marshaling
I. CORBA’s Common Data Representation,
o concerned with external representation for structured and
primitive types that can be passed as arguments and
results of remote method invocations in CORBA.
o can be used by a variety of programming languages.

II. Java’s object serialization: concerned with the flattening

and external data representation of any single object or tree
of objects for transmission in a message or storing on a
disk.
o for use only by Java.
 Three alternative approaches to External data
representation and marshaling
III. XML (Extensible Markup Language) defines a textual
format for representing structured data.

o Originally intended for documents (accessible on the Web)

containing textual self-describing structured data
o but now also used to represent the data sent in messages
exchanged by clients and servers in web services.
Issues regarding design of marshalling methods
1. Who performs marshalling/unmarshalling?
▪ In the first two cases, the marshalling and unmarshalling
activities are carried out by a middleware layer without any
involvement on the part of the application programmer.
▪ Even in the case of XML,
o which is textual and therefore more accessible to hand-
encoding,
o software for marshalling and unmarshalling is available
for all commonly used platforms and programming
environments.
Issues regarding design of marshalling methods
1. Who performs marshalling/unmarshalling? (cont’d)

▪ Because marshaling requires the consideration of finest details

of the representation of primitive components of composite
objects, the process is likely to be error-prone if carried out
by hand.
2. Compactness is another issue that can be addressed in the
design of automatically generated marshalling procedures.
▪ In the first two approaches, the primitive data values are
marshalled into a binary form.
Issues regarding design of marshalling methods
▪ In the third approach (XML), the primitive values are
represented textually.
▪ The textual representation of a data value will generally
be longer than the equivalent binary representation.
▪ What will be the size (in bytes) of each of following
data values: 4560 and “4560”?
3. Whether the marshalled data should include
information concerning the type of its contents?
▪ CORBA’s representation includes just the values of the
objects transmitted, and nothing about their types.
Issues regarding design of marshalling methods
▪ On the other hand, both Java serialization and XML do include type
information, but in different ways.
▪ Java puts all of the required type information into the serialized
form, but XML documents may refer to externally defined sets of
names (with types) called namespaces.
▪ Although we are interested in the external representation for the
arguments and results of RMIs and RPCs,
o it does have a more general use for representing data structures
or structured documents
o in a form suitable for transmission in messages or storing in files.
CORBA’s Common Data Representation (CDR)
▪ CORBA CDR: external data representation defined with
CORBA 2.0.
▪ CDR can represent all of the data types that can be used as
arguments and return values in remote invocations in CORBA.
▪ These consist of primitive as well as composite types.
▪ The 15 primitive types included are
o short (16-bit), long (32-bit), unsigned short, unsigned long,
o float (32-bit), double (64-bit), char, boolean (TRUE, FALSE),
o octet (8-bit), and any (which can represent any basic or
constructed type);
o An any logically contains a TypeCode and a value that is
described by the TypeCode.
CORBA’s Common Data Representation (CDR)
□ CDR also supports a range of composite types
described in Fig 1.

Fig 1: Composite types in CORBA CDR

CORBA’s Common Data Representation (CDR)
▪ Each argument or result in a remote invocation is
represented by a sequence of bytes in the invocation or
result message.
▪ Primitive types: CDR defines a representation for both big-
endian and little-endian orderings.
o Values are transmitted in the sender’s ordering, which is
specified in each message.
o The recipient translates if it requires a different ordering.
▪ Each primitive value is placed at an index in the sequence
of bytes according to its size.
CORBA’s CDR
▪ Suppose that the sequence of bytes is indexed from zero
upwards. 0 1 2 … n-1

▪ Then a primitive value of size n bytes (where n = 1, 2, 4 or 8)

is appended at an index that is a multiple of n in the stream of
bytes.
▪ Floating-point values follow the IEEE standard, in which
o the sign, exponent and fractional part are in bytes 0–n for big-
endian ordering and
o the other way round for little-endian.
▪ Characters represented by a code set agreed between client and
server.
CORBA’s CDR
▪ Constructed types: The primitive values that comprise each
constructed type are added to a sequence of bytes in a
particular order.
▪ Fig 2 shows a message in CORBA CDR that contains the
three fields of a struct whose respective types are string,
string and unsigned long.
▪ The flattened form represents a Person struct with value: {‘Smith’,
‘London’, 1984}

▪ .

Fig 2: Message in CORBA CDR

CORBA’s CDR

Index in sequence of Value

bytes
0–3 5
4 – 11 “Smith ”
12 – 15 6
16 – 23 “London ”
24 – 27 1984
CORBA’s Common Data Representation (CDR)
▪ The representation of each string consists of an unsigned long
representing its length followed by the characters in the string.
▪ For simplicity, we assume that each character occupies just one
byte.
▪ Variable-length data is padded with zeros so that it has a
standard form, enabling marshalled data or its checksum to be
compared.
▪ Note that each unsigned long, which occupies four bytes, starts
at an index that is a multiple of four.
▪ The figure does not distinguish between the big- and little-
endian orderings.
CORBA’s CDR
▪ CORBA CDR can represent any data structure that can be
composed from the primitive and constructed types, but
without using pointers.
▪ Another example of an external data representation is the
Sun XDR standard, which is specified in RFC 1832 and
described in www.cdk5.net/ipc.
▪ It was developed by Sun for use in the messagesexchanged
between clients and servers in Sun NFS.
▪ The type of a data item is not given with the data
representation in the message in either the CORBA CDR
or the Sun XDR standard.
▪ This is because it is assumed that the sender and recipient
have common knowledge of the order and types of the
data items in a message.
Marshalling in CORBA
▪ Marshalling operations can be generated automatically from the
specification of the types of data items.
▪ The types of the data structures and the types of the basic data items
are described in CORBA IDL.
▪ For example, we might use CORBA IDL to describe the data
structure in the message in Fig 2 as follows:-
struct Person{
string name;
string place;
unsigned long year; };
▪ The CORBA interface compiler generates appropriate marshalling
and unmarshalling operations for the arguments and results of remote
methods from the definitions of the types of their parameters and
results.
 Lecture #13
External Data Representation (IPC)
Java object serialization
CS-482
Java object serialization
▪ In Java RMI, both objects and primitive data values may be passed as
arguments and results of method invocations.
▪ Serialization is a mechanism of converting the state of an object into a byte
stream.

▪ Deserialization is the reverse process where the byte stream is used to

recreate the actual Java object in memory.
▪ To make a Java object serializable we implement the java. io.Serializable
interface.
Java object serialization
▪ For example, the Java class equivalent to the Person struct
defined in CORBA IDL might be:-
public class Person implements Serializable {
private String name;
private String place;
private int year;
public Person(String aName, String aPlace, int aYear) {
name = aName;
place = aPlace;
year = aYear;
} // followed by methods for accessing the instance variables }
▪ The above class states that it implements the Serializable
interface.
Java object serialization
▪ Stating that a class implements the Serializable interface
(provided in the java.io package) has the effect of allowing
its instances to be serialized.
▪ Deserialization consists of restoring the state of an object
or a set of objects from their serialized form.
▪ It is assumed that the process that does the deserialization
has no prior knowledge of the types of the objects in the
serialized form.
▪ Therefore some information about the class of each object
is included in serialized form to enable the recipient to
load the appropriate class.
Java object serialization
▪ The information about a class consists of the name of the
class and a version number.
▪ The version number changes when major changes are
made to the class.
▪ It can be set by the programmer or calculated
automatically as a hash of
o the name of the class and its
o instance variables, methods and interfaces.

▪ The process that deserializes an object can check that it has

the correct version of the class.
Java object serialization
▪ Java objects can contain references to other objects.
▪ When an object is serialized, all the objects it references
are serialized with it to ensure that
o when the object is reconstructed,
o all its references can be fulfilled at the destination.

▪ References are serialized as handles.

o the next number in a sequence of positive integers.

▪ The serialization procedure must ensure that each object is

written once only
o on the next occurrence of an object,
o the handle is written instead of the object.
Java object serialization
▪ To serialize an object,
o its class information is written out,
o followed by the types and names of its instance variables.
▪ If the instance variables belong to new classes,
o then their class information must also be written out,
o followed by the types and names of their instance variables.
▪ This recursive procedure continues until the class information
and types and names of instance variables of all of the necessary
classes have been written out.
Java object serialization
▪ The contents of the instance variables that are primitive types
are written in a portable binary format using methods of the
ObjectOutputStream class.
▪ Strings and characters are written by its writeUTF method
using the Universal Transfer Format (UTF-8),
o ASCII characters to be represented remain unchanged (in
one byte),
o whereas Unicode characters are represented by multiple
bytes.
▪ Strings are preceded by the number of bytes they occupy in the
stream.
Java object serialization
▪ As an example, consider the serialization of the following object:-
Person p = new Person(“Smith”, “London”, 1984);
▪ The true serialized form contains additional type markers; h0 and h1 are
handles.

▪ The serialized form above omits

o the values of the handles and
o of the type markers
❖ that indicate the objects, classes, and strings in the full serialized form.
Java object serialization
▪ In order to serialize the Person object,
o create an instance of the class ObjectOutputStream and
o invoke its writeObject method, passing the Person object as its
argument.

▪ To deserialize the person object from a stream of data,

o create an instance of the class ObjectInputStream and
o use its readObject method to reconstruct the original object.
▪ Programmers with special requirements may write their own version
of the read and write methods.
▪ Programmers may also modify the effects of serialization by declaring
variables that should not be serialized as transient.
▪ Examples of things that should not be serialized are references to local
resources such as files and sockets.
The use of reflection
▪ The Java language supports reflection
o the ability to enquire about the properties of a class,
o such as the names and types of its field variables and
methods.
▪ It also enables creation of classes from their names, and creation
of constructor with given argument types for a given class.
▪ Reflection makes it possible to do serialization and
deserialization in a completely generic manner.
▪ This means that there is no need to generate special marshalling
functions for each type of object, like CORBA.
The use of reflection
▪ At the serialization end, Java object serialization uses
reflection to find out
o the class name of the object to be serialized and
o the names, types and values of its instance variables.
▪ For deserialization, the class name in the serialized form is
used to create a class.
o This is used to create a new constructor with argument
types corresponding to those specified in serialized form.
o Finally, the new constructor is used to create a new object
with instance variables whose values are read from the
serialized form.
The use of reflection
The use of reflection
The use of reflection
Remote object references
▪ Apply only to languages such as Java and CORBA that support
the distributed object model.
▪ Not relevant to XML.
▪ A remote object reference is an identifier for a remote object
that is valid throughout a distributed system.
▪ When a client invokes a method in a remote object,
o an invocation message needs to specify a remote object
reference to specify which object is to be invoked.
▪ Remote object references are also passed as arguments and
returned as results of remote method invocations.
▪ Each remote object has a single remote object reference
Remote object references
▪ Remote object references can be compared to see whether
they refer to the same remote object.
▪ Remote object references must be generated in a manner
that ensures uniqueness over space and time.
▪ In general, there may be many processes hosting remote
objects, so remote object references must be unique among
all of the processes in the various computers in a distributed
system.
▪ Even after the remote object is deleted,
o its remote reference should not be reused,
o because its potential invokers may retain obsolete
remote object references.
Remote object references
▪ Any attempt to invoke a deleted object should produce an error
rather than allow access to a different object.
External representation for Remote object references
▪ There are several ways to ensure that a remote object reference
is unique.
▪ One way is to construct a remote object reference by
concatenating the Internet address of its host computer and the
port number of the process that created it with the time of its
creation and a local object number.

▪ The local object number is incremented each time an object is

created in that process.
External representation for Remote object
references
▪ The port number and time together produce a unique object
identifier on that computer.

▪ In the simplest implementations of RMI, remote objects live only

in the process that created them and survive only as long as that
process continues to run.
▪ In such cases, the remote object reference can be used as the
address of the remote object.
▪ In other words, invocation messages are sent to the Internet
address in the remote reference and to the process on that
computer using the given port number.
Remote object references
▪ To allow remote objects to be relocated into a different process on a
different computer, the remote object reference should not be used as
the address of the remote object.
▪ A form of remote object reference allows objects to be activated in
different servers throughout its lifetime.
▪ The last field contains some information about the interface of the
remote object, e.g., the interface name.

▪ This information is relevant to any process that receives a remote

object reference as an argument or result of a remote invocation, as it
needs to know about methods offered by the remote object.
 Lecture #14
Remote Invocation
CS-482
REMOTE INVOCATION
▪ To step through the remote invocation paradigms (upper middleware
layer).

Fig 1: Middleware layers

▪ This layer is concerned with how processes (or entities such as objects
or services) communicate at a higher level of abstraction in a distributed
system.
 Remote Invocation Paradigms
I. Request-reply protocols
II. Remote Procedure Call
III. Remote Method Invocation

IV. Request-reply protocols represent a pattern of two-way

exchange of messages as encountered in client-server
computing.
o Such protocols provide
❖ relatively low-level support for requesting the
execution of a remote operation, and
❖ also provide direct support for RPC and RMI.
Remote Invocation Paradigms
II. The Remote Procedure Call, or RPC model represents the
extension of the conventional procedure call model to distributed
systems
o Conventional procedure call model: An earliest example of a
more programmer-friendly model.
T = P(A,B)

o RPC allows client processes to call procedures transparently in

separate server processes and generally in different computers
from the client.
 Remote Invocation Paradigms
III. In the 1990s, the object-based programming model was
extended to allow Remote Method Invocation (RMI).
o RMI is an extension of Local Method Invocation (LMI)
o that allows an object in one process to invoke the
methods of an object in another process.
Request-Reply Protocols
▪ This form of communication is designed to support the roles and
message exchanges in typical client-server interactions.

▪ In the normal case, request-reply communication is synchronous

because the client process blocks until the reply arrives from the
server.

▪ It can also be reliable because the reply from the server is

effectively an acknowledgement to the client.
Request-Reply Protocols
▪ Asynchronous request-reply communication is an alternative
that may be useful in situations where clients can afford to
retrieve replies later.

▪ The client-server exchanges can be described

o in terms of the send and receive operations in the Java API
for UDP datagrams,
o although many current implementations use TCP streams.
Request-Reply Protocols
▪ A protocol built over datagrams avoids unnecessary
overheads associated with the TCP stream protocol.
▪ In particular:
a) acknowledgements are redundant, since requests are
followed by replies.
b) Establishing a connection involves two extra pairs of
messages in addition to the pair required for a request
and a reply.
c) Flow control is redundant for the majority of
invocations, which pass only small arguments and
results.
Request-Reply Protocols
□ The request-reply protocol Based on a trio of communication
primitives,
o doOperation,
o getRequest and
o sendReply, as shown in Figure 2.

Fig 2: Request-reply communication

Request-Reply Protocols

Fig 2: Request-reply communication

▪ The request-reply protocol matches requests to replies.

▪ It may be designed to provide certain delivery guarantees.
▪ If UDP datagrams are used,
o the delivery guarantees must be provided by the request-reply protocol,
o which may use the server reply message as an acknowledgement of the client request
message.
Request-Reply Protocols

Fig 2: Request-reply communication

▪ The doOperation is used by clients to invoke remote operations.
▪ Its arguments specify
o the remote server and which operation to invoke,
o together with additional information (arguments) required by the operation.
▪ Its result is a byte array containing the reply.
▪ It is assumed that the client calling doOperation marshals the arguments into
an array of bytes and unmarshals the results from the array of bytesthat
is returned.
Request-Reply Protocols
▪ The first argument of doOperation is an instance of the class
RemoteRef, which represents references for remote
servers.
▪ This class provides methods for getting the Internet address
and port of the associated server.
▪ The doOperation method sends a request message to the
server whose remote reference is given as an argument.
▪ getRequest is used by server process to acquire service
requests.
Request-Reply Protocols
▪ After sending the request message, doOperation invokes
blocking receive to get a reply message from the server.
▪ When the server has performed the specified operation, it
then uses sendReply to send the reply message to theclient.
▪ When reply message received by client, the original
doOperation is unblocked and execution of client program
continues.
▪ The doOperation extracts the result and returns it to the
caller.
Request-Reply Message Structure

Fig 3: Request-reply
message structure

▪ A doOperation in the client generates a requestId for each request

message, and the server copies these IDs into the correspondingreply
messages.
▪ This enables doOperation to check that a reply message is the result
of the current request, not a delayed earlier call.
Request-Reply Message Structure

Fig 3: Request-reply
message structure

▪ The fourth field is an identifier for the operation to be invoked.

o e.g., the operations in an interface might be numbered 1, 2, 3, …,
o But if the client and server use a common language that supports
reflection,
o a representation of the operation itself may be put in this field.
Operations of Request-Reply Protocols
1. public byte[] doOperation (RemoteRef s, int operationId, byte[]
arguments)
o Sends a request message to the remote server and returns the
reply.
o The arguments specify the remote server, the operation to be
invoked and the arguments of that operation.
2. public byte[] getRequest ();
o Acquires a client request via the server port.

3. public void sendReply (byte[] reply, InetAddress clientHost, int

clientPort);
o Sends the reply message reply to the client at its Internet
address and port.
Request-Reply Protocols
□Message identifiers
▪ Each message must have a unique message ID by which it
may be referenced
o useful in the management of messages by providing
additional properties such as reliable message delivery.
▪ A message identifier consists of two parts:-
1. a requestId, which is taken from an increasing sequence
of integers by the sending process;
2. an identifier for the sender process, for example, its
Internet address and port number.
Request-Reply Protocols
▪ The first part makes the identifier unique to the sender, and the
second part makes it unique in the distributed system.
▪ (The second part can be obtained independently – for example,
if UDP is in use, from the message received.)
▪ When the value of the requestId reaches the maximum value for
an unsigned integer (for example, 232 – 1) it is reset to zero.
▪ The only restriction here is that the lifetime of a message
identifier should be much less than the time taken to exhaust
the values in the sequence of integers.
 Lecture #14
Remote Invocation (Request-Reply Protocols)

CS-482
Failure Model of Request-Reply Protocols
▪ If the three primitives doOperation, getRequest and
sendReply are implemented over UDP datagrams, then
they suffer from the same communication failures.
▪ That is:
a) They suffer from omission failures.
b) Messages are not guaranteed to be delivered in sender
order.
▪ In addition, the protocol can suffer from the failure of
processes.
Failure Model of Request-Reply Protocols
▪ We assume that processes have crash failures.
▪ That is, when they halt, they remain halted – they do not
produce Byzantine behavior.
▪ To allow for occasions when
o a server has failed or
o a request or reply message is dropped,
o doOperation uses a timeout when it is waiting to get the
server’s reply message.
▪ The action taken when a timeout occurs depends upon the
delivery guarantees being offered.
Timeouts
▪ There are various options as to what doOperation can do after a
timeout.
▪ The simplest option is
o to return immediately from doOperation
o with an indication to the client that the doOperation has
failed.
▪ Not the usual approach
o the timeout may have been due to the request or reply
message getting lost and
o in the latter case, the operation will have been performed.
Timeouts
▪ To compensate for the possibility of lost messages,
o doOperation sends the request message repeatedly until either it
gets a reply or
o it is reasonably sure that the delay is due to lack of response
from the server rather than to lost messages.

▪ Eventually, when doOperation returns, it will indicate

to the client by an exception that no result was received.
Discarding duplicate request messages
▪ In cases when the request message is retransmitted, the server
may receive it more than once.
▪ For example, the server may receive the first request message
but take longer than the client’s timeout to execute the command
and return the reply.
▪ This can lead to the server executing an operation more than once
for the same request.
▪ To avoid this, the protocol is designed to recognize successive
messages (from the same client) with the same request ID and
to filter out duplicates.
▪ If the server has not yet sent the reply, it need take no special
action – it will transmit the reply when it has finished executing
the operation.
Lost reply messages
▪ If the server has already sent the reply when it receives a
duplicate request
o it will need to execute the operation again to obtain theresult,
o unless it has stored the result of the original execution.
▪ Some servers can execute their operations more than once and
obtain the same results each time.
▪ An idempotent operation is an operation that can be performed
repeatedly with the same effect as if it had been performed
exactly once.
Lost reply messages
▪ For example, an operation to add an element to a set is an
idempotent operation because it will always have the sameeffect
on the set each time it is performed,
▪ whereas an operation to append an item to a sequence is not an
idempotent operation because it extends the sequence each time
it is performed.
▪ A server whose operations are all idempotent need not take
special measures to avoid executing its operations more than
once.
History
▪ For servers that require retransmission of replies without
re-execution of operations, a history may be used.
▪ History: a structure that contains a record of (reply) messages
that have been transmitted.
▪ An entry in a history contains
o a request ID,
o a message and
o an ID of the client to which it was sent.

▪ It enables the server to retransmit reply messages when client

processes request them.
History
▪ A problem associated with the use of a history is its
memory cost.
▪ A history will become very large unless the server can tell
when the messages will no longer be needed for
retransmission.
▪ As clients can make only one request at a time, the server
can interpret each request as an acknowledgement of its
previous reply.
▪ Therefore the history need contain only the last reply
message sent to each client.
History
▪ However, the volume of reply messages in a server’s
history may still be a problem when it has a large number
of clients.
▪ This is compounded by the fact that, when a client process
terminates, it does not acknowledge the last reply it has
received
o messages in the history are therefore normally discarded
after a limited period of time.
Styles of exchange protocols
▪ Three protocols that produce different behaviors in the
presence of communication failures.
▪ They were originally identified by Spector [1982]:
o the request (R) protocol;
o the request-reply (RR) protocol;
o the request-reply-acknowledge reply (RRA) protocol.

Fig 1: Messages passed in three protocols

Styles of exchange protocols
▪ In the R protocol, a single Request message is sent by the client
to the server.
▪ The R protocol may be used when
o there is no value to be returned from the remote operation
and
o the client requires no confirmation that the operation has
been executed.
▪ The client may proceed immediately after the request message
is sent.
▪ The R protocol is implemented over UDP datagrams and
therefore suffers from the same communication failures.
Styles of exchange protocols
▪ The RR protocol is useful for most client-server exchanges.
▪ Special acknowledgement messages are not required, because a
server’s reply message is regarded as an acknowledgement of
the client’s request message.
▪ Similarly, a subsequent call from a client may be regarded as
an acknowledgement of a server’s reply message.
▪ Communication failures due to UDP datagrams being lost may
be masked by
o the retransmission of requests with duplicate filtering
o and the saving of replies in a history for retransmission.
Request-Reply Protocols
▪ The RRA protocol is based on the exchange of three messages:
request-reply-acknowledge reply.

▪ The Acknowledge reply message contains the requestId from

the reply message being acknowledged.
▪ This will enable the server to discard entries from its history.
Request-Reply Protocols
▪ The arrival of a requestId in an acknowledgement message will
be interpreted as
o acknowledging the receipt of all reply messages with lower
requestIds,
o so the loss of an acknowledgement message is harmless.

▪ Although the exchange involves an additional message,

o it need not block the client,
o as the acknowledgement may be transmitted after the reply
has been given to the client.
▪ However it does use processing and network resources.
Use of TCP streams to implement the RR protocol
□ Problems with datagram-communication
1. It is often difficult to decide on an appropriate size for the
buffer in which to receive datagrams.
o In the request-reply protocol, this applies to the buffers
used by the server to receive request messagesand by
the client to receive replies.
2. The limited length of datagrams (usually 8 kilobytes)
o may not be adequate for use in transparent RMI or
RPC systems,
o since the arguments or results of procedures may be of
any size.
Use of TCP streams
▪ The desire to avoid implementing multi-packet protocols
o one of the reasons for choosing to implement
request-reply protocols over TCP streams,
o allowing arguments and results of any size to be
transmitted.
▪ In particular, Java object serialization is a stream protocol
o allows arguments and results to be sent over streams
between the client and server,
o making it possible for collections of objects of any size
to be transmitted reliably.
Use of TCP streams
▪ If the TCP protocol is used for RR protocol, following benefits
are achieved:-

1. It ensures that request and reply messages are delivered

reliably,
o so there is no need for the request-reply protocol to deal
with retransmission of messages and
o filtering of duplicates or with histories.
2. In addition the flow-control mechanism allows large
arguments and results to be passed without taking special
measures to avoid overwhelming the recipient.
Use of TCP streams
3. If successive requests and replies between the same client-
server pair are sent over the same stream,
o the connection overhead need not apply to every remote
invocation.
4. Also, the overhead due to TCP acknowledgement messages
is reduced when a reply message follows soon after a request
message.
o Thus the TCP protocol is chosen for request-reply protocols
because it can simplify their implementation.
Use of TCP streams
▪ However, if the application does not require all of the
facilities offered by TCP, a more efficient, specially
tailored protocol can be implemented over UDP.
▪ e.g., Sun NFS does not require support for messages of
unlimited size, since it transmits fixed-size file blocks
between client and server.
▪ In addition to that, its operations designed to be idempotent,
so it doesn’t matter if operations executedmore than once
to retransmit lost reply messages, making it unnecessary to
maintain a history.
Lecture # 16
Lecture # 15
Remote Invocation (RPC)
CS-482
Remote Procedure Call (RPC)
Procedures on remote machines can be called as if they
are procedures in the local address space.
□ Goal: making the programming of distributed systemslook
similar to conventional programming – i.e. achievinga high
level of distribution transparency.
▪ This concept was first introduced by Birrell and Nelson in
1984.
▪ Represents a major intellectual breakthrough indistributed
computing.
Remote Procedure Call (RPC)
▪ The underlying RPC system then hides important
aspects of distribution, including
o the marshalling and unmarshalling of parameters and results,
o the passing of messages and
o the preserving of the required semantics for the procedure call.
 Design issues for RPC
I. Style of programming promoted by RPC – programming
with interfaces;

II. Call semantics associated with RPC;

III. Key issue of transparency and how it relates to RPCs.

Programming with interfaces
▪ Procedures are implemented so as to hide all
information about them except that available through
its interface.
o Service interface: the specification of the procedures offered by
a server, defining the types of arguments and result of each of the
procedures.
o In the client-server model, each server provides a
set of procedures for use by clients.
❖ For example, a file server would provide procedures for
reading and writing files.
Benefits to programming with interfaces
a) Programmers are concerned only with the abstraction
offered by the service interface and not with
implementation details.
b) Programmers also do not need to know the programming
language or underlying platform used to implement the
service.
c) Provides natural support for software evolution in that
implementations can change as long as interface remains
same.
o More correctly, the interface can also change as long as
it remains compatible with the original.
Restrictions for service interfaces
1) It is not possible for a client process to access the
variables in the server process.
o Therefore the service interface cannot specify direct
access to variables.
o Note that CORBA IDL interfaces can specify attributes,
which seems to break this rule.
o However, the attributes are not accessed directly but by
means of some getter and setter procedures added
automatically to the interface.
Restrictions for service interfaces
2) In particular, parameters can be called by value only.
o The specification of a procedure interface describes the
parameters as input or output, or sometimes both.
o Input parameters are passed to remote server by
❖ sending their values in the request message and then
❖ supplying them as arguments to operation executed
on server.
Restrictions for service interfaces
o Output parameters are returned in the reply message and
are used as the result of the call.
o When a parameter is used for both input and output, the
value must be transmitted in both the request and reply
messages.
4) Addresses in one process are not valid in another remote
one.
o Therefore, addresses cannot be passed as arguments or
returned as results of calls to remote modules.
Programming with interfaces
□ Interface Definition Languages (IDLs): provide a notation for defining interfaces.
▪ Advantage: allow procedures implemented in different languages to invoke one another.
▪ The parameters may be described as input or output in addition to having their type specified.

// In file Person.idl interface PersonList {

readonly attribute string listname;
struct Person { void addPerson(in Person p) ;
string name; void getPerson(in string name, out
string place; Person p);
long year; }; long number(); };

Fig 1: CORBA IDL example

▪ The interface named PersonList specifies the procedures available for RPC in a remote server that
implements that interface.
RPC call semantics
▪ Request-reply protocols discussed earlier showed that doOperation can be
implemented in different ways to provide different delivery guarantees.
▪ The main choices are:
o Retry request message
o Duplicate filtering
o Retransmission of results
▪ Their combinations lead to a variety of possible semantics for the
reliability of remote invocations as seen by the invoker.

Fig 2: Call semantics

Maybe semantics
□ The RPC may be executed once or not at all.
▪ Maybe semantics arises when no fault-tolerance measures are
applied and can suffer from the following types of failure:
o omission failures if the request or reply message is lost;
o crash failures when the server containing the remote operation fails.
▪ If the result message has not been received after a timeout, it is
uncertain whether the procedure has been executed.
▪ If the request message was lost, then the procedure will not have
been executed.
Maybe semantics
▪ On the other hand, the procedure may have been executed and the
result message lost.
▪ A crash failure may occur either before or after the procedure is
executed.
▪ Moreover, in an asynchronous system, the result of executing the
procedure may arrive after the timeout.
▪ Maybe semantics is useful only for applications in which
occasional failed calls are acceptable.

▪ .
At-least-once semantics
□ Can be achieved by the retransmission of request messages,
which masks the omission failures of the request or result
message.
▪ the invoker receives either a result, in which case the
procedure was executed at least once, or an exception
informing it that no result was received.
▪ At-least-once semantics can suffer from following types
of failure:
o crash failures when the server containing the remote
procedure fails;
At-least-once semantics
o arbitrary failures – in cases when the request message
is retransmitted, the remote server receives it and
execute the procedure more than once, possibly causing
wrong values to be stored or returned.
o e.g., an operation to increase a bank balance by Rs
5000/- should be performed only once; if it were
repeated, the balance would grow and grow!
▪ At-least-once call semantics may be acceptable if the
operations in a server are idempotent.
At-most-once semantics
□ This semantics can be achieved by using all of the
fault-tolerance measures outlined in Fig 2.
o the caller receives either a result, in which case the procedure
will have been executed once or
o an exception informing it that no result was received.
▪ As in the previous case, the use of retries masks any omission
failures of the request or result messages.
▪ This set of fault tolerance measures prevents arbitrary failures
by ensuring that for each RPC a procedure is never executed
more than once.
▪ Sun RPC provides at-least-once call semantics.
Transparency
▪ The originators of RPC, Birrell and Nelson [1984], aimed to make
RPCs as much like local procedure calls as possible, with no
distinction in syntax between a local and a remote procedure call.
▪ All the necessary calls to marshalling and message-passing
procedures were hidden from the programmer making the call.
▪ Although request messages are retransmitted after a timeout,
o this is transparent to the caller
o to make the semantics of remote procedure calls like that of
local procedure calls.
Transparency
▪ More precisely, RPC strives to offer at least location and
access transparency,
o hiding the physical location of the (potentially remote)
procedure and
o also accessing local and remote procedures in the same way.
▪ Middleware can also offer additional levels of transparency to
RPC.
▪ However, remote procedure calls are more vulnerable to failure
than local ones, since they involve a network, anothercomputer
and another process.
▪ This requires that clients making remote calls are able to recover
from such situations.
Transparency
▪ The latency of RPC is several orders of magnitude greater
than that of a local one.
▪ This suggests that programs making remote calls should
minimize remote interactions.
▪ The designers of Argus suggested that a caller should be
able to abort an RPC that is taking too long in such a way
that it has no effect on the server.
o To allow this, the server would need to be able to restore
things to how they were before the procedure was called.
Transparency
▪ RPCs also require a different style of parameter passing, as
discussed above.
▪ In particular, RPC does not offer call by reference.
▪ Waldo et al. [1994] say that the difference between local and
remote operations should be expressed at the service interface.
▪ Other systems went further by arguing that the syntax of a
remote call should be different from that of a local call:
o in the case of Argus, the language was extended to make
remote operations explicit to the programmer.
Transparency
▪ The choice as to whether RPC should be transparent is also
available to the designers of IDLs.
o For example, in some IDLs, a remote invocation may throw
an exception when the client is unable to communicate with
a remote procedure.
o This requires that the client program handle such exceptions,
allowing it to deal with such failures.
o An IDL can also provide a facility for specifying the call
semantics of a procedure.
o This can help the designer of the service – for example, if at-
least-once call semantics is chosen to avoid the overheads of
at-most-once, the operations must be designed to be
idempotent.
Transparency

□ The current consensus:

o remote calls should be made transparent in the sense that
the syntax of a remote call is the same as that of a local
invocation,
o but that the difference between local and remote calls
should be expressed in their interfaces.
Homework 1
□ Define a class (in Java) whose instances represent remote
object references. It should contain information similar to
that shown in slide 18 (pdf lec #2) and should provide
access methods needed by higher-level protocols (see
request-reply for example). Give a justification for the
type chosen for the instance variable containing
information about the interface of the remote object (see
slide 14 also in lec #2).
Homework 2
□ Define a class (in Java) whose instances represent request
and reply messages as illustrated in slide 14 (pdf lec #3).
The class should provide a pair of constructors, one for
request messages and the other for reply messages,
showing how the request identifier is assigned. It should
also provide a method to marshal itself into an array of
bytes and to unmarshal an array of bytes into an instance.
 Lecture # 16
Remote Invocation (RPC,RMI)
CS-482
Implementation of RPC
▪ Software components required to implement RPC shown in Fig 1.

Fig 1: Role of client and server stub procedures in RPC

▪ The client that accesses a service includes one stub procedure for each procedure in the
service interface.
▪ The stub procedure behaves like a local procedure to the client,
o but instead of executing the call,
o it marshals the procedure identifier and the arguments into a request message,
o which it sends via its communication module to the server.
▪ When the reply message arrives, it unmarshals the results.
Implementation of RPC

▪ The server process contains a dispatcher together with one server stub procedure and
one service procedure for each procedure in the service interface.
▪ The dispatcher selects one of the server stub procedures according to the procedure
identifier in the request message.
▪ The server stub procedure then unmarshals the arguments in the request message, calls
the corresponding service procedure and marshals the return values for the reply
message.
▪ The service procedures implement the procedures in the service interface.
Implementation of RPC

▪ The client and server stub procedures and the dispatcher can be generated
automatically by an interface compiler from the interface definition of the
service.
▪ RPC is generally implemented over a request-reply protocol.
▪ Choices of invocation semantics – at-least-once or at-most-once.
▪ To achieve this, the communication module will implement the choices in terms
of retransmission of requests, dealing with duplicates andretransmission of
results.
Remote Method Invocation (RMI)
▪ RMI: Method invocations between objects in
different processes, whether in the same computer or
not.
o Closely related to RPC but extended into the world of distributed
objects.
▪ LMI: Method invocations between objects in the
same process.
Remote Method Invocation (RMI)
▪ The commonalities between RMI and RPC:-
1. Both support programming with interfaces.
2. They are both typically constructed on top of request-reply
protocols and can offer a range of call semantics.
3. Both offer a similar level of transparency – i.e., local and
remote calls employ the same syntax
❖ but remote interfaces typically expose the
distributed nature of the underlying call, e.g. by
supporting remote exceptions.
Differences between RMI and RPC
1. The programmer is able to use the full expressive power of
OOP in the development of distributed systems applications.
2. Building on the concept of object identity in OO systems,
o all objects in an RMI-based system have unique object
references (whether they are local or remote),
3. RMI allows the programmer to pass parameters not only by
value but also by object reference.
o The remote end can then access this object using RMI.
o RMI thus offers significantly richer parameter-passing
semantics than in RPC.
Design issues for RMI
▪ RMI shares the same design issues as RPC in terms of
o programming with interfaces,
o call semantics and
o level of transparency.
▪ The key added design issue relates to achieving the
transition from objects to distributed objects.
□ Distributed objects the objects physically distributed into
different processes or computers in a distributed system.
Design issues for RMI
▪ Distributed object systems adopt client-server architecture.
o In RMI, the client’s request to invoke a method of an object is
sent in a message to the server managing the object.
o The method of the object is executed at the server and the result is
returned to the client in another message.
o To allow for chains of related invocations, objects in servers may
become clients of objects in other servers.
▪ Distributed objects can assume other architectural models.
o e.g., objects can be replicated for the usual benefits of fault
tolerance and enhanced performance, and
o objects can be migrated for enhancing performance and
availability.
Design issues for RMI
▪ The possibility of concurrent RMIs from objects in
different computers.
o Therefore the possibility of conflicting accesses arises.
o e.g., objects may use synchronization primitives such as
condition variables to protect access to their instance
variables.
▪ Another advantage:
o an object may be accessed via RMI, or
o it may be copied into a local cache and accessed directly.
The distributed object model
▪ Each process contains a collection of objects,
o some of which can receive both local and remote invocations,
o whereas the other objects can receive only local invocations, as shown in
Fig 2.

Fig 2: Remote and local method invocations

▪ Objects that can receive remote invocations are remote objects.
▪ In Fig 3, the objects B and F are remote objects.
▪ All objects can receive local invocations, although they can receive them only
from other objects that hold references to them.
▪ For example, object C must have a reference to object E so that it can invoke
one of its methods.
The distributed object model
□ Remote object references: An identifier that can be used
throughout a distributed system to refer to a particular unique
remote object.

o Other objects can invoke the methods of a remote object if

they have access to its remote object reference.
❖ e.g., a remote object reference for B in Fig 2 must be available to A.
o Remote object references may be passed as arguments and
results of remote method invocations.
The distributed object model
□ Remote interfaces: Every remote object has a remote interface
that specifies which of its methods can be invoked remotely.

o e.g., the objects B and F in Fig 2 must have remote interf aces.
o Objects in other processes can invoke only the methods that
belong to its remote interface, as shown in Fig 3.

Fig 3: A remote object and its remote interface

The distributed object model

▪ Local objects can invoke the methods in the remote interface as well
as other methods implemented by a remote object.
▪ Note that remote interfaces, like all interfaces, do not have constructors.
▪ The CORBA system provides an interface definition language (IDL),
which is used for defining remote interfaces.
▪ The classes of remote objects and the client programs may be
implemented in any language for which an IDL compiler is available,
such as C++, Java or Python.

Fig 3: A remote object and its remote interface

The distributed object model
▪ CORBA clients need not use the same language as the remote object
in order to invoke its methods remotely.
▪ In Java RMI, remote interfaces are defined in the same way as any
other Java interface.
▪ They acquire their ability to be remote interfaces by extending an
interface named Remote.

▪ Both CORBA IDL and Java support multiple inheritance of

interfaces.
▪ That is, an interface is allowed to extend one or more other interfaces.

Fig 4: Illustration of remote interface in Java

The distributed object model
▪ Actions in a distributed object system An action is
initiated by a method invocation,
o which may result in further invocations of methods in
other objects.
▪ When an action leads to instantiation of a new object,
o that object will normally live within the process where
instantiation is requested –
o e.g. where the constructor was used.
The distributed object model
▪ Distributed applications may provide remote objects with
methods for instantiating objects thus effectively providing
the effect of remote instantiation of objects.
▪ e.g., if the object L in Fig 5 contains a method for creating
remote objects, then the remote invocations from C and K
could lead to the instantiation of the objects M and N,
respectively.

▪ .

Fig 5: Instantiation of remote objects

Exceptions
▪ Any remote invocation may fail for reasons related to the invoked
object being in a different process or computer from the invoker.
▪ Therefore, RMI should be able to raise exceptions such as
o timeouts that are due to distribution
o as well as those raised during the execution of the method invoked.
▪ Examples of the later are an attempt to read beyond the end of a file,
or to access a file without the correct permissions.
▪ CORBA IDL provides a notation for specifying application-level
exceptions, and the underlying system generates standard exceptions
when errors due to distribution occur.
▪ CORBA client programs need to be able to handle exceptions.
o e.g., a C++ client program will use the exception mechanisms in
C++.
Lecture # 18
Lecture # 17
Amdahl’s Law
CS-482
Amdahl’s law
▪ For a given fixed amount of computation W, the speedup
of machine 2 relative to machine 1 is

▪ T1 is the time taken by machine 1 to perform W whereas

T2 is the time taken by machine 2 to perform the same
amount of computation (T2 < T1).
▪ In 1967, Gene Amdahl pointed out, in a short paper,
o the inherent limitations in trying to improve
computer-system performance
o by using multiple processors.
▪ The concept has come to be known as Amdahl’s law.
Amdahl’s law Fig 1a

Fig 1b

▪ Amdahl’s argument is essentially that

o the overall performance improvement observed in an
application program
o (with fixed amount of computation W)
o is limited by that portion of the application
o that is unaffected by whatever change was made to the
system.
Amdahl’s law
▪ Consider the execution time lines shown in fig 2.

Fig 2

▪ The top line shows the time, Told, required to execute some program P
on the system before any changes are made.
▪ Now assume that some change is made to the system that reduces
execution time for some operations by a factor of q.
▪ The program now runs in time Tnew, where Tnew < Told, as shown in
the bottom line.
Amdahl’s law
▪ Hence there are many other operations in the program that
are unaffected by this change.

Fig 2

▪ Let α be the fraction of all operations that are unaffected

by the enhancement.
▪ Then, as shown in the bottom line of fig 2, the new
execution time, Tnew, can be divided into two components.
Amdahl’s law

Fig 2
▪ The first component, αTold, is the execution time of that fraction
of the program that is unaffected by the change.
▪ The second component of Tnew, which is the remaining fraction
1-α of the original execution time, has its performance improved
by the factor q.
▪ Thus, the time required for this component is (1-α) Told/q.
▪ The overall speedup caused by this improvement is then found
to be

□ .
Amdahl’s law
▪ This equation can be used to calculate the overall speedup
obtained due to some improvement in the system, assuming
that q and α can be determined.
▪ However, it is interesting to ask what happens as the impact
on performance of the improvement becomes large, that is,
as q → ∞.
▪ It is easy to show that, in the limit as q → ∞, (1-α) Told/q
→ 0.
▪ Thus, the overall speedup, S, is bounded by 1/α.
▪ That is,

▪ .
Amdahl’s law
▪ This result says that, no matter how much one type of
operation in a system is improved,
▪ the overall performance is inherently limited by the
operations that are unaffected by the improvement.
▪ For example, the best speedup that could be obtained in a
parallel computing system with p processors is p.
▪ However, if 10% of a program cannot be executed in
parallel, the overall speedup when using the parallel machine
is at most 1/α = 1/0.1=10, even if an infinite number of
processors were available.
Amdahl’s law
▪ An obvious corollary to Amdahl's law
o any system designer or programmer should concentrate
on making the common case fast.
▪ That is, operations that occur most often will have the
largest value of α.
▪ Thus, improving these operations will have the biggest
impact on overall performance.
▪ Interestingly, the common cases also tend to be the
simplest cases.
▪ As a result, optimizing these cases first tends to be easier
than optimizing the more complex, but rarely used, cases.
Scaling Amdahl’s law
▪ One of the major criticisms concerning Amdahl’s law has been
that it emphasizes the wrong aspect of the performance
potential of parallel-computing systems.
▪ The argument is that purchasers of parallel systems want tosolve
larger problems within the available time.
▪ Following this line of argument leads to the following “scaled” or
“fixed-time” version of Amdahl's law.
▪ It is common to judge the performance of an application
executing on a parallel system by
o comparing the parallel execution time with p processors, Tp,
o with the time required to execute the equivalent sequential
version of the application program, T1,
o using the speedup Sp = T1/Tp.
Scaling Amdahl’s law
▪ With the fixed-time interpretation, however, the
assumption is that
o there is no single-processor system that is capable of
executing an equivalent sequential version of the
parallel application.
o The single-processor may not have a large enough
memory, for example, or
o the time required to execute the sequential version
would be unreasonably long.
Scaling Amdahl’s law
▪ In this case, the parallel-execution time is divided into
o the parallel component, 1-α, and
o the inherently sequential component, α, giving
o Tp = αT1 + (1-α)T1 as shown in fig 3 below.

Fig 3
▪ T1 is the time in which user wants to run application.
▪ Since no single-processor system exists that is capable of executing an
equivalent problem of this size,
o it is assumed that the parallel portion of the execution time would increase
by a factor of p
o if it were executed on a hypothetical single-processor system.
Scaling Amdahl’s law
□
Homework 1
□ An industrial process simulation involves five steps
which are performed sequentially on system A. The
steps 1 and 2 take 1 and 2 minutes respectively
whereas steps 3 – 5 take 3 minutes each. Then system
A was enhanced by introducing some parallelism to
get a system B. The steps 3 to 5 on B now can be
executed in parallel so that they take an overall time
of 4 minutes whereas steps 1 and 2 are still to be
executed in sequence. Calculate speedup using
appropriate form of Amdahl’s law.
Homework 2
□ An industrial process simulation is to be executed on a
system in the available time, ta, of 10 min which
includes parallel execution time (on 10 processors) of
7.5 min. It has been estimated that the parallel portion
of the execution time would increase by a factor of 12
if it were executed on a hypothetical single-processor
system. Calculate the parallel speedup.
Homework 3
In designing a new computer system, we make an
enhancement that improves some mode of execution by a
factor of 10. This enhancement takes 50% of the time when the
enhanced mode is in use. (Recall that Amdahl’s law uses the
fraction of the original, unenhanced execution time to find
speedup)
a) What is the speedup that we have obtained by using this
fast mode?
b) What percentage of the original execution time has been
converted to fast mode?
Multi threading

Lecture # 19
Thread:
□ A thread, also known as a lightweight process, is the
smallest unit of processing that can be scheduled and
executed by an operating system. Threads are a
fundamental part of multithreading, where a single
process can have multiple threads running concurrently,
allowing for parallel execution of tasks within the same
application.
□ Key Characteristics of Threads
□ Shared Resources:
□ Independent Execution:
□ Lightweight:
Common Uses of Threads
□ User Interfaces: Keeping the UI responsive while
performing background operations.
□ Servers: Handling multiple client requests concurrently.
□ Real-time Systems: Executing multiple real-time tasks
in parallel.
□ Simulations: Running different parts of a simulation
simultaneously.
□ How Threads are Used in RPC Systems
□ Concurrent Request Handling:
□ Improved Responsiveness:
□ Resource Utilization:
Multithreading
□ Multithreading is a programming and execution model that allows multiple threads to be
created within a single process, sharing the same memory space but executing
independently. This can improve the performance of applications by enabling parallelism and
better resource utilization.
□ Key Concepts in Multithreading
□ Thread: A thread is the smallest unit of processing that can be performed in an operating
system. It has its own execution context, including its own stack, register set, and program
counter.
□ Process: A process is an instance of a program in execution. It contains one or more
threads, as well as its own memory space, file handles, and other resources.
□ Concurrency vs. Parallelism:
□ Concurrency: Multiple threads make progress within overlapping time periods. It can be achieved
on a single-core CPU by interleaving thread execution.
□ Parallelism: Multiple threads execute simultaneously, which requires multiple cores or processors.
□ Context Switching: The process of storing the state of a thread or process so that it can
be resumed from the same point later. This is managed by the operating system.
□ Synchronization: Techniques to control the access of multiple threads to shared
resources to avoid conflicts and ensure data consistency. Common synchronization
mechanisms include locks, semaphores, and monitors.
Multithreading models
□ Multithreading models refer to different strategies for
implementing and managing threads within a process. The
choice of model impacts the efficiency and behavior of
thread management, including how threads are created,
synchronized, and scheduled. The primary multithreading
models are:
□ Many-to-One Model
□ One-to-One Model
□ Many-to-Many Model
Many-to-One Model
□ In the many-to-one model, many user-level threads are
mapped to a single kernel thread. Thread management is
performed by the thread library in user space, which is
efficient but has significant limitations.
□ Advantages:
□ Efficient thread creation and management since they are done in
user space.
□ Low overhead for context switching between user-level threads.
□ Disadvantages:
□ If one thread makes a blocking system call, the entire process is
blocked because the kernel thread is blocked.
□ Only one thread can execute at a time, even on multiprocessor
systems.
□ Example:
□ Green threads in early versions of Java.
One-to-One Model
□ In the one-to-one model, each user-level thread maps to a
separate kernel thread. This model provides more
concurrency than the many-to-one model and is used by most
modern operating systems.
□ Advantages:
□ True parallelism on multiprocessor systems because each thread
can run on a different processor.
□ If one thread blocks, other threads can continue to run.
□ Disadvantages:
□ Creating a kernel thread for each user thread incurs a higher
overhead.
□ The number of threads per process may be limited by the operating
system.
□ Example:
□ POSIX threads (Pthreads), Windows threads.
Many-to-Many Model
□ In the many-to-many model, many user-level threads are mapped to
a smaller or equal number of kernel threads. The model allows the
operating system to create sufficient kernel threads to handle
multiple user threads efficiently.
□ Advantages:
□ Greater flexibility and efficiency compared to the other models.
□ User-level threads can be created and managed with less overhead.
□ If one thread blocks, the kernel can schedule another thread.
□ Disadvantages:
□ More complex to implement compared to the many-to-one and
one-to-one models.
□ Performance overhead due to the need to manage the mapping between
user-level and kernel-level threads.
□ Example:
□ Solaris, Windows with the Fibers library.
Support for multithreading
□ Support for multithreading can be provided at both the user level and the kernel
level. Each level has its own mechanisms for creating, managing, and synchronizing
threads, with different advantages and trade-offs.
□ User-Level Threads
□ User-level threads are managed by a user-level library or runtime, not the operating
system kernel. All thread operations, such as creation, scheduling, and synchronization,
are performed in user space.
□ Advantages:
□ Efficiency: User-level thread operations are fast because they do not involve system calls,
which can be slow.
□ Portability: User-level threading libraries can be implemented on any operating system,
as they do not rely on kernel support.
□ Customization: Developers have fine control over the scheduling and management
policies of threads.
□ Disadvantages:
□ Blocking System Calls: If a thread makes a blocking system call, the entire process is
blocked because the kernel is unaware of the user-level threads.
□ No True Parallelism: On multiprocessor systems, user-level threads cannot achieve
true parallelism because the kernel sees only one thread per process.
Sample problem
□ When comparing the performance of a single-threaded
and a multi-threaded file server. The following
assumptions are made. It takes 10ms to get a request,
dispatch it and do the rest of the necessary processing
involved in serving the file, assuming the file is cached in
main memory. If the file is not cached, a disk operation is
needed in which case an additional 50ms is required,
during which the thread sleeps. Assume that for one third
of all requests, the file can be served from the cache.
How many requests per second can the single-threaded
server handle?
Solution:
□ To determine how many requests per second a single-threaded
server can handle, let's break down the processing time for each
request based on whether the file is cached or not.

□ .
Solution:

Conclusion
□ The single-threaded server can handle approximately 23.0823.0823.08 requests per second.
 CS-482 Parallel And Distributed Computing Test 1
Name: _____________________________________ Roll No.: ___________________________________

1. Examine the effectiveness of Instruction-Level Parallelism (ILP), Data-Level

Parallelism (DLP), and Task-Level Parallelism (TLP) in microprocessor
architectures? Analyze their incorporation and mention strengths and weaknesses
of leveraging these architectural paradigms in enhancing computational
performance and addressing scalability challenges?(CLO-1)
2. Analyze the strengths and limitations of employing vector clocks? Mention which
one global state is inconsistent and which is consistent. (CLO-2)
 CS-482 Parallel And Distributed Computing Quiz 2
Name: _____________________________________ Roll No.: ___________________________________
Paper A

Question # 01: Examine array processors and their characteristic features, highlighting their role in
parallel computing and data processing tasks. (CLO-01)
Question # 02: Can you analyze architectural classification schemes in computer architecture, detailing
their significance and various categories? (CLO-02)

Paper B

Question # 01: Examine the principles of vector processing and how they underpin computational
efficiency and performance enhancements? (CLO-01)
Question # 02: How do the principles of pipelining and vector processing contribute to enhancing the
performance of modern processors? Analyze the key concepts and mechanisms involved. (CLO-02)

Paper C

Question # 01: Examine multi-processor architecture and discuss its both types? (CLO-01)
Question # 02: Can you examine the design considerations and efficiency of parallel algorithms? Analyze
their advantages, challenges, and applications in solving complex computational problems. (CLO-02)
 Quiz 3

Paper A

Name: ________________________________ Roll No.: ________________________

Q1. Demonstrate reasons for using external data representation in distributed systems?(CLO-1) (2.5
marks)
Q2. Detect which of the following is a correct statement about Java object serialization? (CLO-2) (1
marks)
A. Serialization always results in smaller object sizes compared to the original.
B. Strings and characters are always serialized using UTF-16.
C. Deserialization assumes prior knowledge of object types.
D. Serialization doesn't support handling object references.
Q3. Explain which remote invocation paradigm extends the conventional procedure call model to
distributed systems? (CLO-2) (1.5 marks)

Paper B
Q1. Define the purpose of marshalling in external data representation? (CLO1) (2.5 marks)
Q2. Explain what does deserialization involve in Java object serialization? (CLO2) (1 marks)
A. Converting a byte stream to an object.
B. Encrypting the serialized object.
C. Serializing the object.
D. Converting an object to a byte stream.
Q3. Explain in request-reply protocols, why is asynchronous communication useful? (CLO-2) (1.5 marks)

Paper C
Q1. Demonstrate what distinguishes Remote Method Invocation (RMI) from Remote Procedure Call
(RPC)? (CLO-1) (2.5 marks)
Q2. Confirm in CORBA's Common Data Representation (CDR), how are primitive values transmitted?
(CLO-2) (1 marks)
A. Always in little-endian order.
B. Always in big-endian order.
C. Depending on the recipient's preference.
D. As ASCII characters.
Q3. Explain in Java, which interface must a class implement to enable its objects to be serialized? (CLO-
2) (1.5 marks)
 assignment 1

Question 1 (a): A client attempts to synchronize with a time server. It records the following round-trip times
and timestamps returned by the server:

Round-trip (ms) Time (hr:min:sec)

25 11:27:14.321

21 11:27:16.589

32 11:27:19.247

i) Which of these times should the client use to set its clock?
ii) Estimate the relative accuracy of the setting with respect to the server's clock.
iii) To what time should the client set its clock, considering the calculated server times and potential
averaging?
iv) If it is known that the minimum message transmission time is 6 milliseconds, recalculate the values
in (b) and (c) above, considering if it changes the answer.
Examine how does the minimum message transmission time influence the accuracy of clock synchronization
and the choice of the reference time? (CLO -1)
Question 2 (a):
Consider the space-time diagram of the distributed system below:-

a) Redraw the above diagram and assign lamport time-stamps to different events
b) Again redraw the above diagram and assign vector time-stamps to different events

Based on the causal relationships between these events using Lamport timestamps and vector timestamps.
Analyze both time- stamping techniques and write pros and cons. (CLO -2)
 Assignment 2
Question 01: Given below is the definition of the class Project and two objects of the same class.
Draw the serialized form of the object. Apply Java Object Serialization procedure. Use 8 byte
version. (CLO-1)

public class Project implements Serializable {

private String projID;
private String title;
private unsigned int duration; // in weeks
private float budget; // in millions
public Project (String a, String b, unsigned int c, float d) {
projID = a; title = b;
duration = c; budget = d; } }
Project p1 = new Project(“E-46”, “Examination System”, 24, 2.5);
Project p2 = new Project(“F-22”, “Fee Collection”, 20, 2.0);

Question 02: Explain how the following code snippet can potentially cause a deadlock, and
propose a solution to prevent this problem. (CLO-2)
#include <iostream>
#include <thread>
#include <chrono>
#include <mutex>
using namespace std;

void deadlock(mutex& m1, mutex& m2) {

m1.lock();
cout << "Thread acquired m1" << endl;
this_thread::sleep_for(chrono::milliseconds(100)); // Simulate some work
m2.lock();
cout << "Thread acquired m2" << endl;
m1.unlock();
m2.unlock();
}

int main() {
mutex mut1, mut2;
thread t1([&] { deadlock(mut1, mut2); });
thread t2([&] { deadlock(mut2, mut1); });

t1.join();
t2.join();

return 0;
}
Mid term A
Q1) [CLO-1] [5 Marks]
Explain Flynn's classification in computer architecture and demonstrate its types with examples?

Q2) [CLO-1] [5 Marks]

How does Amdahl's Law generalize and apply to a scenario where 70% of the workload can benefit from
optimization while the remaining 30% remains unaffected? If we optimize the beneficial portion to be 4 times
faster, how does Amdahl's Law help us demonstrate and understand the overall speedup achieved, taking into
account the optimized fraction?

Q3) [CLO-2] [5 Marks]

Can you define a race condition in concurrent programming and differentiate the conditions crucial for
preventing it? Compare various strategies used to mitigate race conditions and evaluate their effectiveness with
examples.

Q4) [CLO-2] [5 Marks]

Can you elaborate on conflicts of serializabity of transactions, comparing them to other types of transaction
conflicts? Differentiate the types of conflicts of serializabity arise and how they differ from conflicts like
deadlocks or livelocks. Additionally, evaluate the impact of conflicts of serializabity on database performance
and integrity.

Mid term B
Q1) [CLO-1] [5 Marks]
Can you define parallel and distributed computing and demonstrate their applications by highlighting at least
five different aspects where they are utilized? Additionally, estimate how these computing paradigms contribute
to improving computational efficiency and scalability in various domains?
Q2) [CLO-2] [5 Marks]
Explain Peterson’s Solution algorithm, its approach in ensuring mutual exclusion and compare its effectiveness
in preventing race conditions with other methods. Lastly, evaluate the advantages of Peterson’s Solution in
solving the requirements of the critical section problem.
Q3) [CLO-1] [5 Marks]
Define mutex and semaphore in the context of concurrent programming and demonstrate their algorithm?
Additionally, could you explain their types and how they are applied to synchronize access to shared resources?
Lastly, estimate the effectiveness of mutex and semaphore in ensuring thread safety and preventing race
conditions in concurrent systems.
Q4) [CLO-2] [5 Marks]
Can you elaborate on conflicts of serializability of transactions, comparing them to other types of transaction
conflicts? Differentiate the types of conflicts of serializability arise and how they differ from conflicts like
deadlocks or livelocks. Additionally, evaluate the impact of conflicts of serializability on database performance
and integrity.
 NED UNIVERSITY OF ENGINEERING & TECHNOLOGY
FINAL YEAR (BACHELOR OF SCIENCE IN COMPUTER
SCIENCE)
SPRING SEMESTER EXAMINATIONS 2024
BATCH 2020
Dated:23-JUL-2024
Time: 3 Hours
Max.Marks:60
Parallel & Distributed Computing - CS-428
1. Classify the types of parallel memory architecture, defining any two types and explain advantages and
disadvantages associated with each? (CLO-1, 5 marks)
2. Define the following networking architectures based on their key characteristics:
a) Client-Server Architecture
b) Peer-to-Peer Architecture
For each architecture, provide a concise definition and classify it by identifying its primary
features. (CLO-1, 4 marks)
3. Provide a concise definition of array processors and mention its key characteristics? (CLO-1, 4 marks)
4. An industrial process simulation is to be executed on a system in the available time, ta, of 10 min which
includes parallel execution time (on 10 processors) of 7.5 min. It has been estimated that the parallel
portion of the execution time would increase by a factor of 12 if it were executed on a hypothetical
single-processor system. Calculate the parallel speedup. (CLO-1, 5 marks)
5. When comparing the performance of a single-threaded and a multi-threaded file server. The following
assumptions are made. It takes 10ms to get a request, dispatch it and do the rest of the necessary
processing involved in serving the file, assuming the file is cached in main memory. If the file is not
cached, a disk operation is needed in which case an additional 50ms is required, during which the thread
sleeps. Assume that for one third of all requests, the file can be served from the cache. Solve how many
requests per second can the single-threaded server handle? (CLO-1, 6 marks)
6. Examine how the following code snippet can potentially cause a conflict, mention the conflict and
propose a solution to prevent this problem. (CLO-1, 6 marks)
#include <iostream>
#include <thread>
#include <chrono>
#include <mutex>
using namespace std;
void determineConflict(mutex& mA, mutex& mB) {
mA.lock();
cout << "Thread acquired mA" << endl;
this_thread::sleep_for(chrono::milliseconds(100)); // Simulate some work
mB.lock();
cout << "Thread acquired mB" << endl;
mA.unlock();
mB.unlock();
}
int main() {
mutex mutexA, mutexB;
thread t1([&] { determineConflict(mutexA, mutexB); });
thread t2([&] { determineConflict(mutexB, mutexA); });
t1.join();
t2.join();
return 0;
}
7. Analyze the concept of RPC and RMI and explain the term "marshalling" in the context of Remote
Method Invocation (RMI) or Remote Procedure Call (RPC), and illustrate its role in data
transmission between client and server. (CLO-2, 4 marks)
8. Consider distributing a file of 𝐹 = 15 𝐺𝑏𝑖𝑡𝑠 to 𝑁 peers. The server has an upload rate of 𝑢𝑠 =
15 𝑀𝑏𝑝𝑠, and each peer i has a download rate of 𝑑𝑖 = 1.5 𝑀𝑏𝑝𝑠 and an upload rate of 𝑢𝑖 . Complete
the chart giving the minimum distribution time (in hours)
a) For different values of N for client-server distribution.
b) For each of the combinations of N and u for P2P distribution.
After the calculations, analyze the difference between the performance of C-S and P2P systems
for the same number of nodes. (CLO-2, 12 mark)

𝑈𝑖 𝑁 20 200 𝑁 𝑇𝑖𝑚𝑒
600kbps 20
1.5 Mbps 200

9. A master computer is coordinating the internal synchronization of five slave computers using the
Berkley algorithm. At a specific instance, the master polls the slaves 1-5 for their current clock
values. Suppose the slaves respond with the values of 210.3, 212.6, 207.8, 209.5, and 208.2 units
respectively. The master finds its own clock value as 211.0 units. Assuming all clocks are correct:
a) Ignoring round-trip time: Analyze the Berkley algorithm to synchronize all clocks in the
system.
b) Considering round-trip times: Given the round-trip times for the slaves are 6.5, 5.8, 7.2, 5.3,
and 6.8 units respectively, Analyze the Berkley algorithm for internal clock synchronization.

Solve the calculations for both parts (a) and (b), showing the steps involved in adjusting the slave
clocks based on the master's clock value. (CLO-2, 10 marks)
10. Outline the failure model associated with Request-Reply Protocols. (CLO-2, 4 marks)
Model solution
Question 1 (a): A client attempts to synchronize with a time server. It records the following round-trip times
and timestamps returned by the server:

Round-trip (ms) Time (hr:min:sec)

25 11:27:14.321

21 11:27:16.589

32 11:27:19.247

i) Which of these times should the client use to set its clock?
ii) Estimate the relative accuracy of the setting with respect to the server's clock.
iii) To what time should the client set its clock, considering the calculated server times and potential
averaging?
iv) If it is known that the minimum message transmission time is 6 milliseconds, recalculate the values
in (b) and (c) above, considering if it changes the answer.
Examine how does the minimum message transmission time influence the accuracy of clock synchronization
and the choice of the reference time? (CLO -1)
Question 2 (a):
Consider the space-time diagram of the distributed system below:-

a) Redraw the above diagram and assign lamport time-stamps to different events
b) Again redraw the above diagram and assign vector time-stamps to different events

Based on the causal relationships between these events using Lamport timestamps and vector timestamps.
Analyze both time- stamping techniques and write pros and cons. (CLO -2)
 Assignment 2
Question 01: Given below is the definition of the class Project and two objects of the same class.
Draw the serialized form of the object. Apply Java Object Serialization procedure. Use 8 byte
version. (CLO-1)

public class Project implements Serializable {

private String projID;
private String title;
private unsigned int duration; // in weeks
private float budget; // in millions
public Project (String a, String b, unsigned int c, float d) {
projID = a; title = b;
duration = c; budget = d; } }
Project p1 = new Project(“E-46”, “Examination System”, 24, 2.5);
Project p2 = new Project(“F-22”, “Fee Collection”, 20, 2.0);

Question 02: Explain how the following code snippet can potentially cause a deadlock, and
propose a solution to prevent this problem. (CLO-2)
#include <iostream>
#include <thread>
#include <chrono>
#include <mutex>
using namespace std;

void deadlock(mutex& m1, mutex& m2) {

m1.lock();
cout << "Thread acquired m1" << endl;
this_thread::sleep_for(chrono::milliseconds(100)); // Simulate some work
m2.lock();
cout << "Thread acquired m2" << endl;
m1.unlock();
m2.unlock();
}

int main() {
mutex mut1, mut2;
thread t1([&] { deadlock(mut1, mut2); });
thread t2([&] { deadlock(mut2, mut1); });

t1.join();
t2.join();

return 0;
}
1. Classify the types of parallel memory architecture, defining any two types and explain one advantage and
disadvantage associated with each? (CLO-1) 5 marks
Shared Memory: 0.5+
 Multiple processors can operate independently, but share the same memory resources 1+
 Changes in a memory location caused by one 0.5+
 CPU are visible to all processors 0.5
Advantages: =
 Global address space provides a user-friendly programming perspective to memory 2.5
 Fast and uniform data sharing due to proximity of memory to CPUs
Disadvantages:
 Lack of scalability between memory and CPUs. Adding more CPUs increases traffic on the
shared memory CPU path
 Programmer responsibility for “correct” access to global memory
Distributed Memory: 0.5+
 Requires a communication network to connect interprocessor memory 1+
 Processors have their own local memory. Changes made by one CPU have no effect on others 0.5+
 Requires communication to exchange data among processors 0.5
Advantages: =
 Memory is scalable with the number of CPUs 2.5
 Each CPU can rapidly access its own memory without overhead incurred with trying to maintain
global cache coherency
Disadvantages:
 Programmer is responsible for many of the details associated with data communication between
processors
 It is usually difficult to map existing data structures to this memory organization, based on global
memory
Hybrid Distributed Shared Memory:
 The largest and fastest computers in the world today employ both shared and distributed memory
architectures.
Advantages and Disadvantages:

 Increased scalability is an important advantage

 Increased programming complexity is a major disadvantage

2. Define the following networking architectures based on their key characteristics:

a) Client-Server Architecture
b) Peer-to-Peer Architecture
For each architecture, provide a concise definition and classify it by identifying its primary features. (CLO-1) 4 marks

Peer-to-Peer Architecture: Peer-to-Peer (P2P) Architecture is a decentralized network 1

architecture where participants (peers) share resources or services directly with each other without the
need for central coordination or control.
Characteristics: Decentralized, with peers sharing resources directly, promoting robustness and 0.5
efficient resource utilization. each
Client-Server Architecture: Client-Server Architecture is a network architecture where tasks or 1
workloads are divided between servers, which are specialized computers that host services or
resources, and clients, which are end-user devices accessing those services.
Characteristics: Centralized, with dedicated servers providing services to clients, simplifying 0.5
management and scalability. each
3. Provide a concise definition of array processors and mention its key characteristics? (CLO-1) 4 marks
Definition: Array processors are specialized computing units designed for efficiently processing arrays or 1
matrices of data. These processors excel at performing parallel computations on large sets of data elements
simultaneously.
 Parallel Processing: Simultaneous processing of array elements using specialized instructions. 0.5
 Specialized Instructions: Optimized for efficient array operations. 0.5
 Vector and Matrix Operations: Excel at concurrent operations on vectors and matrices. 0.5
 Memory Architecture: Supports rapid access with vector registers or specialized memory banks. 0.5
 High Throughput: Provides high-speed processing for regular data structures. 0.5
 Scientific and Engineering Applications: Used extensively for simulations and signal 0.5
processing.
 Data Parallelism: Emphasizes concurrent operations on multiple data elements.

4. Demonstrate the failure model associated with Request-Reply Protocols. (CLO-1) 4 marks
I. Timeouts 1
II. Discarding duplicate request messages 1
III. Lost reply messages 1
IV. History 1

5. Consider distributing a file of to peers. The server has an upload rate of , and each
peer i has a download rate of and an upload rate of . Complete the chart giving the minimum
distribution time (in hours)
a) For different values of N for client-server distribution.
20 200

20 600kbps

200 1.5 Mbps

b) For each of the combinations of N and u for P2P distribution.

After the calculations, observe the difference between the performance of C-S and P2P systems for the same number
of nodes. (CLO-1) 6 marks

6. A master computer is coordinating the internal synchronization of five slave computers using the Berkley algorithm.
At a specific instance, the master polls the slaves 1-5 for their current clock values. Suppose the slaves respond with
the values of 210.3, 212.6, 207.8, 209.5, and 208.2 units respectively. The master finds its own clock value as 211.0
units. Assuming all clocks are correct:
a) Ignoring round-trip time: Apply the Berkley algorithm to synchronize all clocks in the system.
b) Considering round-trip times: Given the round-trip times for the slaves are 6.5, 5.8, 7.2, 5.3, and 6.8 units respectively,
apply the Berkley algorithm for internal clock synchronization.
Solve the calculations for both parts (a) and (b), showing the steps involved in adjusting the slave clocks based on the
master's clock value. (CLO-1) 10 marks
7. When comparing the performance of a single-threaded and a multi-threaded file server. The following assumptions
are made. It takes 10ms to get a request, dispatch it and do the rest of the necessary processing involved in serving
the file, assuming the file is cached in main memory. If the file is not cached, a disk operation is needed in which case
an additional 50ms is required, during which the thread sleeps. Assume that for one third of all requests, the file can
be served from the cache. How many requests per second can the single-threaded server handle? (CLO-1) 6 marks
8. Explain how the following code snippet can potentially cause a conflict, mention the conflict and propose a solution
to prevent this problem. (CLO-1) (6 marks)
#include <iostraem>
#include <thread>
#include <chrono>
#include <mutex>
using namespace std;

void deadlock(mutex& m1, mutex& m2) {

m1.lock();
cout << "Thread acquired m1" << endl;
this_thread::sleep_for(chrono::milliseconds(100)); // Simulate some work
m2.lock();
cout << "Thread acquired m2" << endl;
m1.unlock();
m2.unlock();
}

int main() {
mutex mut1, mut2;
thread t1([&] { deadlock(mut1, mut2); });
thread t2([&] { deadlock(mut2, mut1); });

t1.join();
t2.join();

return 0;

9. Demonstrate the concept of RPC and RMI and explain the term "marshalling" in the context of Remote Method
Invocation (RMI) or Remote Procedure Call (RPC), and illustrate its role in data transmission between client and
server. (CLO-1) (5 marks)

RPC allows a program to execute a procedure (or function) in another address space (commonly on another 1
computer) as if it were a local procedure call, hiding the details of the network communication.
RMI is a Java-specific implementation of RPC that allows an object to invoke methods on an object running in 1
another JVM (Java Virtual Machine). RMI extends the concept of Java interfaces to invoke methods remotely,
making distributed computing simpler within the Java ecosystem.
Marshalling (also known as serialization) is the process of converting the memory representation of an object 1
or data structure into a format suitable for storage or transmission over a network.
Role in Data Transmission: 2

1. Object Serialization: In Java RMI, when an object is passed as a parameter or returned value
in a remote method call, it needs to be serialized (marshalled) into a byte stream before being
transmitted over the network.
2. Network Transmission: The marshalled data, now in the form of a byte stream, is transmitted
from the client to the server (or vice versa).
3. Object Deserialization: On the receiving end, the byte stream is deserialized (unmarshalled)
back into its original object form, allowing the server to process the method call with the
correct parameters.
10. An industrial process simulation is to be executed on a system in the available time, ta, of 10 min which includes
parallel execution time (on 10 processors) of 7.5 min. It has been estimated that the parallel portion of the execution
time would increase by a factor of 12 if it were executed on a hypothetical single-processor system. Calculate the
parallel speedup. (CLO-1) (5 marks)