Distributed Computing
Introduction
Dr. Barsha Mitra
CSIS Dept, BITS Pilani, Hyderabad Campus
BITS Pilani
Hyderabad Campus
1 1
Definition of Distributed Systems
• Collection of independent entities cooperating to collectively
solve a task
• Autonomous processors communicating over a
communication network
Course ID: SS ZG526, Title: Distributed Computing 2 BITS Pilani, Hyderabad Campus
Features of Distributed System
•No common physical clock
•Introduces “distribution” element into the system
•Gives rise to asynchrony amongst processors
•No shared memory
•Absence of common physical clock
•Requires message passing for communication
•Abstraction of common address space may be
provided via distributed shared memory abstraction
Course ID: SS ZG526, Title: Distributed Computing 3 BITS Pilani, Hyderabad Campus
Features of Distributed System
•Geographical seperation
•Processors are geographically wide apart
•Processors may reside on a WAN
•Processors on a LAN – small distributed system
•Network/Cluster of Workstations (NOW/COW)
configuration – processors connected on a LAN
•Popularity of NOW – emergence of low-cost high-
speed off-the-shelf processors
•Google search engine - based on NOW architecture
Course ID: SS ZG526, Title: Distributed Computing 4 BITS Pilani, Hyderabad Campus
Features of Distributed System
•Autonomy and Heterogeneity
•Loosely coupled processors
•Different processor speeds
•Different operating systems
•Processors are not part of a dedicated system
•Cooperate with one another
•May offer services
•May solve a problem jointly
Course ID: SS ZG526, Title: Distributed Computing 5 BITS Pilani, Hyderabad Campus
Distributed System Model
Course ID: SS ZG526, Title: Distributed Computing 6 BITS Pilani, Hyderabad Campus
Relation between Software Components
Interaction of the software components
Course ID: SS ZG526, Title: Distributed Computing 7 BITS Pilani, Hyderabad Campus
Relation between Software Components
•Middleware –
•distributed software
•drives the distributed system
•provides transparency of heterogeneity at platform level
•Middleware standards:
•Common Object Request Broker Architecture (CORBA)
•Remote Procedure Call (RPC) mechanism
•Distributed Component Object Model (DCOM)
•Remote Method Invocation (RMI)
•Message Passing Interface (MPI)
Course ID: SS ZG526, Title: Distributed Computing 8 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Inherently distributed computation
•applications like funds transfer
•reaching a consensus among geographically distributed
parties
Course ID: SS ZG526, Title: Distributed Computing 9 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Resource sharing
•Peripherals, data sets in databases, libraries, data
•Resources can’t be replicated at all sites
•Should not be present at a single site creates single point
of failure
•Segregate resources at multiple sites
•Increases reliability
Course ID: SS ZG526, Title: Distributed Computing 10 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Access to remote resources
•Sensitive data/resources may remain centralized
•Eg., payroll data, supercomputers
•Such resources need to be remotely accessed
•Encouraged development of distributed protocols and
middleware
Course ID: SS ZG526, Title: Distributed Computing 11 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Increased performance/cost ratio
•Due to resource sharing and accessing geographically remote
data & resources
•Task can be partitioned across different computers
•Better performance/cost ratio than parallel machines
•Particularly true for NOW configuration
Course ID: SS ZG526, Title: Distributed Computing 12 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Reliability
•Resource replication
•No single point of failure
•All sites are not likely to crash/malfunction at the same time
•Requirements of reliability
•Availability
•Integrity
•Fault tolerance
Course ID: SS ZG526, Title: Distributed Computing 13 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Scalability
•Processors are connected by wide-area network
•New processors may be easily added
•Addition of more processors does not create bottleneck for the
communication network
Course ID: SS ZG526, Title: Distributed Computing 14 BITS Pilani, Hyderabad Campus
Motivation for Distributed System
• Modularity and Incremental Expandability
•Heterogenous processors may be added
•Does not affect performance if processors are running the same
middleware algorithm
•Existing processors can also be replaced
Course ID: SS ZG526, Title: Distributed Computing 15 BITS Pilani, Hyderabad Campus
Parallel Systems
• Multiprocessor systems (direct access to shared memory,
UMA model)
• E.g., Omega, Butterfly, Clos, shuffle-exchange networks
• Multicomputer parallel systems (no direct access to shared
memory, NUMA model)
• E.g., NYU Ultracomputer, CM* Connection Machine, IBM
Blue gene
• Array processors (collocated, tightly coupled, common system
clock)
• E.g., DSP applications
Course ID: SS ZG526, Title: Distributed Computing 16 BITS Pilani, Hyderabad Campus
UMA Model
• Direct access to shared memory
• Forms a common address space
• Access latency - waiting time to complete an access to any
memory location from any processor
• Access latency is same for all processors
• Processors
• Remain in close proximity
• Connected by an interconnection network
Course ID: SS ZG526, Title: Distributed Computing 17 BITS Pilani, Hyderabad Campus
UMA Model
M – Memory, P -
Processor
Course ID: SS ZG526, Title: Distributed Computing 18 BITS Pilani, Hyderabad Campus
UMA Model
• Interprocess communication occurs through
• read and write operations on shared memory
• message passing primitives provided by MPI
• Processors are of same type
• Remain within same physical box/container with shared memory
• Processors run the same operating system
• Hardware & software are very tightly coupled
• Interconnection network to access memory may be
• Bus
• Multistage switch (with regular & symmetric design) – greater
efficiency
Course ID: SS ZG526, Title: Distributed Computing 19 BITS Pilani, Hyderabad Campus
Omega, Butterfly Interconnections
Interconnection networks for shared memory multiprocessor systems
(a) Omega network (b) Buttery network
Course ID: SS ZG526, Title: Distributed Computing 20 BITS Pilani, Hyderabad Campus
Omega, Butterfly Interconnections
• Multi-stage networks
• Formed by 2 x 2 switching elements
• Data can be sent on any one of the input wires
• Data is switched to upper or lower output wire in a single step
• Collision occurs when data from both input wires is routed to same
output wire at the same step
• Buffering & interconnection design techniques address collisions
Course ID: SS ZG526, Title: Distributed Computing 21 BITS Pilani, Hyderabad Campus
Omega, Butterfly Interconnections
• n-input and n-output network uses:
•log(n) stages
•log(n) bits for addressing
• Routing at stage k uses the kth bit
• Multi-stage networks
• can be constructed recursively or iteratively
•interconnection pattern between any two stages can be expressed
using an iterative or a recursive generating function
Course ID: SS ZG526, Title: Distributed Computing 22 BITS Pilani, Hyderabad Campus
Omega Network
• n processors, n memory banks
• log2(n) stages
• (n/2)log2(n) switching elements of size 2x2
• Interconnection function: Output i of a stage is connected to input j of next
stage:
•j = 2i, for 0 ≤ i ≤ n/2 – 1
= 2i + 1 – n, for n/2 ≤ i ≤ n – 1
•left-rotation operation on the binary representation of i to get j
• upper (lower) input lines for a switch in stage k comes in sequential order
from the upper (lower) half of the switches in stage k - 1
Course ID: SS ZG526, Title: Distributed Computing 23 BITS Pilani, Hyderabad Campus
Omega Network
• For any stage
• for 0 ≤ i ≤ 3, output i is
connected to input 2i of next
stage
• for 4 ≤ i ≤ 7, output i is
connected to input 2i + 1 – n of
next stage
3-stage Omega network (n = 8, M = 4)
Course ID: SS ZG526, Title: Distributed Computing 24 BITS Pilani, Hyderabad Campus
Omega Network
• Routing function: from input line i to output line j
• considers j and stage number s, s ϵ [0, log2(n) – 1]
• in a stage s switch
• if (s + 1)th MSB of j is 0, then route on upper wire
• else [i.e., (s + 1)th MSB of j is 1], route on lower wire
Course ID: SS ZG526, Title: Distributed Computing 25 BITS Pilani, Hyderabad Campus
Butterfly Network
• Interconnection function depends on n and stage number s
• No. of switches per stage M = n/2
• Switch is denoted as áx, sñ , x ϵ [0, M – 1], s ϵ [0, log2(n) -1]
• An edge from switch áx, sñ to switch áy, s + 1ñ exists if
• x = y or,
• (s + 1)th MSB of (x XOR y) is 1
• for stage s, rule is applied for M/2s switches
Course ID: SS ZG526, Title: Distributed Computing 26 BITS Pilani, Hyderabad Campus
Butterfly Network
• n = 8, M = 4, s = 0,…,2
• interconnection pattern is between s = 0
and s = 1 and between s = 1 and s = 2
• x = 0 to 3 (2-bit string)
• for s = 0, first output line from switch 00
goes to the input line of switch 00 of stage
s =1
• for s = 0, second output line of switch 00
goes to input line of switch 10 of stage s =
1
• x = 01 has one output line go to an input
line of switch 11 in stage s = 1
Course ID: SS ZG526, Title: Distributed Computing 27 BITS Pilani, Hyderabad Campus
Butterfly Network
• for stage s = 1 connecting to stage s = 2,
apply the rules considering M/21 = M/2
switches
• i.e., build two butterflies of size M/2 – the
“upper half” and the “lower half”
switches
• recursion terminates for M/2s = 1, when
there is a single switch
Course ID: SS ZG526, Title: Distributed Computing 28 BITS Pilani, Hyderabad Campus
Butterfly Network
• In a stage s switch,
• if the (s+1)th MSB of j is 0, data is routed to the upper
output wire
• otherwise it is routed to the lower output wire
• Note:
• for Butterfly and Omega networks, paths from different
inputs to any one output constitute a spanning tree
• collisions occur when data is destined to the same output
line
Course ID: SS ZG526, Title: Distributed Computing 29 BITS Pilani, Hyderabad Campus
Multicomputer Parallel Systems
• multiple processors do not have direct access to shared memory
• memory of multiple processors may or may not form a common address
space
• usually do not have a common clock
• processors are in close physical proximity
• very tightly coupled (homogenous hardware and software)
• connected by an interconnection network
• communicate either via a common address space or via message-
passing
• eg., NYU Ultracomputer, Sequent shared memory machines, CM*
Connection machine
Course ID: SS ZG526, Title: Distributed Computing 30 BITS Pilani, Hyderabad Campus
NUMA Model
• NUMA - non-uniform memory access architecture
• multicomputer system
• has a common address space
• latency to access various shared memory locations from the
different processors varies
Course ID: SS ZG526, Title: Distributed Computing 31 BITS Pilani, Hyderabad Campus
NUMA Model
Course ID: SS ZG526, Title: Distributed Computing 32 BITS Pilani, Hyderabad Campus
Multicomputer Parallel Systems
• 2-D Mesh with wraparound (a.k.a. torus)
• k x k mesh contains k2 processors
processor + memory
Course ID: SS ZG526, Title: Distributed Computing 33 BITS Pilani, Hyderabad Campus
Multicomputer Parallel Systems
• 4-D hypercube processor + memory
• k-dimensional hypercube has 2k processor-and-memory units
• Each unit is a node and has a unique k-bit label
• Each dimension is associated with a bit position in the label
Course ID: SS ZG526, Title: Distributed Computing 34 BITS Pilani, Hyderabad Campus
Multicomputer Parallel Systems
processor + memory
• labels of any two adjacent nodes differ only in the kth bit position for
dimension k
• shortest path between any two processors is the Hamming distance
between the processor labels
• Hamming distance <= k
Course ID: SS ZG526, Title: Distributed Computing 35 BITS Pilani, Hyderabad Campus
Multicomputer Parallel Systems
• routing in the hypercube is done hop-by-hop
• at any hop, message can be sent along any dimension corresponding to
the bit position in which the current node’s address and the destination
address differ
• construct 4D hypercube:
• connect the corresponding edges of two 3D hypercubes along the 4th dimension
• prepend a “0” to labels of left 3D hypercube
• prepend a “1” to labels of the right 3D hypercube
• can be extended to construct hypercubes of higher dimensions
• multiple routes exist between any pair of nodes provides fault
tolerance and congestion control mechanism
Course ID: SS ZG526, Title: Distributed Computing 36 BITS Pilani, Hyderabad Campus
Benefits of Parallel Systems
• provide higher throughput by dividing the computational
workload among multiple processors
• higher speedup for tasks which
• can be partitioned into subtasks
• involve high computation
• require little communication for synchronization
• Example tasks:
• vector, array, and matrix computations
• searching through large state spaces
Course ID: SS ZG526, Title: Distributed Computing 37 BITS Pilani, Hyderabad Campus
Downside of Parallel Systems
• not economically viable
• lesser number of applications that can attain high speedups
• not cost-effective
• require special compiler and other system support for maximum
throughput
• relatively inexpensive off-the-shelf networked PCs are
competitors
• economy of scale
• high processing power
Course ID: SS ZG526, Title: Distributed Computing 38 BITS Pilani, Hyderabad Campus
Flynn's Taxonomy
• four processing modes
• based on
• whether the processors execute the same or different
instruction streams at the same time
• whether the processors process the same (identical) data at
the same time
Note:
Instruction Stream – sequence of instructions
flowing from memory to control unit
Data Stream – sequence of transmitted/ received
information Course ID: SS ZG526, Title: Distributed Computing 39 BITS Pilani, Hyderabad Campus
Flynn's Taxonomy
• SISD: Single Instruction Stream Single Data Stream
• corresponds to conventional processing
• single CPU and single memory unit connected by a system bus
• SIMD: Single Instruction Stream Multiple Data Stream
• corresponds to processing by multiple homogenous processors
• processors execute in lock-step on different data items
• data sets can be partitioned
• applications involving operations on large arrays and matrices, such as
scientific applications, can use this SIMD mode of operation
• Eg., Illiac-IV, MPP, CM2, MasPar MP-1 earliest parallel computers
• Vector processors, array processors, systolic arrays
• MMX units in Intel processors, DSP chips such as the Sharc
Course ID: SS ZG526, Title: Distributed Computing 40 BITS Pilani, Hyderabad Campus
Flynn's Taxonomy
• MISD: Multiple Instruction Stream Single Data Stream
• execution of different operations in parallel on the same data
• specialized mode of operation with limited applications, e.g., visualization
• not very popular commercially
• MIMD: Multiple Instruction Stream Multiple Data Stream
• processors execute different codes on different data
• mode of operation in distributed systems and majority of parallel systems
• no common clock among the system processors
• most general mode
• allow flexibility in partitioning code and data to be processed among the
processors
• eg., Sun Ultra servers, multicomputer PCs, and IBM SP machines
Course ID: SS ZG526, Title: Distributed Computing 41 BITS Pilani, Hyderabad Campus
Flynn's Taxonomy
SIMD, MISD, and MIMD modes SISD mode
Course ID: SS ZG526, Title: Distributed Computing 42 BITS Pilani, Hyderabad Campus
Flynn's Taxonomy
MIMD architectures
• most general
• allow much flexibility in partitioning code and data to be
processed among the processors
• include the classic mode of execution in distributed
systems
Course ID: SS ZG526, Title: Distributed Computing 43 BITS Pilani, Hyderabad Campus
Classes of Multiprocessor/
Multicomputer Operating Systems
• Network Operating System
• run on loosely coupled processors (i.e., heterogenous
and/or geographically distant processors)
• processors also run loosely coupled software (i.e.,
software that is heterogenous)
• application cannot run any significant distributed
function that is not provided by the application layer of
the network protocol stacks on the various processors
Course ID: SS ZG526, Title: Distributed Computing 44 BITS Pilani, Hyderabad Campus
Classes of Multiprocessor/
Multicomputer Operating Systems
• Distributed Operating System
• run on loosely coupled processors
• processors run tightly coupled software (i.e., the middleware software
on the processors is homogenous)
• Multiprocessor Operating System
• run on tightly coupled processors
• processors run tightly coupled software
• can run sophisticated algorithms contained in the tightly coupled
software Course ID: SS ZG526, Title: Distributed Computing 45
BITS Pilani, Hyderabad Campus
Distributed Communication Models
• Remote Procedure Call (RPC)
• Publish/Subscribe Model
• Message Queues
Course ID: SS ZG526, Title: Distributed Computing 46 BITS Pilani, Hyderabad Campus
RPC
•Called procedure code may reside on a remote machine
•Works like a local procedure call
•Steps:
•Sends message across n/w to invoke remote procedure
•Remote procedure completes execution
•Sends back a reply
Course ID: SS ZG526, Title: Distributed Computing 47 BITS Pilani, Hyderabad Campus
RPC
• make remote procedure call appear like local procedure call
• goal is to hide the details of the network communication (namely,
the sending and receiving of messages)
• calling procedure should not be aware that the called procedure is
executing on a different machine
Source: http://web.cs.wpi.edu/~rek/DCS/D04/Communication.pdf
Course ID: SS ZG526, Title: Distributed Computing 48 BITS Pilani, Hyderabad Campus
RPC
• When making an RPC:
• Calling environment is suspended
• Procedure parameters are transferred across the network to
the environment where the procedure is to execute
• Procedure is executed there
• When the procedure finishes, the results are transferred back
to the calling environment
• Execution resumes as if returning from a regular procedure call
Source: http://web.cs.wpi.edu/~rek/DCS/D04/Communication.pdf
Course ID: SS ZG526, Title: Distributed Computing 49 BITS Pilani, Hyderabad Campus
RPC between Client & Server
Source: http://web.cs.wpi.edu/~rek/DCS/D04/Communication.pdf
Course ID: SS ZG526, Title: Distributed Computing 50 BITS Pilani, Hyderabad Campus
RPC between Client & Server
• The client procedure calls a client stub passing parameters in
the normal way
• The client stub marshals the parameters, builds the message,
and calls the local OS
• The client's OS sends the message (using the transport layer)
to the remote OS
• The server’s remote OS gives transport layer message to a
server stub
• The server stub demarshals the parameters and calls the
desired server routine
Source: http://web.cs.wpi.edu/~rek/DCS/D04/Communication.pdf
Course ID: SS ZG526, Title: Distributed Computing 51 BITS Pilani, Hyderabad Campus
RPC between Client & Server
• The server routine does work and returns result to the
server stub via normal procedures
• The server stub marshals the return values into the message
and calls local OS
• The server OS (using the transport layer) sends the message
to the client's OS
• The client's OS gives the message to the client stub
• The client stub demarshals the result, and execution returns
to the client
Source: http://web.cs.wpi.edu/~rek/DCS/D04/Communication.pdf
Course ID: SS ZG526, Title: Distributed Computing 52 BITS Pilani, Hyderabad Campus
Publish/ Subscribe Model
• allows any number of publishers to communicate with any number of
subscribers asynchronously and anonymously
• form of multicasting
• events are produced by publishers
• interested subscribers are notified when the events are available
• information is provided as notifications
• subscribers continue their tasks until they receive notifications
Course ID: SS ZG526, Title: Distributed Computing 53 BITS Pilani, Hyderabad Campus
Publish/ Subscribe Model
Course ID: SS ZG526, Title: Distributed Computing 54 BITS Pilani, Hyderabad Campus
Publish/ Subscribe Model
Advantages:
• Enhanced response time
• Resource utilization
• Increased capacity
• loosely coupled relationship between publishers and subscribers
• scalability
Course ID: SS ZG526, Title: Distributed Computing 55 BITS Pilani, Hyderabad Campus
Message Queue
• Asynchronous communication
• Sender and receiver of the message do not need to interact with the message
queue at the same time
• Messages placed onto the queue are stored until the recipient retrieves them
• Message queues have limits on
• size of data that may be transmitted in a single message
• number of messages that may remain outstanding on the queue
• Producer consumer decoupling
Course ID: SS ZG526, Title: Distributed Computing 56 BITS Pilani, Hyderabad Campus
Message Queue
Course ID: SS ZG526, Title: Distributed Computing 57 BITS Pilani, Hyderabad Campus
Message Queue
• Commercial implementations message queueing software - IBM WebSphere
MQ, Oracle Advanced Queuing
• Java standard - Java Message Service
• Cloud-based message queuing service options - Amazon Simple Queue
Service (SQS)
Course ID: SS ZG526, Title: Distributed Computing 58 BITS Pilani, Hyderabad Campus
Design Issues & Challenges –
System Perspective
• Communication mechanisms:
• designing appropriate mechanisms for communication among the processes in the
network
• E.g., Remote Procedure Call (RPC), Remote Object Invocation (ROI), message-oriented
communication, stream-oriented communication
• Processes:
• code migration
• process/thread management at clients and servers
• design of software and mobile agents
• Naming:
• easy to use identifiers needed to locate resources and processes transparently and
scalably
Course ID: SS ZG526, Title: Distributed Computing 59 BITS Pilani, Hyderabad Campus
Design Issues & Challenges –
System Perspective
• Synchronization:
•mutual exclusion
•leader election
•synchronizing physical clocks
•devising logical clocks that capture the essence of the passage of
time
•global state recording algorithms
• Data storage and access
•schemes for data storage, search, and lookup should be fast and
scalable across network
•need to revisit traditional file system design
Course ID: SS ZG526, Title: Distributed Computing 60 BITS Pilani, Hyderabad Campus
Design Issues & Challenges –
System Perspective
• Consistency and replication
• replication required for fast access, scalability, avoiding bottlenecks
• require consistency management among replicas
• Fault-tolerance:
• correct and efficient operation despite link, node, process failures
• mechanisms for fault-tolerance process resilience, reliable
communication, distributed commit, checkpointing and recovery,
agreement and consensus, failure detection, self-stabilization
• Security:
• secure channels, access control, key management (key generation and
key distribution), authorization, secure group management
Course ID: SS ZG526, Title: Distributed Computing 61 BITS Pilani, Hyderabad Campus
Design Issues & Challenges –
System Perspective
• API and transparency:
• required for ease of use and wider adoption of distributed systems services
• Transparency hiding the implementation policies from the user
• Access transparency hides differences in data representation on different systems
and provides uniform operations to access system resources
• Location transparency makes locations of resources transparent to users
• Migration transparency relocating resources without changing names
• Relocation transparency ability to relocate the resources as they are being
accessed
• Replication transparency does not let the user become aware of any replication
• Concurrency transparency masking the concurrent use of shared resources for
user
• Failure transparency reliability and fault-tolerance 62 BITS Pilani, Hyderabad Campus
Design Issues & Challenges –
System Perspective
• Scalability and modularity:
• algorithms, data (objects), and services must be distributed
• various techniques such as replication, caching and cache management, and
asynchronous processing help to achieve scalability
Course ID: SS ZG526, Title: Distributed Computing 63 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Designing useful execution models and frameworks: to reason with and
design correct distributed programs
• Interleaving model
• Partial order model
• Input/Output automata model
• Temporal Logic of Actions (TLA)
• Dynamic distributed graph algorithms and routing algorithms
• System topology: distributed graph, with only local neighborhood knowledge
• Distributed graph algorithms: building blocks for group communication, data
dissemination, object location
• Algorithms need to deal with dynamically changing graphs, model varying link loads
• Algorithm efficiency: impacts resource consumption, latency, traffic, and congestion
Course ID: SS ZG526, Title: Distributed Computing 64 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Time and global state
• 3D space, 1D time
• providing accurate physical time and logical time
• Logical time
• captures inter-process dependencies and tracks relative time progression
• eliminates the overheads of providing physical time for applications where
physical time is not required
• Global state observation:
• involves time dimension for consistent observation
• not possible for any one process to directly observe a meaningful global
state across all the processes, without using extra state-gathering effort
• Concurrency measures: concurrency depends on program logic, execution
speeds within logical threads, communication speeds
Course ID: SS ZG526, Title: Distributed Computing 65 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Synchronization/coordination mechanisms
• Physical clock synchronization: hardware drift needs correction
• Leader election: select a distinguished process, due to inherent asymmetry
• Mutual exclusion: coordinate access to critical resources
• Distributed deadlock detection and resolution: need to observe global state,
avoid duplicate detection, unnecessary aborts
• Termination detection: requires cooperation among the processes to detect the
specific global state of stability
• Garbage collection: Reclaim objects no longer pointed to by any process,
requires coordination among processes
Course ID: SS ZG526, Title: Distributed Computing 66 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Group communication, multicast, and ordered message delivery
• Group: processes sharing a context and collaborating on a common task
• Algorithms for efficient group communication & group management
• Multiple processes can join & leave groups
• Concurrent sends: semantics of order of delivery to different recipients
• Monitoring distributed events and predicates
• Predicate: specify condition on global system state
• Used for debugging, environmental sensing, industrial process control, analyzing
event streams
• Distributed program design and verification tool
• well designed and verifiably correct programs reduce the overhead of software
design, debugging, andCourse
engineering
ID: SS ZG526, Title: Distributed Computing 67 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Debugging distributed programs
• Harder than traditional debugging
• Reason: concurrency in actions and uncertainty due to the large number of
possible executions defined by the interleaved concurrent actions
• Data replication, consistency models, and caching
• For fast, scalable access
• Coordinate replica updates
• Optimize replica placement
• World Wide Web design: caching, searching, scheduling
• Global scale distributed system; end-users
• Read-intensive operations; prefetching over caching
• Object search and navigation are resource-intensive
Course ID: SS ZG526, Title: Distributed Computing 68 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Distributed shared memory abstraction
• Wait-free algorithm design: process completes execution, irrespective of actions
of other processes, i.e., n - 1 fault-resilience in n process system
• Mutual exclusion
• Bakery algorithm, semaphores, based on atomic hardware primitives, fast
algorithms when contention-free access
• Register constructions
• Revisit assumptions about memory access
• What behavior under concurrent unrestricted access to memory?
• Foundation for future architectures, decoupled with technology
(semiconductor, biocomputing, quantum)
• Consistency models: consistency among replicas
• Coherence versus cost of implementation
Course ID: SS ZG526, Title: Distributed Computing 69 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Reliable and fault-tolerant distributed systems
• Consensus algorithms: processes reach agreement in spite of faults (under
various fault models)
• Replication and replica management
• Voting and quorum systems
• Distributed databases, commit: ACID properties
• Self-stabilizing systems: "illegal" system state changes to "legal" state; requires
built-in redundancy
• Checkpointing and recovery algorithms: roll back and restart from earlier
"saved" state
• Failure detectors:
• Difficult to distinguish a "slow" process/message from a failed process/ never
sent message
• Algorithms that "suspect" a process as having failed and converge on a
determination of its up/down status
Course ID: SS ZG526, Title: Distributed Computing 70 BITS Pilani, Hyderabad Campus
Algorithmic Challenges
• Load balancing: to reduce latency, increase throughput, done
dynamically
• Computation migration: relocate processes to redistribute workload
• Data migration: move data, based on access patterns
• Distributed scheduling: across processors
• Real-time scheduling: difficult without global view, network delays make
task harder
• Performance modeling and analysis: Network latency to access
resources must be reduced
• Metrics: theoretical measures for algorithms, practical measures for systems
• Measurement methodologies and tools
Course ID: SS ZG526, Title: Distributed Computing 71 BITS Pilani, Hyderabad Campus
References
• Ajay D. Kshemkalyani, and Mukesh Singhal, Chapter 1, “Distributed
Computing: Principles, Algorithms, and Systems”, Cambridge
University Press, 2008.
• http://www.ois.com/Products/what-is-corba.html
• http://wiki.c2.com/?PublishSubscribeModel
• https://www.slideshare.net/ishraqabd/publish-subscribe-model-
overview-13368808/5
• https://www.cloudamqp.com/blog/2014-12-03-what-is-message-
queuing.html
• https://en.wikipedia.org/wiki/Message_queue
Course ID: SS ZG526, Title: Distributed Computing 72 BITS Pilani, Hyderabad Campus
Thank You
Course ID: SS ZG526, Title: Distributed Computing 73 BITS Pilani, Hyderabad Campus
