Distributed Systems,

cs5223
Lecture 02
Seif Haridi
Department of Computer Science,
NUS

[email protected]

2002-08-15 S. Haridi, CS5223, Lecture 02 1

 Communication

2002-08-15 S. Haridi, CS5223, Lecture 02 2

Layered Protocols
 Low Level Layers
 Transport Layer

 Application Layer

 Middleware Layer

2002-08-15 S. Haridi, CS5223, Lecture 02 3

Basic Networking Level

2002-08-15 S. Haridi, CS5223, Lecture 02 4

Lower Levels Layer
 Physical layer
 contains the specification and implementation of bits, and
their transmission between sender and receiver
 Data link layer
 prescribes the transmission of a series of bits into a frame
to allow for error and flow control
 Network layer
 describes how packets in a network of computers are to be
routed
 Observation: for many distributed systems, the
lowest level interface is that of the network layer

2002-08-15 S. Haridi, CS5223, Lecture 02 5

Transport Layer
 The transport layer provides the actual
communication facilities for most distributed
systems

 Standard Internet protocols

 TCP: connection-oriented, reliable, stream-
oriented communication
 UDP: unreliable (best-effort) datagram
communication
 IP multicasting is generally considered a
standard available service
2002-08-15 S. Haridi, CS5223, Lecture 02 6
 Client Server TCP

 TCP for transactions (T/TCP): A transport

protocol aimed to support client–server
interaction

2002-08-15 S. Haridi, CS5223, Lecture 02 7

Client Server TCP

2002-08-15 S. Haridi, CS5223, Lecture 02 8

Application Level Protocols
 Many application protocols are directly
implemented on top of transport protocols,
doing a lot of application-independent work
 FTP
 Transfer Protocol: FTP
 Encoding: 7-bit text
 Naming: Host + path
 Security: Username + Password

2002-08-15 S. Haridi, CS5223, Lecture 02 9

Application Level Protocols
 Many application protocols are directly
implemented on top of transport protocols,
doing a lot of application-independent work
 WWW
 Transfer Protocol: HTTP
 Encoding: 8-bit + content type
 Naming: URL
 Security: Username + Password

2002-08-15 S. Haridi, CS5223, Lecture 02 10

Middleware Layer
 Middleware is invented to provide common
services and protocols that can be used by
many different applications
 A rich set of communication protocols, but
which allow different applications to
communicate
 Marshaling and unmarshaling of data,
necessary for integrated systems
 Naming protocols, so that different
applications can easily share resources
2002-08-15 S. Haridi, CS5223, Lecture 02 11
Middleware Layer
 Security protocols, to allow different
applications to communicate in a secure way
 Scaling mechanisms, such as support for
replication and caching
 What remains are truly application-specific
protocols
 Like what?

2002-08-15 S. Haridi, CS5223, Lecture 02 12

Remote Procedure call (RPC)
 Basic RPC operation

 Parameter passing

 Variations

2002-08-15 S. Haridi, CS5223, Lecture 02 13

Remote Procedure call (RPC)
 Application developers are familiar with
simple procedure model
 Well-engineered procedures operate in
isolation (black box)
 There is no fundamental reason not to
execute procedures on separate machine
 Conclusion: network communication
between caller & callee can be hidden by
using procedure-call mechanism
2002-08-15 S. Haridi, CS5223, Lecture 02 14
Remote Procedure call (RPC)

2002-08-15 S. Haridi, CS5223, Lecture 02 15

RPC Implementation I
 Local procedure call
read(int fd, char* buf, int nbytes)
2. Push parameter values of the procedure on
a stack
3. Call procedure
4. Use stack for local variables
5. Pop results (in parameters)

2002-08-15 S. Haridi, CS5223, Lecture 02 16

RPC Implementation II

2002-08-15 S. Haridi, CS5223, Lecture 02 17

RPC Implementation III

 Principle: “communication” with local

procedure is handled by copying data to/from
the stack (with a few exceptions)

2002-08-15 S. Haridi, CS5223, Lecture 02 18

RPC Implementation IV

2002-08-15 S. Haridi, CS5223, Lecture 02 19

RPC Parameter Passing
 Parameter marshaling: There’s more than just
wrapping parameters into a message
 Client and server machines may have different data
representations (think of byte ordering)
 Wrapping a parameter means transforming a value into a
sequence of bytes

 Client and server have to agree on the same encoding:

 How are basic data values represented (integers, floats,
characters)
 How are complex data values represented (arrays, unions)

2002-08-15 S. Haridi, CS5223, Lecture 02 20

RPC Parameter Passing
 Client and server need to properly interpret
messages, transforming them into machine-
dependent representations
 RPC assumes copy in/copy out semantics:
while procedure is executed, nothing can be
assumed about parameter values (only Ada
supports this model)

2002-08-15 S. Haridi, CS5223, Lecture 02 21

RPC Parameter Passing
 RPC assumes all data that is to be operated
on is passed by parameters. Excludes
passing references to (global) data
 Full access transparency cannot be realized
 If we introduce a remote reference
mechanism, access transparency can be
enhanced
 Remote reference offers unified access to remote
data
 Remote references can be passed as parameter
in RPCs
2002-08-15 S. Haridi, CS5223, Lecture 02 22
Asynchronous RPCs
 Try to get rid of the strict request-reply behavior,
but let the client continue without waiting for an
answer from the server

2002-08-15 S. Haridi, CS5223, Lecture 02 23

Deferred synchronous RPC:

2002-08-15 S. Haridi, CS5223, Lecture 02 24

RPC in Practice DCE
 Essence: Let the developer concentrate on
only the client- and server-specific code; let
the RPC system (generators and libraries) do
the rest

2002-08-15 S. Haridi, CS5223, Lecture 02 25

DCE Development Cycle

2002-08-15 S. Haridi, CS5223, Lecture 02 26

 Client-to-Server Binding (DCE)
 Issues: (1) Client must
locate server machine,
and (2) locate the server
 Example: DCE uses a
separate daemon for
each server machine

2002-08-15 S. Haridi, CS5223, Lecture 02 27

RPC
Summary
 Client-Server remote invocation
 Procedures are uniquely identified

 Parameter passing by value or copy-restore

 No call-by-reference (procedures or data)

 Works for static configurations

 client code has to know the procedures in
advance (compile time)
 Concurrency cannot work with copy-restore
2002-08-15 S. Haridi, CS5223, Lecture 02 28
Remote Object Invocation
 Distributed
objects
 Remote method invocation

 Parameter passing

2002-08-15 S. Haridi, CS5223, Lecture 02 29

Remote Distributed Objects
 Data and operations encapsulated in an object
 Operations are implemented as methods, and are
accessible through interfaces
 Object offers only its interface to clients

 Object server is responsible for a collection of

objects
 Client stub (proxy) implements interface
 Server skeleton handles (un)marshaling and object
invocation

2002-08-15 S. Haridi, CS5223, Lecture 02 30

Remote Distributed Objects

2002-08-15 S. Haridi, CS5223, Lecture 02 31

Remote Distributed Objects
 Compile-time objects
 Language-level objects, from which proxy and
skeletons are automatically generated

 Runtime objects
 Can be implemented in any language, but require
use of an object adapter that makes the
implementation appear as an object

2002-08-15 S. Haridi, CS5223, Lecture 02 32

Remote Distributed Objects
 Transient objects
 live only by virtue of a server: if the server exits,
so will the object

 Persistent objects
 live independently from a server
 if a server exits, the object’s state and code
remain (passively) on disk

2002-08-15 S. Haridi, CS5223, Lecture 02 33

Client-to-Object Binding
 Object reference: Having an object
reference allows a client to bind to an object

 Reference denotes server, object, and

communication protocol
 Client loads associated stub code

 Stub is instantiated and initialized for specific

object
2002-08-15 S. Haridi, CS5223, Lecture 02 34
Client-to-Object Binding
 Implicit: Invoke methods directly on the
referenced object

 Explicit: Client must first explicitly bind to

object before invoking it

2002-08-15 S. Haridi, CS5223, Lecture 02 35

Client-to-Object Binding
Example C++-like syntax
 Implicit

Distr object* obj_ref;

Obj_ref = ...;
Obj_ref→something();

2002-08-15 S. Haridi, CS5223, Lecture 02 36

Client-to-Object Binding
Example C++-like syntax
 Explicit

Distr object* obj_ref;

Local object* obj_ptr;
Obj_ref = ...;
Obj_ptr = bind(obj_ref);
Obj_ptr→something();

2002-08-15 S. Haridi, CS5223, Lecture 02 37

Client-to-Object Binding
 Some remarks
 Reference may contain a URL pointing to an
implementation file
 (Server,object) pair is enough to locate target object
 We need only a standard protocol for loading and
instantiating code

 Observation
 Remote-object references allow us to pass references as
parameters
 This was difficult with ordinary RPCs
2002-08-15 S. Haridi, CS5223, Lecture 02 38
Remote Method Invocation
RMI Basics I
 Assume client stub and server skeleton are in
place
 Client invokes method at stub
 Stub marshals request and sends it to server
 Server ensures referenced object is active
(persistent):
 Create separate process to hold object
 Load the object into server process
 ...
2002-08-15 S. Haridi, CS5223, Lecture 02 39
Remote Method Invocation
Basics II
 Request is unmarshaled by object’s skeleton,
and referenced method is invoked
 If request contained an object reference,
invocation is applied recursively (i.e., server
acts as client)
 Result is marshaled and passed back to
client
 Client stub unmarshals reply and passes
result to client application
2002-08-15 S. Haridi, CS5223, Lecture 02 40
Remote Method Invocation
Parameter Passing
 Object reference: Much easier than in the
case of RPC:

 Server can simply bind to referenced object,

and invoke methods
 Unbind when referenced object is no longer
needed

2002-08-15 S. Haridi, CS5223, Lecture 02 41

Remote Method Invocation
Parameter Passing
 Object-by-value: A client may also pass a complete
object as parameter value
 An object has to be marshaled
 Marshall its state
 Marshall its methods, or give a reference to where an
implementation can be found

 Server unmarshals object. Note that we have now

created a copy of the original object
 Object-by-value passing tends to introduce nasty
problems

2002-08-15 S. Haridi, CS5223, Lecture 02 42

Remote Method Invocation
Parameter Passing

Question: What’s the problem with passing objects by

value?
2002-08-15 S. Haridi, CS5223, Lecture 02 43
Message-Oriented
Communication
 Synchronous versus asynchronous
communications
 Message-Queuing System

 Message Brokers

 Example: IBM MQSeries

2002-08-15 S. Haridi, CS5223, Lecture 02 44

Synchronous Communication
 Some observations: Client/Server computing is
generally based on a model of synchronous
communication

 Client and server have to be active at the time of

communication
 Client issues request and blocks until it receives
reply
 Server essentially waits only for incoming requests,
and subsequently processes them
2002-08-15 S. Haridi, CS5223, Lecture 02 45
Synchronous Communication
Drawbacks

 Client cannot do any other work while waiting

for reply
 Failures have to be dealt with immediately
(the client is waiting)
 In many cases the model is simply not
appropriate (mail, news)

2002-08-15 S. Haridi, CS5223, Lecture 02 46

Asynchronous Communication
Messaging
 Message-oriented middleware: Aims at
high-level asynchronous communication

 Processes send each other messages, which

are queued
 Sender need not wait for immediate reply, but
can do other things
 Middleware often ensures fault tolerance

2002-08-15 S. Haridi, CS5223, Lecture 02 47

Asynchronous Communication
Messaging

2002-08-15 S. Haridi, CS5223, Lecture 02 48

Persistent vs. Transient
Communication
 Persistent communication
 A message is stored at a communication server
as long as it takes to deliver it at the receiver

 Transient communication
 A message is discarded by a communication
server as soon as it cannot be delivered at the
next server, or at the receiver

2002-08-15 S. Haridi, CS5223, Lecture 02 49

Messaging Combinations
Persistency and Synchrony

(a) Persistent asynchronous communication

(b) Persistent synchronous communication
2002-08-15 S. Haridi, CS5223, Lecture 02 50
 Messaging Combinations
Persistency and Synchrony

(c) Transientasynchronous communication

(d) Receipt-based transient synchronous communication

2002-08-15 S. Haridi, CS5223, Lecture 02 51

 Messaging Combinations
Persistency and Synchrony

(e) Delivery-based transient synchronous communication at

message delivery
(f) Response-based transient synchronous communication
2002-08-15 S. Haridi, CS5223, Lecture 02 52
Message-Oriented Middleware
 Essence: Asynchronous persistent
communication through support of
middleware-level queues
 Queues correspond to buffers at
communication servers

 Canonical example: IBM MQSeries

2002-08-15 S. Haridi, CS5223, Lecture 02 53

IBM MQSeries
Basic concepts
 Application-specific messages are put into,
and removed from queues

 Queues always reside under the regime of a

queue manager

 Processes can put messages only in local

queues, through an RPC mechanism

2002-08-15 S. Haridi, CS5223, Lecture 02 54

IBM MQSeries
Message transfer
 Messages are transferred between queues
 Message transfer between queues at different
processes, requires a channel
 At each endpoint of channel is a message channel
agent
 Message channel agents are responsible for
 Setting up channels using lower-level network
 communication facilities (e.g., TCP/IP)
 (Un)wrapping messages from/in transport-level packets
 Sending/receiving packets
2002-08-15 S. Haridi, CS5223, Lecture 02 55
IBM MQSeries
Message transfer

2002-08-15 S. Haridi, CS5223, Lecture 02 56

IBM MQSeries
Message transfer
 Channels are inherently unidirectional

 MQSeries provides mechanisms to automatically

start MCAs when messages arrive, or to have a
receiver set up a channel

 Any network of queue managers can be created

 routes are set up manually (system administration)

2002-08-15 S. Haridi, CS5223, Lecture 02 57

IBM MQSeries
Message transfer
Attribute Description

Transport type Determines the transport protocol to be used

Indicates that messages are to be delivered in the order they are

FIFO delivery
sent

Message
Maximum length of a single message
length

Setup retry
Specifies maximum number of retries to start up the remote MCA
count

Delivery retries Maximum times MCA will try to put received message into queue

2002-08-15 S. Haridi, CS5223, Lecture 02 58