INTERPROCESS

COMMUNICATION
 Topics

 INTRODUCTION
 The API for the INTERNET PROTOCOLS
 EXTERNAL DATA REPRESENTATION
 CLIENT-SERVER COMMUNICATION
 GROUP COMMUNICATION
 What is IPC?
 IPC mechanisms allow processes to
communicate and to synchronize their actions
without sharing the same address space.

 IPC mechanism in a distributed system should

make communication transparent.
 Interprocess Communications
- Exchange of data between two or more separate,
independent processes/threads.
- Operating systems provide facilities/resources for inter-
process communications (IPC), such as message
queues and shared memory.
- Distributed computing systems make use of these
facilities/resources to provide application
programming interface (API) which allows IPC to be
programmed at a higher level of abstraction. (e.g.,
send and receive)
- Distributed computing requires information to be
exchanged among independent processes.
 Unicast vs. Multicast

P2 P2 P3 ... P4

m
m m m

P1 P1

uni c as t m ul ti c as t
Interprocess Communications in
Distributed Computing

P roc es s 1 P roc es s 2

data

s ender rec eiver

 Operations provided in
Interprocess Communications API

• Receive ( [sender], message storage

object)
• Connect (sender address, receiver
address), for connection-oriented
communication.
• Send ( [receiver], message)
• Disconnect (connection identifier), for
connection-oriented communication.
 Distributed IPC
 How do we support communication over a
network?
 How do we hide the distinction between local
and remote communication?
 How do we deliver information with minimum
latency, maximum throughput and minimum
jitter?
 How do we shield one process from failures of
another?
 How do we ensure security?
Distributed Communication
 All communication in distributed system is
based on low-level message passing as
offered by the underlying network.

 Process A builds a message in its own

address space and executes a system call that
causes the operating system to send the
message over the network to process B.
 What Do We Need?
 Good network infrastructure.
 Efficient protocol stack.
 Transport protocol to handle various types of
data.
 Communication models.
 Security.
 Introduction – Understanding Concepts
 The java API for interprocess communication
in the internet provides both datagram and
stream communication.
 The two communication patterns that are
most commonly used in distributed
programs:
 Client-Server communication
 The request and reply messages provide the
basis for remote method invocation (RMI) or
remote procedure call (RPC).
Introduction – Understanding Concepts
 Group communication
 The same message is sent to several
processes.

 Remote Method Invocation (RMI)

 It allows an object to invoke a method in
an object in a remote process.
 E.g. CORBA and Java RMI
 Remote Procedure Call (RPC)
 It allows a client to call a procedure in a
remote server.
Introduction – Understanding Concepts

 Request-reply protocols are designed to

support client-server communication in the
form of either RMI or RPC.
 Group multicast protocols are designed to
support group communication.

 Group multicast is a form of interprocess

communication in which one process in a
group of processes transmits the same
message to all members of the group.
Characteristics of Inter-
process Communication
The Characteristics of Interprocess Communication

 Synchronous and asynchronous

communication

 In the synchronous form, both send and

receive are blocking operations.
 In the asynchronous form, the use of the
send operation is non-blocking and the
receive operation can have blocking and
non-blocking variants.
Synchronization in message passing (1)

 Message passing may be blocking or non-

blocking.
 Blocking is considered synchronous
 Blocking send has the sender block until the message
is received
 Blocking receive has the receiver block until a
message is available
 Non-blocking is considered asynchronous
 Non-blocking send has the sender send the message
and continue
 Non-blocking receive has the receiver receive a valid
message or null
Synchronization in message passing (2)
 For the sender: it is more natural not to be
blocked after issuing send:
can send several messages to multiple
destinations.
but sender usually expect acknowledgment of
message receipt (in case receiver fails).
 For the receiver: it is more natural to be
blocked after issuing receive:
the receiver usually needs the information
before proceeding.
but could be blocked indefinitely if sender
process fails before send.
Synchronization in message passing (3)

 Hence other possibilities are

sometimes offered.
 Example: blocking send, blocking
receive:
both are blocked until the message is
received.
occurs when the communication link is
unbuffered (no message queue).
Synchronization in message passing (4)
 There are really 3 combinations here that
make sense:
1. Blocking send, Blocking receive
2. Nonblocking send, Nonblocking receive
3. Nonblocking send, Blocking receive –
most popular – example:
 Server process that provides
services/resources to other processes. It
will need the expected information before
proceeding.
 Event Synchronization
 Interprocess communication may require
that the two processes synchronize their
operations: one side sends, then the other
receives until all data has been sent and
received.
 Ideally, the send operation starts before
the receive operation commences.
 In practice, the synchronization requires
system support.
 Synchronous vs.
Asynchronous Communication
 The IPC operations may provide the synchronization
necessary using blocking. A blocking operation issued
by a process will block further processing of the process
until the operation is fulfilled.
 Alternatively, IPC operations may be asynchronous or
nonblocking. An asynchronous operation issued by a
process will not block further processing of the process.
Instead, the process is free to proceed with its
processing, and may optionally be notified by the system
when the operation is fulfilled.
Synchronous send and receive
pr o c e s s 1 pr o c e s s 2
r unni ng o n ho s t 2
r unni ng o n ho s t 1
Event Diagram

b lo c k in g r ec eiv e s tar ts

b lo c k in g s en d s tar ts an o p er atio n

ex ec u tio n f lo w
ac k n o w led g em en t o f d ata r ec eiv ed s u s p en d ed p er io d
b lo c k in g s en d r etu r n s p r o v id ed b y th e I P C f ac ility b lo c k in g r ec eiv e en d s

Sync hr o no us Se nd and R e c e i ve

Client Server
Sender Receiver
Asynchronous send and
synchronous receive
P rocess 2
P rocess 1
Event Diagram

b lo c k in g r e c e iv e s ta r ts

n o n b lo c k in g s e n d

o p e r a tio n
e x e c u tio n f lo w
s u s p e n d e d p e r io d
b lo c k in g r e c e iv e r e tu r n s

As yn c h r o n o u s S e n d an d
S yn c h r o n o u s R e c e i ve

Client Server
Sender Receiver
Synchronous send and Async. Receive - 1

P rocess 2
P rocess 1

b lo c k in g s e n d is s u ed

n o n b lo c k in g r e c e iv e is s u e d
tr a n s p a r e n t a c k n o w le d g e m e n t
p r o v id e d b y th e I P C f a c ility e x e c u tio n f lo w
s u s p e n d e d p e r io d

S yn c h r o n o u s S e n d an d
As yn c h r o n o u s R e c e i ve
S ce n a rio A

Data from P1 was received by P2

before issuing a non-blocking receive op in P2
Synchronous send and Async. Receive - 2

P rocess 2
P rocess 1

n o n b lo c k in g r e c e iv e is s u e d
a n d r e tu r n e d im m e d ia te ly
b lo c k in g s e n d is s u e d

in d e f in ite
b lo c k in g e x e c u tio n f lo w
s u s p e n d e d p e r io d

P rocess 2
P rocess 1 S yn c h r o n o u s S e n d an d
As yn c h r o n o u s R e c e i ve
S c e n a r io B

Data from P1 arrived to P2 after P2 issued a

non-blocking receive op
Synchronous send and Async. Receive - 3

P ro cess 2
P ro cess 1

n o n b lo c k in g r e c e iv e is s u e d
a n d r e tu r n e d im m e d ia te ly
b lo c k in g s e n d is s u e d
p r o c e s s is n o tif ie d
tr a n s p a r e n t a c k n o w le d g e m e n t o f th e a r r iv a l o f
p r o v id e d b y th e I P C f a c ility d a ta
e x e c u tio n f lo w
s u s p e n d e d p e r io d

S yn c h r o n o u s S e n d an d
As yn c h r o n o u s R e c e i ve
S ce n a rio C

Data from P1 arrived to P2 before P2 issues a non-blocking

receive op. P2 is notified of the arrival of data
Asynchronous send and
Asynchronous receive
P rocess 2
P rocess 1

n o n b lo c k in g r e c e iv e is s u e d
a n d r e tu r n e d im m e d ia te ly
b lo c k in g s e n d is s u e d
p r o c e s s is n o tif ie d
o f th e a r r iv a l o f
d a ta
e x e c u tio n f lo w
s u s p e n d e d p e r io d

As yn c h r o n o u s S e n d an d
As yn c h r o n o u s R e c e i ve
S ce n a rio C

Does P1 need an acknowledgement from P2?

 Event diagram
P ro ce s s B
P ro ce s s A

t im e
r eq u es t 1

r es p o n s e 1

r eq u es t 2
in ter p r o c es s c o m m u n ic atio n
ex ec u tio n f lo w
r es p o n s e2
p r o c es s b lo c k ed

E ve nt di ag r am fo r a pr o to c o l

Synchronous send and receive

Blocking, deadlock, and timeouts
 Blocking operations issued in the wrong sequence can cause
deadlocks.
 Deadlocks should be avoided. Alternatively, timeout can be
used to detect deadlocks.
P r o c es s 1 P r o c es s 2

r ec eiv e f r o m p r o c es s 2 is s u ed
p r o c es s 1 b lo c k ed p en d in g d ata
f r o m p r o c es s 2 .
r ec eiv ed f r o m p r o c es s 1 is s u ed
p r o c es s 2 b lo c k ed p en d in g d ata
f r o m p r o c es s 1 .

P1 is waiting for P2’s data; P2 is waiting for P1’s data.

 Using threads for asynchronous IPC

 When using an IPC programming interface, it is important to note

whether the operations are synchronous or asynchronous.
 If only blocking operation is provided for send and/or receive, then it
is the programmer’s responsibility to using child processes or threads
if asynchronous operations are desired.

pr o c e s s

m a in th r e a d

n e w th r e a d is s u e s a b lo c k in g I P C o p e r a tio n

th r e a d is b lo c k e d
m a in th r e a d c o n tin u e s w ith
o th e r p r o c e s s in g
th r e a d is u n b lo c k e d a f te r th e o p e r a tio n is f u lf ille d
 Deadlocks and Timeouts
 Connect and receive operations can result in indefinite
blocking
 For example, a blocking connect request can result in
the requesting process to be suspended indefinitely if the
connection is unfulfilled or cannot be fulfilled, perhaps
as a result of a breakdown in the network .
 It is generally unacceptable for a requesting process to
“hang” indefinitely. Indefinite blocking can be avoided by
using timeout.
 Indefinite blocking may also be caused by a deadlock
Indefinite blocking due to a
deadlock
Pro ce s s 1 Pro ce s s 2

" re ce iv e fro m pro ce s s 2 " is s u e d;

pro ce s s 1 blo ck e d pe n din g da ta
fro m pro ce s s 2 .
" re ce iv e fro m pro ce s s 1 " is s u e d;
pro ce s s 2 blo ck e d pe n din g da ta
a n o pe ra ti o n
fro m pro ce s s 1 .
pro ce s s
e x e cu ti n g
pro ce s s
bl o ck e d

P1 is waiting for P2’s data; P2 is waiting for P1’s data.

 IPC Requirements
 If P and Q wish to communicate, they
need to:
establish communication link between them.
exchange messages via send/receive.
 Implementation of communication link:
physical (e.g., shared memory, hardware bus)
logical (e.g., logical properties)
The Characteristics of Interprocess Communication

 Message destinations
 A local port is a message destination
within a computer, specified as an integer.
 A port has an exactly one receiver but can
have many senders.
 The Characteristics of Interprocess Communication

 Reliability
 A reliable communication is defined in
terms of validity and integrity.
 A point-to-point message service is
described as reliable if messages are
guaranteed to be delivered despite a
reasonable number of packets being
dropped or lost.
 For integrity, messages must arrive
uncorrupted and without duplication.
 The Characteristics of Interprocess Communication

 Ordering
 Some applications require that messages
be delivered in sender order.
Sockets
 Sockets
 A socket is one endpoint of a two-way
communication link between two programs running
on the network. A socket is bound to a port number so
that the TCP layer can identify the application that
data is destined to be sent to.
 A socket is defined as an endpoint for communication.
 Concatenation of IP address and port.
 The socket 161.25.19.8:1625 refers to port 1625 on
host 161.25.19.8.
 Communication consists between a pair of sockets.
Socket Communication
 Sockets

 Messages sent to some internet address

and port number can only be received by a
process using a socket that is bound to
this address and port number.
 Processes cannot share ports (exception:
TCP multicast).
 Sockets
 Both forms of communication, UDP and
TCP, use the socket abstraction, which
provides and endpoint for communication
between processes.

 Interprocess communication consists of

transmitting a message between a socket
in one process and a socket in another
process.
(Figure 2)
 Sockets

Figure 2. Sockets and ports

 UDP Datagram Communication
 UDP datagram properties
 No guarantee of order preservation
 Message loss and duplications are
possible
 Necessary steps
 Creating a socket
 Binding a socket to a port and local
Internet address
 A client binds to any free local port
 A server binds to a server port
 UDP Datagram Communication
 Receive method
 It returns Internet address and port of
sender, plus message.

 Issues related to datagram communications are:

 Message size
 IP allows for messages of up to 216 bytes.
 Most implementations restrict this to around 8
kbytes.
 Any application requiring messages larger than
the maximum must fragment.
 If arriving message is too big for array allocated
to receive message content, truncation occurs.
 UDP Datagram Communication
 Blocking
 Send: non-blocking
• upon arrival, message is placed in a queue for the
socket that is bound to the destination port.
 Receive: blocking
• Pre-emption by timeout possible
• If process wishes to continue while waiting for
packet, use separate thread

 UDP datagrams suffer from following failures:

 Omission failure
 Messages may be dropped occasionally,
 Ordering
 TCP Stream Communication

 The API to the TCP protocol provides the

abstraction of a stream of bytes to be
written to or read from.
 Characteristics of the stream abstraction:
 Message sizes
 Lost messages
 Flow control
 Message destinations
 TCP Stream Communication

 Issues related to stream communication:

 Matching of data items
 Blocking
 Threads
 TCP Stream Communication

 Use of TCP
 Many services that run over TCP
connections, with reserved port number
are:
 HTTP (Hypertext Transfer Protocol)
 FTP (File Transfer Protocol)
 Telnet
 SMTP (Simple Mail Transfer Protocol)
External Data Representation
 External Data Representation

 The information stored in running programs is

represented as data structures, whereas the
information in messages consists of sequences
of bytes.

 Irrespective of the form of communication used,

the data structure must be converted to a
sequence of bytes before transmission and
rebuilt on arrival.
 External Data Representation

 External Data Representation is an agreed

standard for the representation of data
structures and primitive values.

 Data representation problems are:

 Using agreed external representation, two conversions
necessary
 Using sender’s or receiver’s format and convert at the other
end
 External Data Representation
 Marshalling
 Marshalling is the process of taking a
collection of data items and assembling
them into a form suitable for transmission
in a message.
 Unmarshalling
 Unmarshalling is the process of
disassembling a collection of data on
arrival to produce an equivalent collection
of data items at the destination.
 Marshalling Goals
 Linearize the data structures for transport in
messages and reconstructing the original data
structures at the other end.

 Convert data structures from the data

representation on the calling process to that of
the called process.
 External Data Representation

 Marshalling and unmarshalling activities is

usually performed automatically by
middleware layer.
 Marshalling is likely error-prone if carried
out by hand.
CORBA Common Data Representation (CDR)
 CORBA Common Data Representation (CDR)
 CORBA CDR is the external data representation
defined with CORBA 2.0.
 It consists 15 primitive types:
 Short (16 bit)
 Long (32 bit)
 Unsigned short
 Unsigned long
 Float(32 bit)
 Double(64 bit)
 Char
 Boolean(TRUE,FALSE)
 Octet(8 bit)
 Any(can represent any basic or constructed type)

 Composite type are shown in Figure 8.

CORBA Common Data Representation (CDR)

Figure 8. CORBA CDR for constructed types

RPC
 Remote Procedure Calls (RPCs)
 RPC abstracts a Local Procedure Call (LPC)
between processes on a networked system.
 Stubs – client-side proxy for the actual
procedure existing on the server.
 The client-side stub locates the server and
marshals the parameters.
 The server-side stub/skeleton receives this
message, unpacks the marshaled parameters,
and performs the procedure on the server.
 Vice versa happens on the opposite direction.
 Procedure Calls
 Procedure calls - more natural way to
communicate:
 Every language supports them.
 Natural for programmers to use.
 Idea: Servers can export a set of procedures
that can be called by client programs
 Similar to module interfaces, class definitions, etc.
 Clients just do a procedure call as it they were directly
linked with the server.
 Under the covers, the procedure call is converted into
a message exchange with the server.
 Implementing RPC
 Trick:
Create stub functions that make it appear to
the user that the call is really local.
A stub function looks like the function that the
user intends to call but really contains code
for sending and receiving messages over a
network.
Functional Steps in RPC
 Benefits
 Procedure call interface.
 Writing applications simplified:
RPC hides all network code into stub
functions.
Application programmers don’t have to worry
about details:
Sockets, port numbers, byte ordering.
 Presentation layer in OSI model.
 RPC Stubs
 A client-side stub looks to the client as if it were a
callable server procedure.
 A server-side stub looks to the server as if a
client called it.
 The client program thinks it is calling the server
(In fact, it is calling the client stub).
 The server program thinks it is called by the client
(In fact, it is called by the server stub).
 The stubs send messages to each other to make
the RPC happen “transparently”.
Remote Procedure Call Mechanism
 RPC Transparency
 One goal of RPC is to be as transparent as possible:
 Make remote procedure calls look like local procedure
calls.
 Failures – remote nodes/networks can fail in more ways
than with local procedure calls:
 Need extra support to handle failures well.
 Performance – remote communication is inherently
slower than local communication.
 If program is performance-sensitive, could be a
problem.
RMI
 Remote Method Invocation

 Remote Method Invocation (RMI) is a Java

mechanism similar to RPCs.
 RMI allows a Java program on one
machine to invoke a method on a remote
object.
 Remote Object References

 Remote object references are needed when a

client invokes an object that is located on a
remote server.
 A remote object reference is passed in the
invocation message to specify which object is to
be invoked.
 Remote object references must be unique over
space and time.
 Remote Object References

 In general, may be many processes hosting

remote objects, so remote object referencing
must be unique among all of the processes in the
various computers in a distributed system.
 generic format for remote object references is
shown in Figure 11.

Figure 11. Representation of a remote object references

 Remote Object References

 internet address/port number: process which

created object
 time: creation time
 object number: local counter, incremented each
time an object is created in the creating process
 interface: how to access the remote object (if
object reference is passed from one client to
another)
Client/Server Communication
 Client-Server Communication

 The client-server communication is designed to

support the roles and message exchanges in
typical client-server interactions.
 In the normal case, request-reply communication
is synchronous because the client process
blocks until the reply arrives from the server.
 Asynchronous request-reply communication is
an alternative that is useful where clients can
afford to retrieve replies later.
 Client-Server Communication
 Often built over UDP datagrams
 Client-server protocol consists of
request/response pairs, hence no
acknowledgements at transport layer are
necessary
 Avoidance of connection establishment
overhead
 No need for flow control due to small amounts
of data are transferred
 Client-Server Communication
 The request-reply protocol is shown in Figure 12.

Figure 12. Request-reply communication

 Client-Server Communication
 The information to be transmitted in a request
message or a reply message is shown in Figure 14.

Figure 14. Request-reply message structure

 Client-Server Communication
 In a protocol message
 The first field indicates whether the message is a
request or a reply message.
 The second field request id contains a message
identifier.
 The third field is a remote object reference .
 The forth field is an identifier for the method to be
invoked.
 Client-Server Communication

 HTTP: an example of a request-reply

protocol
 HTTP is a request-reply protocol for the
exchange of network resources between
web clients and web servers.
 Client-Server Communication

 HTTP protocol steps are:

 Connection establishment between client
and server at the default server port or at a
port specified in the URL
 client sends a request
 server sends a reply
 connection closure
 The Basic HTTP protocol
w eb s erver w eb brow s er

reques t is a m es s age in 3 parts :

reques t
- < c om m and> < doc um ent adddres s > < HT T P vers ion>
- an optional header
- optional data for C G I data us ing pos t m ethod

res pons e res pons e is a m es s age c ons is ting of 3 parts :

- a s tatus line of the form at < protoc ol> < s tatus c ode> < des c ription>
- header inform ation, w hic h m ay s pan s everal lines ;
- the doc um ent its elf.

W e w ill explore HT T P in details later this quarter.

 A sample HTTP session
Script started on Tue Oct 10 21:49:28 2000
9:49pm telnet www.csc.calpoly.edu 80
Trying 129.65.241.20...
Connected to tiedye2-srv.csc.calpoly.edu.
Escape character is '^]'.
GET /~mliu/ HTTP/1.0 HTTP Request

HTTP/1.1 200 OK HTTP response status line

Date: Wed, 11 Oct 2000 04:51:18 GMT HTTP response header
Server: Apache/1.3.9 (Unix) ApacheJ Serv/1.0
Last-Modified: Tue, 10 Oct 2000 16:51:54 GMT
ETag: "1dd1e-e27-39e3492a"
Accept-Ranges: bytes
Content-Length: 3623
Connection: close
Content-Type: text/html

<HTML> document content

<HEAD>
<TITLE>Mei-Ling L. Liu's Home Page
</TITLE>
</HEAD>
<BODY bgcolor=#ffffff>
…
Group Communication
 Group Communication
 The pairwise exchange of messages is not
the best model for communication from
one process to a group of other
processes.
 A multicast operation is more appropriate.
 Multicast operation is an operation that
sends a single message from one process
to each of the members of a group of
processes.
 Group Communication
 The simplest way of multicasting, provides
no guarantees about message delivery or
ordering.
 Multicasting has the following
characteristics:
 Fault tolerance based on replicated
services
 A replicated service consists of a group of
servers.
 Group Communication
 Client requests are multicast to all the
members of the group, each of which
performs an identical operation.
 Finding the discovery servers in
spontaneous networking
 Multicast messages can be used by servers
and clients to locate available discovery
services in order to register their interfaces
or to look up the interfaces of other services
in the distributed system.
 Group Communication
 Better performance through replicated
data
 Data are replicated to increase the
performance of a service.
 Propagation of event notifications
 Multicast to a group may be used to notify
processes when something happens.