Programming III

Parallel and distributed programming

Basic terms and concepts parallel

and distributed programming

Janez Žibert, Jernej Vičič UP FAMNIT

Overview (1. chapter in the book)
 What is a parallel program?
 Computer architectures for running parallel programs.
 Basic concepts in (parallel) programming: task, process,
process state, ...
 Interprocess communication:
 communication with shared memory,
 communication through linking structures.
 Basic concepts of parallel/distributed programming:
 iterative parallelism,
 recursive parallelism,
 principle manufacturer/consumer,
 client/server principle,
 principle of interaction among equivalent entities.

2 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Sequential an parallel program
 Sequential program:
 represents a sequence of commands - operations
performed one after the other in order to solve (perform) a
particular task,
 a running program is called a process.
 Parallel program:
 it involves two or more processes involved in solving a
particular task:
 each process implements its sequence of commands,
 processes participate/communicate with each other through
shared memory or with the help of messages
 running:
 simultaneously on single-processor systems (task-switcing),
 parallel to multiprocessor or multi-computer systems.

3 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Forms of programs with regard to the running
of processes
 Simultaneous computing applications
 processes are run simultaneously on one processor (or multiple processes per processor)
 communication is done by reading and writing to shared memory,
 applications:
 modern operating systems,
 graphic interfaces,
 real-time systems.
 Distributed computing applications
 processes are run/executed on several computers that are connected to a network,
 communication is done through the exchange of messages (by receiving and transmitting messages),
 applications:
 wherever there is a requirement for distributed resources:
 file and data servers,
 www servers, p2p, iptv, ...
 Parallel computing applications
 processes run parallel on multiple processors (as many processes as processors)
 communication:
 using shared memory or with messages
 applications:
 speeding up procedures for processing large amounts of data:
 modeling: optimization, statistical modeling, image and video processing (graphics cards).

4 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Why parallel programs?
 Motivations:
 speeding up procedures (due to the division of work
between multiple processors, computers)
 requirements for the distributed operation of individual
services (client/server, p2p),
 ”natural" approach to solving certain problems:
 multitasking in operating systems,
 graphical interfaces, ...,
 increasing scalability of individual programs/applications

5 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Efficiency of parallel programs
 In parallel programming, we are looking for such
algorithms that lead us to the desired goal rather
than a strictly sequential solution to the problem.
 Measuring the speedup:

 actual speedup:
sequential algorithm on 1 proc. of a parall.omputer
S(N) =
parallel algorithm on N proc. of a parall. computer

 relative speedup:
parallel algorithm on 1 proc. paral. computer
S(N) =
parallel algorithm on N proc. paral. computer

6 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Factors limiting the speedup
 problem decomposition costs
 increment of the number of instructions - increasing the
program load (additional operations, new variables, ...),
 uneven load distribution (uneven distribution of data,
unbalanced complexity of suboperations),
 an increase in the communication burden (synchronization)
 sequential nature of the problem:
 Amdahl Law:
s - the degree of sequentionality of the
problem, the percentage of time we
need for solving the sequential part of
the program.
The speedup of the problem is limited by 1/s.

7 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Computer models for paralell programming
 The computer executes commands over the data:
 the sequence of commands (flow of commands) specifies the
operations that the computer executes one after another,
 the sequence of data (data flow) determines over what
operations are executed,
 Given the number of simultaneous flows of commands
and data, we divide the computers into the following
models (Flynn):
 SISD (Single Instruction Single Data Stream)
 SIMD (Single Instruction Multiple Data Stream)
 MISD (Multiple Instruction Single Data Stream)
 MIMD (Multiple Instruction Multiple Data Stream)
 shared memory,
 distributed memory.

8 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

SISD computer

KE

IS instruction flow

PE
DS data flow

Pomnilnik

 von Neumannov type computer

 sequential computer:
 the computer executes instructions one-by-one
 Example: add N numbers

9 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

SIMD computer
KE
KE
IS instruction flow

PE1 PE2 … PEn

DS1 DS2 DSn data flows

Pomnilnik

 the computer executes at a given moment one

operation over more than one data,
 simultaneous access to data via vectors or arrays
 Example: simultaneously adding multiple pairs of
numbers
10 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT
MISD computer

KE1 KE2 … KEn

IS1 IS2 ISn instruction flows

PE1 PE2 … PEn

data flow

Memory

 the computer executes at a given moment several

operations over one datam
 the model was not implemented
 Example: Is the number N prime?
11 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT
MIMD computer with shared memory

KE1 KE2 … KEn

IS1 IS2 ISn instruction flows

PE1 PE2 … PEn

DS1 DS2 DSn data flows

Memory

 the computer executes at a given moment a number of operations over different data,
 Multi-processor system:
 processors share shared memory,
 processors access global memory through a common bus (bottleneck if we have a lot of
processors).
 processors communicate with each other by reading and writing to a shared memory.

12 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

MIMD computer with distributed memory

P1 P2 … Pn computers

LM1 LM2 … LMn

Connection network

 Multi computer system:

 system of multiple computers connected to a connection network (do not share
memory!),
 they communicate with each other by exchanging messages,
 various structures of multi-computing systems:
 same/different performance computers (processors),
 topology of interconnection networks (networks, hypercube, ...),
 resource management: centralized, distributed,
 types of connections: myrinet, infiniband, ethernet...

13 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Basic concepts in parallel programming: PROCESS
 task – unit of execution of a computer program
performing a task,
 at PP: commands (operations) in a task are executed in
succession,
 process – computer program in execution (when the
program starts running it becomes a process):
 can perform one or more tasks,
 during the execution the process changes state:
in
spawned ready ended
execution

stopped

14 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

PROCESSES in operating systems
 In a multi-tasking OS, there are several processes over a
certain period, each of them is at a certain stage of
progression.
 In the running state: one process per processor,
 In standby mode or stopped: any number,
 In each OS, we have a scheduler:
 it determines which process to run next (and for how long),
 Process switching in the processor is performed by the
dispatcher:
 context switch:
 Process context - all that is needed to implement the process:
 address space: text, data, stack, shared memory ...,
 control data,
 process ID, program counter, stack pointer, registers, ...

15 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

PROCESSES and THREADS
 Each process can use one or more threads.
 A thread is a new process that shares the father's
address space:
 has its own stack, its own registers, has its own ID,
 with the other threads of the common process:
 shares a common address space and,
 global variables.
 has access to common files,
 has access rights to everything within this process. Wikipedia

 Example: LINUX fork () and clone ():

 fork() creates a new process with a new process context,
 clone() creates a new process with its own identity,
 but with the sharing of data structures of the father.

16 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

PROCESSES and THREADS
UNIX processes hreads in UNIX processes

https://computing.llnl.gov/tutorials/pthreads/
17 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT
 Processes in one-processor system
Computer Computer
application application

Process Process Process

A B C

Each process has

one or more
(S. Divjak, Lectures OS II, FRI, UNILJ)
threads
T1 T2 T3 T4 T5

Scheduler

Scheduler determines
which thread will be CPE
executed

18 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Types of parallel programming
 In parallel programming, we determine how processes communicate
with one another (interprocess communication) in solving a specific
task and what methods of synchronization are used.
 Programming using shared memory
 processes share common memory,
 communication is done by reading and writing data into common (global)
variables (data structures),
 explicit synchronization is needed when processes can be write/read from
common variables.
 Distributed memory programming
 processes do not have shared memory,
 they share interconnection structures (communication channels),
 communication is done by sending and receiving messages via
communication channels,
 synchronization is implicit: in order for the message to be accepted, it
must be sent.

19 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Synchronisation and communication
mechanisms
 Synchronisation mechanisms (shared memory):
 locks,
 bariers,
 condition variables,
 semaphores,
 monitors.
 Communication mechanisms (distributed memory):
 message exchange
 asynchronous and synchronous message exchange,
 with calls to perform remote procedures (operations)
 with the RPC (remote procedure call) method and the
"rendezvous”,
 with the RMI method (remote method invocation).

20 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Pseudo code, used in the book
 C-like language.
 Declarations – as in C or/and matlab
 variables: basic types (int, double, char, …)
 arrays: int c[1:n], double c[n,n]
 process declaration process – start execution of one or more
processes in the background.
 Instructions:
 assignment a= y*x + f(z)
 flow controll: if, while, for (as in C or Java)
 for loops (matlab)
 for [i=0 to n-1, j=0 to n-1]
 for [i=1 to n by 2] #all odd numbers from 1 to n
 for [i=1 to n st i != x] #all numbers from 1 to, except i==x (st = “such
that”)
 concurrent execution – co
 the command begins with the simultaneous execution of two or more threads
waiting for it until they are finished
 synchronisation – await
 the command is waiting until a condition is met (it will be presented below)
21 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT
co
 branching form
 each thread is defined in one branch
 threads are executed in parallel
 at the oc we wait until
 all threads finish
 using iterative variables
 threads are defined by a combination of iterative variables,
 defined in the co command
 threads are executed in parallel,
 at the oc point we wait until all the threads finish

22 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Declaration of a process
 Syntax is similar to co.
 Two forms:
P0
 one process
process bar1 {
for [i=1 to n]
process
write(i);
} P1
process
 many processes
process bar2 [i= 1 to n] {
P2
write(i);
}

 The process is executed at the declaration and is run in the background.

 the program continues its execution,
 we do not wait for the processes to be completed.
 Within the process we can define local variables and use global variables.

23 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Concepts of parallel/distributed programming

 iterative parallelism,
 recursive parallelism,
 principe producer/consumer,
 principe client/server,
 principe interaction among equals.

24 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Iterative parallelism
 Parallelization of independent iterations in sequential procedures.
 For example. parallelization of loops, where the loop iteration is
performed independently of other iterations.
 Example: multiplication of matrices C = A*B
 sequential program parallel program
double a[n,n], b[n,n], c[n,n] double a[n,n], b[n,n], c[n,n]
for [i=0 to n-1] { co [i=0 to n-1] {
for [j=0 to n-1] { for [j=0 to n-1] {
c[i,j] = 0; c[i,j] = 0;
for [k=0 to n-1] for [k=0 to n-1]
c[i,j] = c[i,j]+a[i,k]*b[k,j]; c[i,j] = c[i,j]+a[i,k]*b[k,j];
} }
} } oc;

 substitute for and co,

 we get n threads for each line i, that is executed at the same time,
 we could parallelize on wors and columns and get (n^2 threads)
 co [i=0 to n-1, j=0 to n-1]

25 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Recursive parallelism
 Parallelization of independent recursive calls.
 we have a program that is recursively executed on
data
 recursions are independent among themselves =>
each recursive call is performed in its thread.
 Examples:
 quick sort, recursive calculation of a given integral,
...wherever we can use recursion according to the ”divide
and conquer" principle - so where we can use the same
process on smaller pieces of data to solve the problem on
all data.

26 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Approximation of the calculation of a definite integral
 Task:
 Approximate the calculation of a definite integral of a function on an interval!

 Iterative procedure (sequential program)

 use a trapezoidal rule

double fl = f(a), fr = f(b), pl = 0.0;

double dx = (b-a)/ni;
for [x = (a+dx) to b by dx] {
fr = f(x);
pl = pl + (fl+fr)*dx/2;
fl = fr;
}

27 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Recursive calculation of a definite integral

first approx second approx

pl lpl + dpl

 Recursion:
 calculate area pl on a given interval
 split the interval into two halves and calculate left and right area,
lpl and rpl
 recursion is stopped when |(lpl+dpl) – pl|< eps

28 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Recursive approximation of the calculation of a definite
integral

Sequential program Parallel program

double ninteg(double l,d,fl,fd,pl) {
double ninteg(double l,d,fl,fd,pl) {
double m = (l+d)/2;
double m = (l+d)/2;
double fm = f(m);
double fm = f(m);
double lpl = (fl+fm)*(m-l)/2;
double lpl = (fl+fm)*(m-l)/2;
double dpl = (fm+fd)*(d-m)/2;
double dpl = (fm+fd)*(d-m)/2;
if (abs((lpl+dpl)-pl) > eps) {
if (abs((lpl+dpl)-pl) > eps) {
co lpl = ninteg(l,m,fl,fm,lpl);
lpl = ninteg(l,m,fl,fm,lpl);
|| dpl = ninteg(m,d,fm,fd,dpl);
dpl = ninteg(m,d,fm,fd,dpl);
oc;
}
}
return (lpl+dpl);
return (lpl+dpl);
}
}

starting call
pl = ninteg(a,b,f(a),f(b),(f(a)+f(b))+(b-a)/2)

 Recursive calls are independent and can be executed in parallel.

29 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Producer/consumer principle
 The producer produces data
 The consumer uses the data
 It is a reciprocal relationship, the production and consumption of data
is carried out simultaneously.
 The flow of data is directed from the producer to the consumer.
 In the middle we can have "filters".
 Example:$ catUnix pipes
list-1 list-2 list-3 | sort | uniq > final.list
# Concatenates the list files,
# sorts them,
# removes duplicate lines,
# and finally writes the result to an output file.

 each program reads from the input and writes to the output, the program
uses the data produced by the previous program and produces data for its
successor,
 data between programs (processes) are stored in a buffer organized by a
FIFO queue.

30 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Client/server principle
request
client1 answer
server
 Client-server relationship answer
 the client requires a service, client2
request
 the server that provides the service responds to the request and sends an answer
 mutual communication: demand, response
 Parallelism must be enabled on the server side, which must ”service"
multiple clients:
 multithread process, sometimes synchronization between services is also needed,
 there can be many clients.
 Implementation:
 in distributed systems: communication by sending messages,
 with remote procedure calls (RPC, RMI, ...)
 in shared memory: requirements performed as program functions (eg open, read,
write), need for synchronization
 Example, a file server
 client requests: open, read, write, close
 server responses: data from the file, request execution status (it succeeded,
problems ...)

31 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Interaction of equals
 Here we solve a problem by distributing the work among
several "workers", all of whom perform approximately the
same amount of work.
 Workers solve their problem with their work and
cooperation.
 Usually:
 we solve a problem that involves the processing of
input data,
 the data is divided into smaller parts
 then the same procedure is carried out on every part
of the data (workers).
 Normally, communication between these procedures is
required for the final solution.
32 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT
Interaction of equals
 Methods of work distribution and interaction:
 coordinator, workers delavec1 … delavecN
rezultati podatki

podatki koordinator rezultati

 circular interaction
delavec1 … delavecN

 grid interaction

 ...

33 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Coordinator/workers: matrix multiplication
worker 1 … worker N
results dat

data coordinator results

 Worker  Coordinator

34 Parallel and distributed programming, Janez Žibert, Jernej Vičič UP FAMNIT

Circular interaction: matrix multiplication
worker … worker N

35