Parallel Programming

& Parallel Computers

Dr. Eng. Marwa Dahdouh

 ‫مفهوم التفرعية‬
‫‪ ‬كل البرامج يتم تنفيذها سطرا ً تلو اآلخر‪.‬‬
‫‪ ‬ال يتم تنفيذ التعليمة أو السطر قبل المرور على األسطر التي تسبقه‪.‬‬
‫‪ ‬إنجاز مجموعة من األعمال في وقت متزامن‪ ,‬يدعى هذا اإلنجاز إنجازا ً تفرعيا ً‪.‬‬
‫‪ ‬تكون التفرعية بتقسيم كمية العمل على عدد أكبر من األفراد‪.‬‬
‫‪ ‬أو بزيادة عدد الموارد مما يسمح بتنفيذ مجموعة من المهمات بشكل متزامن‪.‬‬

‫‪Dr. Eng. Marwa DAHDOUH‬‬

 ‫فوائد استخدام احلوسبة التفرعية‬
‫‪ ‬الوقت‪ :‬ينقص وقت تنفيذ التطبيقات‪.‬‬
‫‪ ‬األداء‪ :‬يحسن األداء بشكل كبير‪.‬‬
‫‪ ‬الكلفة‪/‬األداء‪ :‬الكلفة عالية جدا ً‪ ,‬لكن التحسين في األداء يبرر ذلك‪.‬‬
‫‪ ‬الذاكرة‪ :‬تتطلب التطبيقات ذاكرة أكبر من تلك المعنونة من قبل معالج وحيد‪.‬‬
‫‪ ‬تستفيد معظم الخوارزميات الهامة من التنفيذ المتوازي‪ ,‬مثل معالجة الصورة ومعالجة اإلشارة والذكاء‪.‬‬
‫‪ ‬الحياة نفسها فيها الكثير من المعالجات المتوازية‪ ,‬مثل جداول تدفق المياه والطرق‪ ,‬وبالتالي العلماء‬
‫يحاكون الطبيعة من خالل استخدام نماذج شبيهة‪.‬‬

‫‪Dr. Eng. Marwa DAHDOUH‬‬

 ‫مشكلة تنظيم الكتب على الرفوف حسب تصنيفها‬
A single worker trying to stack all the books in their proper places cannot accomplish the task
faster than a certain rate. We can speed up this process, however, by employing more than one
worker.
Solution 1:
Assume that books are organized into shelves and that the shelves are grouped
into bays.
One simple way to assign the task to the workers is:
◦ To divide the books equally among them. Each worker stacks the books one a time.
This division of work may not be most efficient way to accomplish the task since
◦ The workers must walk all over the library to stack books.
Dr. Eng. Marwa DAHDOUH
 ‫مشكلة تنظيم الكتب على الرفوف حسب تصنيفها‬
Solution 2:
An alternative way to divide the work is to assign a fixed and disjoint set of bays
to each worker.(task partitioning)
As before, each worker is assigned an equal number of books arbitrarily.
◦ If the worker finds a book that belongs to a bay assigned to him or her.
◦ He or she places that book in its assignment spot.
◦ Otherwise,
◦ He or she places it on to the worker responsible for the bay it belongs to.(communication task)

The second approach requires less effort from individual workers.

Dr. Eng. Marwa DAHDOUH

 ‫بعض المفاهيم الخاطئة المتعلقة بالحوسبة التفرعية‬
‫‪ ‬تحتاج إلى لغات تفرعية جديدة‪ :‬يمكن استخدام اللغات القياسية مثل ‪ Fortran, java, c++‬مع العلم أن هناك بعض‬
‫اللغات التفرعية الخاصة مثل ‪.Qlisp, Mul-T‬‬

‫‪ ‬تحتاج كود جديد‪ :‬معظم الكود البرمجي يتم استخدامه كما هو‪ ,‬وقد يحتاج أحيانا ً إلى تعديل بسيط‪.‬‬

‫‪ ‬تحتاج ملحقات تفرعية مربكة‪ :‬هذا األمر ليس سيئا ً‪ ,‬وذلك اعتمادا ً على تعقيد المسألة‪ .‬فمن عمل ال شيء على اإلطالق‬
‫إلى أن تقوم بتثبيت السيمافورات بنفسك‪.‬‬

‫‪ ‬الحوسبة التفرعية صعبة‪ :‬ليس كذلك وإنما مختلفة وممتعة‪.‬‬

‫‪Dr. Eng. Marwa DAHDOUH‬‬

Problems are parallelizable to different degrees
For some problems, assigning partitions to other processors might be
more time-consuming than performing the processing locally.

Other problems may be completely serial.

For example, consider the task of digging a post hole.
 Although one person can dig a hole in a certain amount of time,

 Employing more people does not reduce this time.

Dr. Eng. Marwa DAHDOUH
 :‫فعالية وأمثلية الخوارزميات التفرعية‬
A parallel algorithm is efficient if:
◦ It is fast
◦ The product of the parallel time and number of processors is close to the time
of at the best know sequential algorithm:

T_sequential= T_parallel * N_processors

A parallel algorithms is optimal if this product is of the same order
as the best known sequential time.

Dr. Eng. Marwa DAHDOUH

 :‫الحوسبة التفرعية مقابل الموزعة‬
Parallel: several processing elements concurrently solving a single same problem.
Distributed: processing elements do not share memory or system clock.

The Demand for Computational Speed

Continual demand for greater computational speed from a computer system than is currently
possible
Areas requiring great computational speed include numerical modeling and simulation of
scientific and engineering problems.
Computations must be completed within a "reasonable" time period.

Dr. Eng. Marwa DAHDOUH

 Grand Challenge Problems
One that cannot be solved in a reasonable amount of time with today’s
computers. Obviously, an execution time of 10 years is always unreasonable.

Examples:
◦ Global weather forecasting.
◦ Modeling motion of astronomical bodies.
◦ Modeling large DNA structures.

Dr. Eng. Marwa DAHDOUH

 Global Weather Forecasting Example
Atmosphere modeled by dividing it into 3-dimensional cells. Calculations
of each cell repeated many times to model passage of time.
Suppose whole global atmosphere divided into cells of size 1 mile  1 mile  1 mile to a height
of 10 miles (10 cells high) - about 5  108 cells.
Suppose each calculation requires 200 floating point operations.
In one time step, 1011 floating point operations necessary.
To forecast the weather over 7 days using 1-minute intervals, a computer operating
at 1Gflops (109 floating point operations/s) takes 106 seconds or over 10 days.
To perform calculation in 5 minutes requires computer operating at 3.4 Tflops (3.4
 1012 floating point operations/sec).
Dr. Eng. Marwa DAHDOUH
 Modeling Motion of Astronomical Bodies
Each body attracted to each other body by gravitational forces. Movement of each body
predicted by calculating total force on each body.
With N bodies, N - 1 forces to calculate for each body, or approx. N2 calculations. (N log2N for
an efficient approx. algorithm.)
After determining new positions of bodies, calculations repeated.

A galaxy might have, say, 1011 stars. Even if each calculation done in 1 ms, it takes
109 years for one iteration using N2 algorithm and almost a year for one iteration
using an efficient N log2N approximate algorithm.

Dr. Eng. Marwa DAHDOUH

 Parallel Computing
Using more than one computer, or a computer with more than one
processor, to solve a problem.
Motives
Usually faster computation - very simple idea - that n computers
operating simultaneously can achieve the result n times faster.
Other motives include: fault tolerance, larger amount of memory
available, ...

Dr. Eng. Marwa DAHDOUH

Introduction to Parallel Programming

Dr. Eng. Marwa DAHDOUH

 Process
Thread

Dr. Eng. Marwa DAHDOUH

Dr. Eng. Marwa DAHDOUH
Dr. Eng. Marwa DAHDOUH
 Synchronization
Ideal Time

Dr. Eng. Marwa DAHDOUH

Dr. Eng. Marwa DAHDOUH
 ‫التنظيمات متعددة المعالجات‪ :‬أنواع أنظمة الحاسوب وفق العالم ‪:Flynn‬‬

‫‪Dr. Eng. Marwa DAHDOUH‬‬

Flynn’s Classifications

classification for computers

based upon instruction
streams and data streams.

Dr. Eng. Marwa DAHDOUH

Flynn’s Classifications

Single instruction stream - single data stream (SISD) computer

Single processor computer - single stream of instructions generated from
program. Instructions operate upon a single stream of data items.

Dr. Eng. Marwa DAHDOUH

 Single Program Multiple Data (SPMD) Structure
Single source program written and each processor executes its
personal copy of this program, although independently and not in
synchronism.

Source program can be constructed so that parts of the program are

executed by certain computers and not others depending upon the
identity of the computer.

Dr. Eng. Marwa DAHDOUH

 Single Instruction Stream - Multiple Data Stream (SIMD) Computer

A specially designed computer - a single instruction stream from a single

program, but multiple data streams exist. Instructions from program broadcast to
more than one processor. Each processor executes same instruction in
synchronism, but using different data.

Dr. Eng. Marwa DAHDOUH

‫أكثر من تسلسل من التعليمات تطبق على نفس التسلسل من المعطيات ومثالها مجموعة من المرشحات التي تطبق عليها إشارة‬
‫وحيدة‪.‬‬

‫‪Dr. Eng. Marwa DAHDOUH‬‬

 Multiple Instruction Stream Multiple Data Stream
(MIMD) Computer
General-purpose multiprocessor system - each processor has a
separate program and one instruction stream is generated from each
program for each processor. Each instruction operates upon different
data.

Both the shared memory and the message-passing multiprocessors

so far described are in the MIMD classification.

Dr. Eng. Marwa DAHDOUH

Dr. Eng. Marwa DAHDOUH
 ‫تعتبر تطبيقات الحوسبة وفق تمرير الرسائل من األمثلة عليها حيث تقوم كل عقدة بتطبيق تسلسل تعليمات خاص بها‬
‫على المعطيات المحلية التي تملكها‪ .‬وعلى عكس األنواع األخرى فإن حواسيب ‪ MIMD‬تعمل بشكل متزامن‪.‬‬
‫‪Dr. Eng. Marwa DAHDOUH‬‬
 Multiple Program Multiple Data (MPMD) Structure
Within the MIMD classification, each processor will have its own program to
execute:
Program Program

Instructions Instructions

Processo r Processo r

Data Data
Dr. Eng. Marwa DAHDOUH