Session 5: MPI Communication
Summary so far…
• Distributed systems and cluster computing
• Nodes, NICs, switches, topologies
• Communication requirements in DL and HPC applications
• Nearest neighbour communication and collectives
• Collective communication
• Allreduce, all2all, reducescatter, allgather
MPI Standard
• 1.0: Communicators, P2P and collective communication
• 2.0: one-sided communication, parallel I/O, …
• 3.0: non-blocking collectives, …
https://skirt.ugent.be/skirt9/_parallel_messaging.html https://hpc-wiki.info/hpc/MPI
MPI program
https://skirt.ugent.be/skirt9/_parallel_messaging.html
MPI communicators
https://skirt.ugent.be/skirt9/_parallel_messaging.html
MPI Point-to-Point
https://skirt.ugent.be/skirt9/_parallel_messaging.html
MPI Point-to-Point: Blocking send and recv
https://skirt.ugent.be/skirt9/_parallel_messaging.html
MPI Point-to-Point: Non-blocking send and recv
https://skirt.ugent.be/skirt9/_parallel_messaging.html
MPI Collective communication
https://skirt.ugent.be/skirt9/_parallel_messaging.html
Buffering
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture23.pdf
Avoiding buffering
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture23.pdf
What is message passing?
• Data transfer plus synchronization
• Requires cooperation of sender and receiver
• Cooperation not always apparent in code
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture24.pdf
Message passing terms and protocols
• Non-blocking: operation does not wait for completion
• Synchronous: Completion of send requires initiation (but not
completion) of receive
• Asynchronous: communication and computation take place
simultaneously
• Message consists of “envelope” and data
• Envelope contains tag, communicator, length, source
information, private data
• Message protocols
• Eager (message data assuming destination can store)
• Rendezvous (message not sent until destination Oks)
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture24.pdf
Eager Protocol
• Data delivered to process 1
• No matching receive may exist; Process 1 must then buffer and copy
• Reduces synchronization delays
• Significant buffering required
• May require active requirement of CPU
• Additional copies
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture24.pdf
Rendezvous Protocol
• Enveloped sent first; data sent when user-buffer available
• Robust and safe
• May remove copies
• More complex
• May introduce synchronization delays
https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture24.pdf
