Parallel and Distributed Programming
Kenjiro Taura

1 What’s new (in the newest-first order)

• (Posted: Oct. 06, 2024) Plan for Oct. 07
  • make sure you can see the course page in UTOL and check the comments from the instructor of “Assignment 0: Jupyter password”
  • Introduction
  • Play with Jupyter
  • pd00_intro
  • OpenMP
  • pd01_omp
• (Posted: Sep. 28, 2024) Site up

2 Slides

1. Introduction
2. OpenMP
3. CUDA
4. SIMD
5. How to get nearly peak FLOPS (with CPU)
6. What You Must Know about Memory, Caches, and Shared Memory
7. OpenMP for GPU
8. Divide and Conquer
9. Neural Network Basics
10. Analyzing Data Access of Algorithms and How to Make Them Cache-Friendly?
11. Understanding Task Scheduling Algorithms

3 Languages

• All written materials (slides, home pages, etc.) will be in English
• Lectures will be in English

4 Hands-on programming exercise

• You will have an access to latest CPU and GPU machines and hands-on experiences on parallel programming
• This year, I emphasize a programming model targetting both CPUs and GPUs (OpenMP + GPU offloading)

5 How to get the credit

1. participate in in-class exercise (some may be group work)
2. submit programming exercise assignments
3. write and submit a final report (term paper).
   • abstract deadline: (late December or early January)
   • final deadline: (February, a few weeks after the course)
   • options for the final report (details will be announced later)
     1. parallelize/optimize a common predefined problem
     2. solve your problem

6 Topics covered

• Parallel Programming in Practice
  • It’s easy! — a quick and gentle introduction to parallel problem solving
  • Some examples of parallel problem solving and programming
• Taxonomy of parallel machines and programming models
  • What today’s machines look like — parallel computer architecture
    • Distributed memory machines
    • Multi-core / multi-socket nodes
    • SIMD instructions
  • Parallel programming models
    • Finding and expressing parallelism
    • Mapping computation onto compute resources
    • Coordination and communication
    • Examples of parallel programming languages/models
• Understanding performance of parallel programs (and achieving high performance)
  • The maximum performance of your CPU/GPU and why you don’t get it for your program?
  • The maximum performance of memory and why you don’t get it for your program?
  • How to reason about memory traffic of your programs
  • Provable bounds of greedy schedulers
  • Provable bounds of work-stealing schedulers
  • Cache miss bounds of work-stealing schedulers