Parallel and Distributed Programming
Kenjiro Taura

What’s new (in the newest-first order)

• (Posted: Sep. 28, 2026)
  • Go to the UTOL course page and submit “Assignment 0: send info to issue your account for exercise environment”
  • If you cannot see the assignment, self-register for the course by pressing the “register a course” button (UTOL Manual for Students p.37)
  • In case you cannot see the course page above, use “Search Course” with:
    • Keyword: parallel and distributed programming
    • Term: All
• (Posted: Sep. 28, 2026) The site is up!

Slides

1. Introduction
2. OpenMP
3. CUDA
4. OpenMP for GPU
5. SIMD
6. How to get nearly peak FLOPS (with CPU)
7. What You Must Know about Memory, Caches, and Shared Memory
8. Analyzing Data Access of Algorithms and How to Make Them Cache-Friendly
9. Divide and Conquer
10. Neural Network Basics
11. Understanding Task Scheduling Algorithms

Languages

• All written materials (slides, home pages, etc.) will be in English
• Lectures will be in English

Hands-on programming exercise

• You will have an access to latest CPU and GPU machines and hands-on experiences on parallel programming
• This year, I emphasize a programming model targetting both CPUs and GPUs (OpenMP + GPU offloading)

How to get the credit

• Jan. 6 Details released in a separage page

Topics covered

• Parallel Programming in Practice
  • It’s easy! — a quick and gentle introduction to parallel problem solving
  • Some examples of parallel problem solving and programming
• Taxonomy of parallel machines and programming models
  • What today’s machines look like — parallel computer architecture
    • Distributed memory machines
    • Multi-core / multi-socket nodes
    • SIMD instructions
  • Parallel programming models
    • Finding and expressing parallelism
    • Mapping computation onto compute resources
    • Coordination and communication
    • Examples of parallel programming languages/models
• Understanding performance of parallel programs (and achieving high performance)
  • The maximum performance of your CPU/GPU and why you don’t get it for your program?
  • The maximum performance of memory and why you don’t get it for your program?
  • How to reason about memory traffic of your programs
  • Provable bounds of greedy schedulers
  • Provable bounds of work-stealing schedulers
  • Cache miss bounds of work-stealing schedulers