Before CUDA: Learning to See Parallelism
I first got introduced to CUDA programming back in early 2024 when I decided to take the course "How to Write Fast Code II"(I know very chic) in grad school. Since then, I have worked quite a bit in that area for work and while the field has gained a lot more traction in the recent past, there isn't a lot of discourse with regards to parallel programming in a general sense. Of course there are a lot of great courses and papers out there but I am trying to see if I can get you, the reader, to get your mental model of parallel programming right, before you dive first into the world of GPUs, CUDA, ROCm and what not.
Parallel Thinking
Given a loop, can you tell whether each iteration is independent? Given an accumulation, can you rewrite it as a tree? Given a prefix sum, can you see the scan hiding inside it? Given a histogram, can you spot the race condition before it happens? This is what I call parallel thinking.