Main memory has emerged as a critical bottleneck resource in modern datacenters. Memory capacity and bandwidth bottlenecks have resulted in unsustainable costs for datacenter operators and severe performance impact for modern applications (e.g., key-value stores, data analytics, graph processing). Memory bottlenecks are even more acute for AI applications running on GPUs (e.g., LLM inference). In this course, we will explore approaches to mitigate and manage datacenter memory bottlenecks across the hardware and software stack spanning operating systems, distributed systems and computer architecture.
This seminar-style course will cover a wide range of topics, including but not limited to: OS memory management, memory offloading, tiering and disaggregation, address translation optimizations, memory bandwidth management, LLM inference memory management, and new memory technologies. Topic discussions will primarily be guided by recent research papers in top-tier OS, distributed systems and computer architecture conferences.
This course is primarily intended for PhD students in computer science who are interested in pursuing research in one or more of the operating systems, distributed systems and computer architecture areas.
MS students and undergraduates interested in the course material are also welcome to join. The following prerequisites are required for undergrads: a grade of B or higher in CS 35400.
Note that the below schedule and reading list is subject to change. Please keep an eye on the course website and Brightspace/email announcements for updates.
| Date | Topic | Readings |
|---|---|---|
| Warmup | ||
| Aug 25 | No class | |
| Aug 27 | No class | |
| Sep 1 | Course introduction and preliminaries | |
| Classical OS memory management | ||
| Sep 3 | Linux memory management | |
| Sep 8 | NUMA memory management | |
| Sep 10 | VM memory management | |
| Memory offloading, tiering and disaggregation | ||
| Sep 15 | Saving memory capacity by offloading | |
| Sep 17 | Making swap faster and more efficient |
Required
Mage
|
| Sep 22 | OS memory tiering |
Suggested
Placeholder paper
|
| Sep 24 | Hardware-managed memory tiering | |
| Sep 29 | Memory disaggregation or pooling | |
| Virtual to physical address translation | ||
| Oct 1 | Pages vs. hugepages |
Required
Ingens
|
| Oct 6 | TLB compression |
Required
Mosaic pages
|
| Oct 8 | Address translation for VMs |
Required
DMT
|
| Oct 13 | October break | |
| Oct 15 | Address translation for IO devices |
Required
F&S
|
| Oct 20 | Mid-term exam | |
| Memory bandwidth problems | ||
| Oct 22 | Industry observations on memory bandwidth bottlenecks | |
| Oct 27 | Interplay between host interconnects |
Required
Understanding the Host Network
Suggested
hostCC
|
| Oct 29 | DDR and DRAM architecture | |
| Nov 3 | Memory controller scheduling |
Required
STFM
|
| GPU/AI memory bottlenecks | ||
| Nov 5 | Memory capacity and bandwidth limits for LLM inference |
Required
LIMINAL
|
| Nov 10 | Inference memory management: Paged attention | |
| Nov 12 | Inference memory management: Heterogeneous models |
Required
Jenga
|
| Nov 17 | Inference memory management: Long context |
Required
Contextra
|
| Nov 19 | Processing in memory for inference |
Required
CENT
|
| Looking into the future | ||
| Nov 24 | New memory technologies | |
| Nov 26 | Thanksgiving break | |
| Dec 1 | Thanksgiving break | |
| Final project presentations | ||
| Dec 3 | Project presentations | |
| Dec 8 | Project presentations | |
| Dec 10 | No class | |
The course is structured around student-led presentations and discussion held during weekly sessions, with the instructor providing guidance and facilitating exploration of the material. Course evaluation is based on two assignments, one midterm exam, and a research project.
The exact assignments and their details will be announced during the course of the class. Below is a tentative description to students a high-level idea.
Assignment 1: Students will work on an implementation exercise that will require programming in the Linux kernel and interacting with the kernel's memory management subsystem. The exercise will be sufficiently self-contained and will not require intricate knowledge of kernel internals. Students will test their implementations for correctness and conduct basic performance measurements.
Assignment 2: Students can select one of two options
Midterm: The midterm will focus on the core concepts underlying the material covered in weekly readings and discussions. Exact format will be announced during the course of the class.
Final project: Students can choose one of two options: (1) undertake an open-ended research project related to the topics of the course, done individually or perhaps in groups of two (2) conduct a detailed literature survey on a topic related to the course, done individually. In both cases, an initial proposal, and a final report (both written and presented) are required.