EE 599: Systems for Machine Learning

Instructor: Arash Saifhashemi | Contact:

University of Southern California

Ming Hsieh Department of Electrical and Computer Engineering

Course Description

This course offers a comprehensive exploration of systems for machine learning (ML), focusing on the latest research and advancements. It introduces students to essential models and techniques, including Convolutional Neural Networks (CNNs), Large Language Models (LLMs), gradient descent, and Stochastic Gradient Descent (SGD). We'll dive into optimization techniques like quantization and pruning to enhance model efficiency without sacrificing performance.

Beyond theoretical concepts, the course provides an introduction to Deep Neural Network (DNN) accelerator design and the architecture of GPUs and TPUs. These sections aim to give students a foundational understanding of how ML algorithms leverage parallel computing architectures, such as chip multiprocessors, multithreading, and GPUs with CUDA support, for improved performance and efficiency.

Throughout the course, we emphasize optimizing ML performance while considering power efficiency and technical constraints. Our goal is to equip students with the knowledge and skills necessary to navigate and contribute to the field of machine learning systems effectively.

Learning Objectives

  • Gain a solid foundation in machine learning systems, including an overview of various models and techniques such as CNNs (Convolutional Neural Networks), LLMs (Large Language Models), gradient descent, and Stochastic Gradient Descent (SGD).
  • Introduction to DNN (Deep Neural Network) accelerator design and the architecture of GPUs/TPUs, focusing on how these hardware systems accelerate ML tasks.
  • Develop skills to map ML algorithms on GPUs and TPUs to optimize performance, using CUDA for effective programming on NVIDIA GPUs.
  • Understand the importance of power efficiency and constraints in ML systems, aiming to optimize computational resources and energy usage.
  • Learn about model optimization techniques such as quantization and pruning, which enhance performance and efficiency by adjusting the precision of model parameters and reducing neural network complexity, respectively.
  • Explore the privacy and security challenges inherent in ML systems, including strategies to protect data and ensure the integrity of models against potential attacks.
  • Introduction to federated learning, emphasizing its role in training ML models across decentralized devices to enhance data privacy and security.

Course Schedule

Week Topics/Daily Activities Readings/Preparation Deliverables
1 Introduction and applications Programming/Homework assignment (PHA)#1 Posted
2 An overview of ML algorithms and their computational demands Programming/Homework assignment (PHA)#2 Posted
3-5 CNNs, ML runtime systems (PyTorch, TensorFlow), Deep dive in DNN accelerators Reading Assignment 1 posted Programming/Homework assignment (PHA)#3 Posted
6 Kernel Computation. Toeplitz matrices Reading Assignment 2 posted
7-8 Dataflows (weight, input, and output stationary systems), loop nests Project topics posted, PHA#4 posted, Project initial updates
9 Midterm Exam, Basics of GPU and TPU microarchitecture. Mid project updates
10-11 More on GPU and TPU microarchitecture. PHA#5 posted
12 Distributed Learning PHA#6 posted
13 Privacy in Deep Learning
14 Introduction to NLP and Language Models Final project updates
15 RNNs, LSTM, and Transformers, LLMs
16 Generative models, Backup for overflow, Final Exam Refer to the final exam schedule in the USC Schedule of Classes at classes.usc.edu.

Technological Proficiency and Hardware/Software Required

This course requires basic proficiency in Python, a primary language used in machine learning. You'll need a computer capable of running ML algorithms and simulations. We'll use ML libraries and frameworks such as TensorFlow and PyTorch. Familiarity with tools like Anaconda or Jupyter Notebook will aid your coding and experimentation.

Students must have a GitHub account for version control and code submissions. For those new to GitHub, here's a GitHub Tutorial to get you started.

For hands-on coding without a high-performance computer, Google Colab is recommended. It's a cloud-based platform that supports Python code execution, with access to powerful GPUs and TPUs.

Explore these resources to get started:

Required Readings and Supplementary Materials

  • Efficient Processing of DNNs by Sze, Chen, Yang, and Emer.
  • Eyeriss Mit Tutorial.
  • Optional Reading: Parallel Computer Organization and Design by Michel Dubois, Murali Annavaram, and Per Stenström. This textbook provides background material on chip multiprocessors and discussions relevant to the course.

Grading Breakdown

Assessment Tool Points % of Grade
Midterm and Quizzes 20 20%
Programming, reading, and homework assignments 40 40%
Project 20 20%
Final exam 20 20%

Grading Scale

Letter Grade Numerical Point Range
A 95-100
A- 90-94
B+ 87-89
B 83-86
B- 80-82
C+ 77-79
C 73-76
C- 70-72
D+ 67-69
D 63-66
D- 60-62
F 59 and below

Course Policies

Academic Integrity

Academic integrity is foundational to our learning community. This course follows USC's standards on academic integrity. Students are expected to submit original work and properly attribute sources. Violations, including plagiarism and cheating, will result in disciplinary action, which may include failure in the course and further sanctions by the university.

Attendance

Attendance is mandatory and integral to the success of all students. Absences can impact your final grade. If you must miss a class, please notify the instructor ahead of time. Accommodations for student athletes and religious observances will be made.

Zoom Etiquette

For classes held via Zoom, maintain professionalism in all communications. Keep your camera on if required, participate actively, and use features like raise hand and chat responsibly. Ensure your microphone is muted when not speaking to minimize background noise.

Assignment Submission Policy

All assignments must be submitted electronically. Ensure your submissions are complete and include any required documentation or README files. Assignments will undergo both automated and manual review to ensure integrity.

Support and Accommodations for Students with Disabilities

USC is committed to inclusivity and provides support for students with disabilities. If you require accommodations, please register with the Office of Student Accessibility Services (OSAS) and inform the instructor to make necessary arrangements.

Support Systems

The university offers various support systems, including counseling, mental health services, and resources for addressing bias or harassment. For more information or assistance, please refer to the following resources: