Syllabus
Course Description
The course will primarily consist of two components: 1) fundamental theories and 2) recent flagship applications/algorithms. First, the course will introduce the basic theoretical foundation of reinforcement learning, including Markov Decision Process, Bellman’s principle of optimality, value iteration, policy iteration, and etc. Then, build upon this foundation, we will introduce various flagship algorithms that contribute greatly to the evolution of modern reinforcement learning, including Deep Q Networks, Advantage Actor-Critic, Proximal Policy Optimization, AlphaZero, and so on. After finishing this course, you will 1) be able to understand majority of research papers in this field and 2) also be able to apply (or even design) RL algorithms to solve practical problems.
Prerequisite
(MA381 and CSSE 220) or (Talk with the instructor to get permission)
Many key concepts in the course involve probability knowledge. It is required for students to have solid understanding on basics of probability, e.g, gaussian/normal distribution, conditional probability, expectation and so on. In addition, due to that the assignments may include fairly complicated coding tasks (to implement RL algorithms from scratch), students need to have strong independent programming skill. However, if you don’t meet the prerequisite course requirement but have equivalent experience, feel free to talk with the instructor to get a special permission.
Instructor
Lixing Song
Moench F212
877-8345 (Office)
Textbook
No textbook is required. The course will heavily leverage existing online materials. The following is a list of useful online materials:
Materials:
Textbooks:
- Deep Reinforcement Learning Hands-On: Apply modern RL methods to practical problems of chatbots, robotics, discrete optimization, web automation, and more, 2nd Edition by Maxim Lapan
- Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto
Online materials:
- CS 285 Deep Reinforcement Learning at UC Berkeley Deep Reinforcement Learning taught by Sergey Levine
- UCL Course on RL taught by David Silver
- OpenAI Spinning Up
Laboratories
The design of this course is a practice of Richard Feynman’s famous quote: What I cannot create, I do not understand. The course will involve heavy hands-on experience. For each week, there will be one class periods dedicated as lab hours (depending on the subject). In addition to the introductory lab(s) that will equip you with the basics of deep learning on Pytorch, the labs will guide you to implement the flagship algorithms from scratch.
Paper Reading
As upper-class elective course, another important aspect of this course is to cultivate students’ research ability. As one of the core components of research, reading research papers can not only equip researchers with necessary literature backgroup but also often inspire researchers to explore new problems and methodologies. Therefore, this course will include several paper reading/presenting assignments. Beyond that, in coding assignments, there are opportunities for students to explore novel approaches in recent papers .
Paper Implementation
As the term project, students will have chance to implement a research paper from scratch. It is expectedly challenging but very rewarding. As the deliverables, students should give an in-class presentation plus live demo. The final submission should include complete source code with proper documentation.
Grading
| Item | Weight |
|---|---|
| Labs (6 - 7) | 70% |
| Paper Reading | 15% |
| Paper Implementation | 15% |
IMPORTANT: Missing any TWO assignments will automatically fail you in this class.
Generally, 90-100% is an A, 80-89% is a B, etc.