DS551/CS525 - Reinforcement Learning - Fall 2023

Version: June 24th, 2023

Tentative Schedule:

-1. Week 1 (8/29 T):

Topic: Overview of Reinforcement Learning and Class Logistics
Readings: N/A

-2. Week 2 (9/5 T):

Topic: RL components, Markov Decision Process, Model-based Planning..
Optional readings: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition Sec 3.1-3.6, 4.1-4.4.
Note: Project 1 starts.

-3. Week 3 (9/12 T):

Topic: Model-free Policy Evaluation..
Optional readings: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition Sec 5.1-5.3, 6.1-6.6.

-4. Week 4 (9/19 T):

Topic: Model-free Control. .
Note: Quiz 1 on Markov Decision Process and Model-based Control (30mins).
Optional readings: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition Sec 5.1-5.3, 6.1-6.6.
Note: Project 1 due.

-5. Week 5 (9/26 T):

Topic: Value Function Approximation. .
Optional readings: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition Sec 9.1-9.4.
Note: Project 2 starts.

-6. Week 6 (10/3 T):

Topic: Review of Deep Learning.
Topic: Deep Reinforcement Learning.
Optional readings: Mnih, Volodymyr, et al., Playing Atari with Deep Reinforcement Learning, arXiv preprint arXiv:1312.5602 (2013).
Note: Quiz 2 on Model-free Policy Evaluation.

-7. Week 7 (10/10 T):

Topic: Advanced Deep Reinforcement Learning by Prof Li, and Deep Learning Implementation in Pytorch (by TA).
Optional Reading #1: [AAAI 2016, Double DQN] Deep Reinforcement Learning with Double Q-learning, Hado van Hasselt and Arthur Guez and David Silver Google DeepMind https://arxiv.org/pdf/1509.06461.pdf.
Optional Reading #2: [ICLR 2016] PRIORITIZED EXPERIENCE REPLAY, Tom Schaul, John Quan, Ioannis Antonoglou and David Silver Google DeepMind https://arxiv.org/pdf/1511.05952.pdf.
Optional Reading #3: [ICML 2016, Dueling DQN] Dueling Network Architectures for Deep Reinforcement Learning, Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, Nando de Freitas https://arxiv.org/pdf/1511.06581.pdf.
Optional Reading #4: [AAAI 2018, Rainbow] Rainbow: Combining Improvements in Deep Reinforcement Learning, Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, AAAI 2018, https://arxiv.org/pdf/1710.02298.pdf.
Note: Quiz 3 on Model-free Control.
Note: Project 2 due.
Note: Project 3 starts.

-8. Week 8 (10/17 T): No Class; Fall Break

Fall Break

-9. Week 9 (10/24 T): .

Topic: Advanced DQNs (Continued) and Inverse Reinforcement Learning and Imitation learning..
Note: Quiz 4 on linear function approximation for policy evaluation and Control.
Note: We will have an inclass selfintroduction session, so you can start forming a team for project 4.

-10. Week 10 (10/31 T):

Topic: Imitation Learning (Continued!) and Policy as a Deep Neural Network: Policy Gradient Reinforcement Learning.
Optional readings: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition Chapter 13.
optional Reading: Policy Gradient RL algorithms (a good and comprehensive blog) (For this class, reading the sections we covered is sufficient.)
Note: Project 4 starts.

-11. Week 11 (11/7 T):

Topic: Policy Gradient RL (continued) (See the slides from last week.)
Topic: Advanced Policy Gradient (PPO, TRPO, PPO2) (continued), Actor-Critic Approaches (A2C, A3C, Pathwise Derivative PG), Sparse Reward, Hierarchical RL..
Optional Reading #1: [TRPO] https://arxiv.org/pdf/1502.05477.pdf
Optional Reading #2: [PPO] https://arxiv.org/pdf/1707.06347.pdf
Note: Project 4 Proposal due.

-12. Week 12 (11/14 T):

Topic: Actor-Critic Approaches (A2C, A3C, Pathwise Derivative PG) (continued), Sparse Reward, Hierarchical RL..
Note: Project 3 due.
Note: Quiz 5 on policy gradient (including Basic PG, REINFORCE PG, and Vanilla PG).

-13. Week 13 (11/21 T):

Topic: Multi-Agent RL (MARL), and DeepMind AlphaTensor.
Optional Readings: DDPG, MA-DDPG, AlphaTensor.
Note: Project #4 Progressive Report is due. Please submit it to Canvas discussion board in teams.

-14. Week 14 (11/28 T):

Topic:Deep Inverse Reinforcement Learning, Multi-agent IRL, Meta-RL, and Class Review..
Optional Readings: Meta-RL, GAIL, MA-GAIL.

-15. Week 15 (12/5 T):

Topic: Project #4 Presentations.
Note: Project 4 due.