DS595/CS525 - Reinforcement Learning - Fall 2022

Version: Aug 24th, 2022

Tentative Schedule:

-1. Week 1 (8/30 T):

Topic: Overview of Reinforcement Learning and Class Logistics
Readings: N/A

-2. Week 2 (9/6 T):

Topic: RL components, Markov Decision Process, Model-based Planning..
Note: Project 1 starts.

-3. Week 3 (9/13 T):

Topic: Model-free Policy Evaluation..

-4. Week 4 (9/20 T):

Topic: Model-free Control. .
Note: Quiz 1 on Markov Decision Process and Model-based Control (30mins).
Note: Project 1 due.

-5. Week 5 (9/27 T):

Topic: Value Function Approximation. .
Note: Project 2 starts.

-6. Week 6 (10/4 T):

Topic: Review of Deep Learning and Deep Reinforcement Learning.
Note: Quiz 2 on Model-free Policy Evaluation.

-7. Week 7 (10/11 T):

Topic: Advanced Deep Reinforcement Learning by Prof Li, and Deep Learning Implementation in Pytorch (by TA).
Optional Reading #1: [AAAI 2016, Double DQN] Deep Reinforcement Learning with Double Q-learning, Hado van Hasselt and Arthur Guez and David Silver Google DeepMind https://arxiv.org/pdf/1509.06461.pdf.
Optional Reading #2: [ICLR 2016] PRIORITIZED EXPERIENCE REPLAY, Tom Schaul, John Quan, Ioannis Antonoglou and David Silver Google DeepMind https://arxiv.org/pdf/1511.05952.pdf.
Optional Reading #3: [ICML 2016, Dueling DQN] Dueling Network Architectures for Deep Reinforcement Learning, Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, Nando de Freitas https://arxiv.org/pdf/1511.06581.pdf.
Optional Reading #4: [AAAI 2018, Rainbow] Rainbow: Combining Improvements in Deep Reinforcement Learning, Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, AAAI 2018, https://arxiv.org/pdf/1710.02298.pdf.
Note: Quiz 3 on Model-free Control.
Note: Project 2 due.
Note: Project 3 starts.

-8. Week 8 (10/18 T): No Class; Fall Break

Fall Break

-9. Week 9 (10/25 T): .

Topic: Advanced DQNs (Continued) and Inverse Reinforcement Learning and Imitation learning..
Note: Quiz 4 on linear function approximation for policy evaluation and Control.
Note: We will have an inclass selfintroduction session, so you can start forming a team for project 4.

-10. Week 10 (11/1 T):

Topic: Imitation Learning (Continued!) and Policy as a Deep Neural Network: Policy Gradient Reinforcement Learning.
Suggested Reading: Policy Gradient RL algorithms (a good and comprehensive blog) (For this class, reading the sections we covered is sufficient.)
Note: Project 4 starts.
Note: Project 3 due.

-11. Week 11 (11/8 T):

Topic: Policy Gradient RL (continued) (See the slides from last week.)
Note: Project 4 Proposal due.

-12. Week 12 (11/15 T):

Topic: Advanced Policy Gradient (PPO, TRPO, PPO2) (continued), Actor-Critic Approaches (A2C, A3C, Pathwise Derivative PG), Sparse Reward, Hierarchical RL..
Optional Reading #1: [TRPO] https://arxiv.org/pdf/1502.05477.pdf
Optional Reading #2: [PPO] https://arxiv.org/pdf/1707.06347.pdf
Note: Quiz 5 on policy gradient (including Basic PG, REINFORCE PG, and Vanilla PG).

-13. Week 13 (11/22 T):

Topic: Multi-Agent RL (MARL), and DeepMind AlphaTensor.
Optional Readings: DDPG, MA-DDPG, AlphaTensor.
Note: Project #4 Progressive Report is due. Please submit it to Canvas discussion board in teams.

-14. Week 14 (11/29 T):

Topic:Deep Inverse Reinforcement Learning, Multi-agent IRL, Meta-RL, and Class Review..
Optional Readings: Meta-RL, GAIL, MA-GAIL.

-15. Week 15 (12/6 T):

Topic: Project #4 Presentations.
Note: Project 4 due.