1. How to read papers

Watch this wonderful video lecture from Dr. Andrew Ng. It is very insightful that teach you not only reading research papers but also some wisdom on career advice.

Here is a summary article on the lecture.

2. Papers pool

I selected the following interesting research papers as something easy to read. In addition to the original paper, I also provide some related articles/tutorials. I encourage you to read the papers and use the other tutorials just to enhance your learning.

Phastic Policy Gradient

Dueling Network Architectures for Deep Reinforcement Learning

Curiosity-driven Exploration by Self-supervised Prediction

Unifying Count-Based Exploration and Intrinsic Motivation

Model-Based Value Expansion for Efficient Model-Free Reinforcement Learning

Model-Based Reinforcement Learning via Meta-Policy Optimization

Feel free to send me RL papers you think are interesting and easy to read (i.e., with great writing quality).

Paper Presentation

Each group will be assigned one paper from the pool above to present. You need to carefully read the paper assigned (most likely multiple passes) and prepare a slide deck to present the paper in the class.

The presentation time is 20 minutes + discussion. All members of your group should participate in the presentation. Ideally, the presentation time should be equally divided among group members, i.e., if you have 3 people in your group, each should present around 6 (20/3) minutes. In the presentation, you need to cover the following questions:

  • What problem(s) the paper proposes to solve?
  • Why the proposed method is novel, i.e., different from previous approaches?
  • What is the motivation to come up with the new idea, i.e., the intuition/philosophy behind the idea?
  • What kinds of experiments the paper conducts to evaluate/validate the proposed method?
  • What other experiments do you think are interesting/necessary but the paper doesn’t include?
  • Your critic on the paper.

Paper Presentation Rubrics

All the presentations will be evaluated by other students in this class. The following rubric items will be used as guidance for the audience to evaluate the presentation. As presenter(s), you want to make sure your presentation provides solid coverage on the various aspects mentioned below.

Item Points
Motivation: Is the problem (and its significance) that the paper tries to solve well presented? 20
Clarity: Is the general idea (e.g., non-math part) proposed in the paper clear to you based on the presentation? 30
Depth: Does the presentation provide a clear explanation of the mathematical models involved in the paper? 30
Discussion: Do you think the presenter(s) provide correct/proper answers to the questions raised by the audience? 20

Final Presentation

Different from the paper reading presentation, the final presentation will focus more on result presenting. The time for final presentation is 15 min + discussion. The presentation should answer the following questions:

  • How did you you implement the method?
    • What neural nets are included?
    • What are the loss functions to optimize each of the neural nets?
    • How to update policy/value networks?
  • What are the major challenges to write/debug of code?
  • How is the performance of the implemented method?
    • How does it compare with the baseline(s), e.g., an algorithm we implemented in this class.
    • Given the experiment results (e.g., losses, reward), how do you interpret the differences or similarities?
  • What are the future directions you can see to improve this work?

Final Presentation Rubrics

The final presentation will be evaluated by the instructor. The following rubric items will be used to evaluate the presentation.

Item Points
Implementation Effort: What is the difficulty to implement this method ? 20
Correctness: Are the various models/neural nets correctly trained? 40
Performance: Does the experiment result match expectation? 30
Vision: Does the presentation propose reasonable future works? 10