Control, Inference and Statistical Physics

Path integral control is a method for solving stochastic optimal control problems by using a Monte Carlo (MC) approach to approximate the Hamilton-Jacobi-Bellman (HJB) equation. It transforms the value function into an expectation over all possible trajectories (a path integral) and then uses Monte Carlo simulations to find the optimal control. This approach can be applied to control problems with moderate to large dimensions by avoiding the need for a global grid of the HJB equation.

Presentations given by Dr. Bert Kappen:

• Control theory 1 Slides Video
• Control theory 2 Slides Video
• KL Control Theory and Decision Making under Uncertainty Video
• An efficient approach to stochastic optimal control Video

Control as Inference by Sergey Levine:

• Part 1
• Part 2
• Part 3
• Part 4
• Part 5

• Todorov(2006) Linearly-solvable Markov decision problems, pdf
• Todorov(2008) General duality between optimal control and estimation, pdf
• Kappen(2009) Optimal control as a graphical model inference problem, pdf
• Ziebart(2010) Modeling Interaction via the Principle of Maximum Causal Entropy, pdf
• Rawlik, Toussaint, Vijayakumar (2013) On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference, pdf
• Nachum, Norouzi, Xu, Schuurmans(2017) Bridging the Gap Between Value and Policy Based Reinforcement Learning, pdf
• Schulman, Chen, Abbeel(2017) Equivalence Between Policy Gradients and Soft Q-Learning, pdf
• Haarnoja, Zhou, Abbeel, Levine (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, pdf
• Levine(2018) Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, pdf