Physics First Reinforcement Learning for Cassie

Reinforcement learning has enabled some exceptional gait behaviors on legged robots, exceeding more established approaches in certain ways. In this talk I will describe the recent work from Jonathan Hurst’s Dynamic Robotics Lab on learned control for Cassie. We approach the learning problem from a physics first perspective, which led us to a reward formulation that is based on careful definition of the dynamics of legged locomotion, including footfall timing. This single reward structure was able to generate all common bipedal gaits - standing, walking, hopping, running, and skipping, as well as smooth gait transitions. Further, by carefully engineering the distribution of training terrains we were able to train controllers that scale stairs with no perception information or prior knowledge of the terrain. These controllers are particularly interesting to analyze from a pseudo-biomechanics viewpoint because much of the behavior is emergent. Notably, we observe familiar leg retraction strategies, efficiency/robustness tradeoffs and ground reaction force profiles.

Kevin Green