Introduction
Recent advances in GPU-based physics simulation and deep Reinforcement Learning (RL) have enabled the rapid training of control policies for complex, highly articulated robots such as quadrupeds and humanoids. Parallelized simulators now make it possible to obtain policies that transfer to hardware in a matter of minutes of simulated experience [1]. Despite this progress, a significant discrepancy often remains between performance in simulation and on the real robot, commonly referred to as the Sim2Real gap.
This gap arises from several sources along the RL control pipeline [2, 3, 4]. First, the reality gap captures mismatches between simulated and real-world dynamics (e.g., contact properties, friction, or unmodeled compliance). Second, the perception gap stems from idealized sensing in simulation, where cameras and LiDARs are often noise-free and perfectly calibrated. Both aspects are important but lie outside the scope of this project.
This work instead focuses on a third, critical component: dynamics, control, and actuation. Robot models used in simulation are typically defined through description files that encode joint types, link inertias, and other physical parameters. These models are often provided by manufacturers but rarely match the real system perfectly, leading to modeling errors in the rigid-body dynamics. An even more pronounced source of mismatch comes from the actuators themselves. Real actuators (e.g., hydraulic drives or electric motors) exhibit nonlinear, non-smooth dissipation, cascaded feedback loops, and internal states that are not directly observable [1, 5, 6]. Accurately capturing these effects is key to narrowing the Sim2Real gap for RL-based controllers.
Objectives
The goal of this project is to study and improve data-driven actuator models for legged robots, with a focus on learning neural network (NN) models that map commanded joint quantities (e.g., desired position or torque) and state histories to realized joint torques. Building on prior work [1, 5, 6], the student will:
- Implement and validate several NN-based actuator modeling methods from the literature. These methods typically learn a model for a single actuator using real-world data, and the resulting model is then shared across all identical actuators (e.g., the 12 motors of a quadruped).
- Systematically evaluate the reproduced methods, including simple variations such as architecture changes, input histories, and hyperparameter tuning. Compare their ability to reduce the Sim2Real gap, for example by measuring prediction error on held-out data and downstream performance of RL policies.
- Move beyond single-motor models by learning a joint model for all actuators simultaneously (e.g., a network that outputs 12 torques at once). Analyze whether such models capture coupling effects between actuators and how they compare to independent single-motor models in terms of accuracy and transfer.
- Optional/Bonus: Explore alternative modeling strategies (e.g., structured networks, latent-state models, or hybrid physics–ML approaches). If a novel method significantly improves the fidelity of the actuator model and leads to better real-world performance, the work may be extended into a thesis project with potential for publication.
Contact
Georges Jetti: georges.jetti@polimi.it
Michael Khayyat: michael.khayyat@polimi.it
Stefano Arrigoni: stefano.arrigoni@polimi.it
References
[1] Rudin, Nikita, et al. “Learning to walk in minutes using massively parallel deep reinforcement learning.” Conference on robot learning. PMLR, 2022.
[2] Aljalbout, Elie, et al. “The Reality Gap in Robotics: Challenges, Solutions, and Best Practices.” Annual Review of Control, Robotics, and Autonomous Systems 9 (2025).
[3] He, Tairan, et al. “Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills.” arXiv preprint arXiv:2502.01143 (2025).
[4] Aljalbout, Elie, et al. “On the role of the action space in robot manipulation learning and sim-to-real transfer.” IEEE Robotics and Automation Letters 9.6 (2024): 5895-5902.
[5] Hwangbo, Jemin, et al. “Learning agile and dynamic motor skills for legged robots.” Science Robotics 4.26 (2019): eaau5872.
[6] Bjelonic, Filip, Fabian Tischhauser, and Marco Hutter. “Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robots.” arXiv preprint arXiv:2509.06342 (2025).