∇Sim: DIFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL
Research output: Contribution to conference › Paper › Research › peer-review
Standard
∇Sim : DIFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL . / Jatavallabhula, Krishna Murthy; Macklin, Miles; Golemo, Florian; Voleti, Vikram; Petrini, Linda; Weiss, Martin; Considine, Breandan; Parent-Lévesque, Jérôme; Xie, Kevin; Erleben, Kenny; Paull, Liam; Shkurti, Florian; Nowrouzezahrai, Derek; Fidler, Sanja.
2021. Paper presented at 9th International Conference on Learning Representations, ICLR 2021, Virtual, Online.Research output: Contribution to conference › Paper › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - CONF
T1 - ∇Sim
T2 - 9th International Conference on Learning Representations, ICLR 2021
AU - Jatavallabhula, Krishna Murthy
AU - Macklin, Miles
AU - Golemo, Florian
AU - Voleti, Vikram
AU - Petrini, Linda
AU - Weiss, Martin
AU - Considine, Breandan
AU - Parent-Lévesque, Jérôme
AU - Xie, Kevin
AU - Erleben, Kenny
AU - Paull, Liam
AU - Shkurti, Florian
AU - Nowrouzezahrai, Derek
AU - Fidler, Sanja
N1 - Publisher Copyright: © 2021 ICLR 2021 - 9th International Conference on Learning Representations. All rights reserved.
PY - 2021
Y1 - 2021
N2 - We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present ∇Sim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph - spanning from the dynamics and through the rendering process - enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
AB - We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present ∇Sim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph - spanning from the dynamics and through the rendering process - enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
M3 - Paper
AN - SCOPUS:85150230814
Y2 - 3 May 2021 through 7 May 2021
ER -
ID: 390240203