gradSim: Differentiable simulation for system identification and visuomotor control

Research output: Contribution to conference › Paper › Research

Standard

gradSim : Differentiable simulation for system identification and visuomotor control. / Jatavallabhula, Krishna Murthy ; Macklin, Miles ; Golemo, Florian ; Volet, Vikram ; Petrini, Linda ; Weis, Martin ; Considine, Breandan ; Parent-Lévesque, Jérôme ; Xie, Kevin ; Erleben, Kenny; Pauli, Liam ; Shkurti, Florian ; Nowrouzezahrai, Derek ; Fidler, Sanja.

2021. Paper presented at 9th International Conference on Learning Representations - ICLR 2021, Virtual.

Research output: Contribution to conference › Paper › Research

Harvard

Jatavallabhula, KM, Macklin, M, Golemo, F, Volet, V, Petrini, L, Weis, M, Considine, B, Parent-Lévesque, J, Xie, K, Erleben, K, Pauli, L, Shkurti, F, Nowrouzezahrai, D & Fidler, S 2021, 'gradSim: Differentiable simulation for system identification and visuomotor control', Paper presented at 9th International Conference on Learning Representations - ICLR 2021, Virtual, 03/05/2021 - 07/05/2021. <https://openreview.net/forum?id=c_E8kFWfhp0>

APA

Jatavallabhula, K. M., Macklin, M., Golemo, F., Volet, V., Petrini, L., Weis, M., Considine, B., Parent-Lévesque, J., Xie, K., Erleben, K., Pauli, L., Shkurti, F., Nowrouzezahrai, D., & Fidler, S. (2021). gradSim: Differentiable simulation for system identification and visuomotor control. Paper presented at 9th International Conference on Learning Representations - ICLR 2021, Virtual. https://openreview.net/forum?id=c_E8kFWfhp0

Vancouver

Jatavallabhula KM, Macklin M, Golemo F, Volet V, Petrini L, Weis M et al. gradSim: Differentiable simulation for system identification and visuomotor control. 2021. Paper presented at 9th International Conference on Learning Representations - ICLR 2021, Virtual.

Author

Jatavallabhula, Krishna Murthy ; Macklin, Miles ; Golemo, Florian ; Volet, Vikram ; Petrini, Linda ; Weis, Martin ; Considine, Breandan ; Parent-Lévesque, Jérôme ; Xie, Kevin ; Erleben, Kenny ; Pauli, Liam ; Shkurti, Florian ; Nowrouzezahrai, Derek ; Fidler, Sanja. / gradSim : Differentiable simulation for system identification and visuomotor control. Paper presented at 9th International Conference on Learning Representations - ICLR 2021, Virtual.25 p.

Bibtex

@conference{8076bb084daa48f399796ead08e21866,
title = "gradSim: Differentiable simulation for system identification and visuomotor control",
abstract = "In this paper, we tackle the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current best solutions to the problem require precise 3D labels which are labor intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. In this work we present gradSim, a framework that overcomes the dependence on 3D supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Furthermore, our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to/better than techniques that require precise 3D labels.",
author = "Jatavallabhula, {Krishna Murthy} and Miles Macklin and Florian Golemo and Vikram Volet and Linda Petrini and Martin Weis and Breandan Considine and J{\'e}r{\^o}me Parent-L{\'e}vesque and Kevin Xie and Kenny Erleben and Liam Pauli and Florian Shkurti and Derek Nowrouzezahrai and Sanja Fidler",
year = "2021",
language = "English",
note = "9th International Conference on Learning Representations - ICLR 2021 ; Conference date: 03-05-2021 Through 07-05-2021",

}

RIS

TY - CONF

T1 - gradSim

T2 - 9th International Conference on Learning Representations - ICLR 2021

AU - Jatavallabhula, Krishna Murthy

AU - Macklin, Miles

AU - Golemo, Florian

AU - Volet, Vikram

AU - Petrini, Linda

AU - Weis, Martin

AU - Considine, Breandan

AU - Parent-Lévesque, Jérôme

AU - Xie, Kevin

AU - Erleben, Kenny

AU - Pauli, Liam

AU - Shkurti, Florian

AU - Nowrouzezahrai, Derek

AU - Fidler, Sanja

PY - 2021

Y1 - 2021

N2 - In this paper, we tackle the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current best solutions to the problem require precise 3D labels which are labor intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. In this work we present gradSim, a framework that overcomes the dependence on 3D supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Furthermore, our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to/better than techniques that require precise 3D labels.

AB - In this paper, we tackle the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current best solutions to the problem require precise 3D labels which are labor intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. In this work we present gradSim, a framework that overcomes the dependence on 3D supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Furthermore, our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to/better than techniques that require precise 3D labels.

M3 - Paper

Y2 - 3 May 2021 through 7 May 2021

ER -

ID: 276648735