no code implementations • ECCV 2020 • Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi
Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.
no code implementations • 2 Apr 2024 • Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, Gerard Pons-Moll
Generating realistic hand motion sequences in interaction with objects has gained increasing attention with the growing interest in digital humans.
no code implementations • 22 Mar 2024 • Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Moll, Jia-Bin Huang, Vladislav Golyanik, Eddy Ilg
Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision.
no code implementations • 17 Mar 2024 • Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Ilya Petrov, Vladimir Guzov, Helisa Dhamo, Eduardo Pérez-Pellitero, Gerard Pons-Moll
Our key insight is that human motion is dictated by the interrelation between the force exerted by the human and the perceived resistance.
no code implementations • 5 Mar 2024 • Yannan He, Garvita Tiwari, Tolga Birdal, Jan Eric Lenssen, Gerard Pons-Moll
Faithfully modeling the space of articulations is a crucial task that allows recovery and generation of realistic poses, and remains a notorious challenge.
no code implementations • 22 Jan 2024 • Dimitrije Antić, Garvita Tiwari, Batuhan Ozcomlekci, Riccardo Marin, Gerard Pons-Moll
Additionally, we propose CloSe-Net, the first learning-based 3D clothing segmentation model for fine-grained segmentation from colored point clouds.
no code implementations • 7 Jan 2024 • Xianghui Xie, Xi Wang, Nikos Athanasiou, Bharat Lal Bhatnagar, Chun-Hao P. Huang, Kaichun Mo, Hao Chen, Xia Jia, Zerui Zhang, Liangxian Cui, Xiao Lin, Bingqiao Qian, Jie Xiao, Wenfei Yang, Hyeongjin Nam, Daniel Sungho Jung, Kihoon Kim, Kyoung Mu Lee, Otmar Hilliges, Gerard Pons-Moll
Modeling the interaction between humans and objects has been an emerging research direction in recent years.
1 code implementation • 21 Dec 2023 • Riccardo Marin, Enric Corona, Gerard Pons-Moll
NSR achieves the state of the art over public benchmarks, and the release of its code and checkpoints will provide the community with a powerful tool useful for many downstream tasks like dataset alignments, cleaning, or asset animation.
no code implementations • 18 Dec 2023 • Kim Youwang, Tae-Hyun Oh, Gerard Pons-Moll
We present Paint-it, a text-driven high-fidelity texture map synthesis method for 3D meshes via neural re-parameterized texture optimization.
no code implementations • 12 Dec 2023 • Xianghui Xie, Bharat Lal Bhatnagar, Jan Eric Lenssen, Gerard Pons-Moll
We generate 1M+ human-object interaction pairs in 3D and leverage this large-scale data to train our HDM (Hierarchical Diffusion Model), a novel method to reconstruct interacting human and unseen objects, without any templates.
no code implementations • 22 Nov 2023 • Berna Kabadayi, Wojciech Zielonka, Bharat Lal Bhatnagar, Gerard Pons-Moll, Justus Thies
For controlling the model, we learn a mapping from 3DMM facial expression parameters to the latent space of the generative model.
no code implementations • ICCV 2023 • Yuxuan Xue, Bharat Lal Bhatnagar, Riccardo Marin, Nikolaos Sarafianos, Yuanlu Xu, Gerard Pons-Moll, Tony Tung
Compared to existing approaches, our method eliminates the expensive per-frame surface extraction while maintaining mesh coherency, and is capable of reconstructing meshes with arbitrary resolution without retraining.
1 code implementation • CVPR 2023 • Ilya A. Petrov, Riccardo Marin, Julian Chibane, Gerard Pons-Moll
The intimate entanglement between objects affordances and human poses is of large interest, among others, for behavioural sciences, cognitive psychology, and Computer Vision communities.
no code implementations • 4 Apr 2023 • Aymen Mir, Xavier Puig, Angjoo Kanazawa, Gerard Pons-Moll
We decompose the continual motion synthesis problem into walking along paths and transitioning in and out of the actions specified by the keypoints, which enables long generation of motions that satisfy scene constraints without explicitly incorporating scene information.
no code implementations • CVPR 2023 • Xianghui Xie, Bharat Lal Bhatnagar, Gerard Pons-Moll
In this work, we propose a novel method to track the 3D human, object, contacts between them, and their relative translation across frames from a single RGB camera, while being robust to heavy occlusions.
no code implementations • 21 Oct 2022 • Marc Habermann, Lingjie Liu, Weipeng Xu, Gerard Pons-Moll, Michael Zollhoefer, Christian Theobalt
Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings.
1 code implementation • 28 Jul 2022 • Zhouyingcheng Liao, Jimei Yang, Jun Saito, Gerard Pons-Moll, Yang Zhou
We present the first method that automatically transfers poses between stylized 3D characters without skeletal rigging.
1 code implementation • 27 Jul 2022 • Garvita Tiwari, Dimitrije Antic, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, Gerard Pons-Moll
The resulting high-dimensional implicit function can be differentiated with respect to the input poses and thus can be used to project arbitrary poses onto the manifold by using gradient descent on the set of 3-dimensional hyperspheres.
no code implementations • 2 Jun 2022 • Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll
Indeed, we show that it is possible to train dense segmentation models using only bounding box labels.
3D Instance Segmentation 3D Semantic Instance Segmentation +2
no code implementations • 16 May 2022 • Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, Gerard Pons-Moll
The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction.
no code implementations • 12 May 2022 • Enric Corona, Gerard Pons-Moll, Guillem Alenyà, Francesc Moreno-Noguer
An exhaustive evaluation demonstrates that our approach is able to capture the underlying body of clothed people with very different body shapes, achieving a significant improvement compared to state-of-the-art.
no code implementations • 5 May 2022 • Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, Gerard Pons-Moll
In order for widespread adoption of such emerging applications, the sensor setup used to capture the interactions needs to be inexpensive and easy-to-use for non-expert users.
no code implementations • 1 May 2022 • Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, Gerard Pons-Moll
In this work, we study the problem of synthesizing scene interactions conditioned on different contact positions on the object.
no code implementations • 22 Apr 2022 • Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, Gerard Pons-Moll
With the aim of obtaining interpretable and controllable scene representations, our model couples learnt scene-specific feature volumes with a scene agnostic neural rendering network.
1 code implementation • CVPR 2022 • Bharat Lal Bhatnagar, Xianghui Xie, Ilya A. Petrov, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll
We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them.
1 code implementation • 5 Apr 2022 • Xianghui Xie, Bharat Lal Bhatnagar, Gerard Pons-Moll
However, humans are constantly interacting with the surrounding objects, thus calling for models that can reason about not only the human but also the object and their interaction.
Ranked #2 on 3D Object Reconstruction on BEHAVE
no code implementations • 20 Nov 2021 • Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.
1 code implementation • ICCV 2021 • Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, Gerard Pons-Moll
Neural-GIF can be trained on raw 3D scans and reconstructs detailed complex surface geometry and deformations.
no code implementations • 6 Jul 2021 • Marc Habermann, Weipeng Xu, Helge Rhodin, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
Our texture term exploits the orientation information in the micro-structures of the objects, e. g., the yarn patterns of fabrics.
no code implementations • 4 May 2021 • Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery.
1 code implementation • CVPR 2021 • Julian Chibane, Aayush Bansal, Verica Lazova, Gerard Pons-Moll
Recent neural view synthesis methods have achieved impressive quality and realism, surpassing classical pipelines which rely on multi-view reconstruction.
1 code implementation • CVPR 2021 • Vladimir Guzov, Aymen Mir, Torsten Sattler, Gerard Pons-Moll
We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment using wearable sensors.
1 code implementation • CVPR 2021 • Enric Corona, Albert Pumarola, Guillem Alenyà, Gerard Pons-Moll, Francesc Moreno-Noguer
In this paper we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry.
no code implementations • 13 Feb 2021 • Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, Christian Theobalt
We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input.
Ranked #4 on Gesture Generation on BEAT2
no code implementations • 1 Feb 2021 • Keyang Zhou, Bharat Lal Bhatnagar, Bernt Schiele, Gerard Pons-Moll
The remarkable result is that with only self-supervision, ART facilitates learning a unique canonical orientation for both rigid and nonrigid shapes, which leads to a notable boost in performance of aforementioned tasks.
1 code implementation • CVPR 2021 • Albert Pumarola, Enric Corona, Gerard Pons-Moll, Francesc Moreno-Noguer
In this paper we introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a \emph{single} camera moving around the scene.
1 code implementation • 2 Nov 2020 • Denis Tome, Thiemo Alldieck, Patrick Peluse, Gerard Pons-Moll, Lourdes Agapito, Hernan Badino, Fernando de la Torre
The quantitative evaluation, on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric approaches.
1 code implementation • NeurIPS 2020 • Julian Chibane, Aymen Mir, Gerard Pons-Moll
NDF represent surfaces at high resolutions as prior implicit models, but do not require closed surface data, and significantly broaden the class of representable shapes in the output.
no code implementations • 26 Oct 2020 • Alexandre Saint, Anis Kacem, Kseniya Cherenkova, Konstantinos Papadopoulos, Julian Chibane, Gerard Pons-Moll, Gleb Gusev, David Fofi, Djamila Aouada, Bjorn Ottersten
Additionally, two unique datasets of 3D scans are proposed, to provide raw ground-truth data for the benchmarks.
no code implementations • NeurIPS 2020 • Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll
Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined.
1 code implementation • 20 Sep 2020 • Julian Chibane, Gerard Pons-Moll
Instead, we focus on 3D texture and geometry completion from partial and incomplete 3D scans.
1 code implementation • ECCV 2020 • Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, Gerard Pons-Moll
SizerNet allows to estimate and visualize the dressing effect of a garment in various sizes, and ParserNet allows to edit clothing of an input mesh directly, removing the need for scan segmentation, which is a challenging problem in itself.
1 code implementation • ECCV 2020 • Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll
In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing.
1 code implementation • ECCV 2020 • Keyang Zhou, Bharat Lal Bhatnagar, Gerard Pons-Moll
The experiments on datasets of 3D humans, faces, hands and animals demonstrate the generality of our approach.
2 code implementations • ECCV 2020 • Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, Bernt Schiele
In this work, we propose a novel method for monocular video-based 3D object detection which carefully leverages kinematic motion to improve precision of 3D localization.
Ranked #6 on 3D Object Detection on Rope3D
no code implementations • CVPR 2020 • Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.
2 code implementations • CVPR 2020 • Chaitanya Patel, Zhouyingcheng Liao, Gerard Pons-Moll
While the low-frequency component is predicted from pose, shape and style parameters with an MLP, the high-frequency component is predicted with a mixture of shape-style specific pose models.
1 code implementation • CVPR 2020 • Aymen Mir, Thiemo Alldieck, Gerard Pons-Moll
In this paper, we present a simple yet effective method to automatically transfer textures of clothing images (front and back) to 3D garments worn on top SMPL, in real time.
1 code implementation • CVPR 2020 • Julian Chibane, Thiemo Alldieck, Gerard Pons-Moll
To solve this, we propose Implicit Feature Networks (IF-Nets), which deliver continuous outputs, can handle multiple topologies, and complete shapes for missing or sparse input data retaining the nice properties of recent learned implicit functions, but critically they can also retain detail when it is present in the input data, and can reconstruct articulated humans.
no code implementations • 6 Dec 2019 • Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi
Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.
no code implementations • 20 Aug 2019 • Verica Lazova, Eldar Insafutdinov, Gerard Pons-Moll
In order to learn our model in a common UV-space, we non-rigidly register the SMPL model to thousands of 3D scans, effectively encoding textures and geometries as images in correspondence.
6 code implementations • ICCV 2019 • Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, Gerard Pons-Moll
We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video.
3D Human Pose Estimation 3D Shape Reconstruction From A Single 2D Image
1 code implementation • CVPR 2020 • Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, Michael J. Black
To our knowledge, this is the first generative model that directly dresses 3D human body meshes and generalizes to different poses.
4 code implementations • 1 Jul 2019 • Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, Christian Theobalt
The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals. We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy.
Ranked #7 on 3D Multi-Person Pose Estimation on MuPoTS-3D
3D Multi-Person Human Pose Estimation 3D Multi-Person Pose Estimation +1
no code implementations • 27 May 2019 • Hosnieh Sattar, Katharina Krombholz, Gerard Pons-Moll, Mario Fritz
Modern approaches to pose and body shape estimation have recently achieved strong performance even under challenging real-world conditions.
1 code implementation • ICCV 2019 • Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, Marcus Magnor
From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing.
4 code implementations • ICCV 2019 • Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, Michael J. Black
We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model; here we use SMPL [doi:10. 1145/2816795. 2818013], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh.
no code implementations • CVPR 2019 • Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Gerard Pons-Moll, Christian Theobalt
Convolutional Neural Network based approaches for monocular 3D human pose estimation usually require a large amount of training images with 3D pose annotations.
no code implementations • CVPR 2019 • Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, Yebin Liu
This paper proposes a new method for live free-viewpoint human performance capture with dynamic details (e. g., cloth wrinkles) using a single RGBD camera.
1 code implementation • CVPR 2019 • Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll
We present a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving, in less than 10 seconds with a reconstruction accuracy of 5mm.
1 code implementation • 10 Oct 2018 • Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
no code implementations • 5 Oct 2018 • Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
Our method is the first real-time monocular approach for full-body performance capture.
no code implementations • ECCV 2018 • Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, Gerard Pons-Moll
In this work, we propose a method that combines a single hand-held camera and a set of Inertial Measurement Units (IMUs) attached at the body limbs to estimate accurate 3D poses in the wild.
2 code implementations • 17 Aug 2018 • Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler, Bernt Schiele
Direct prediction of 3D body pose and shape remains a challenge even for highly parameterized deep learning models.
Ranked #1 on Monocular 3D Human Pose Estimation on Human3.6M (Use Video Sequence metric)
1 code implementation • 3 Aug 2018 • Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll
We present a novel method for high detail-preserving human avatar creation from monocular video.
no code implementations • 9 Jul 2018 • Hosnieh Sattar, Gerard Pons-Moll, Mario Fritz
To study the correlation between clothing garments and body shape, we collected a new dataset (Fashion Takes Shape), which includes images of users with clothing category annotations.
no code implementations • CVPR 2018 • Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu
We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance.
1 code implementation • CVPR 2018 • Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll
This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving.
6 code implementations • 9 Dec 2017 • Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, Christian Theobalt
Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene.
Ranked #3 on 3D Multi-Person Pose Estimation (root-relative) on MuPoTS-3D (MPJPE metric)
3D Human Pose Estimation 3D Multi-Person Pose Estimation (absolute) +2
no code implementations • CVPR 2017 • Federica Bogo, Javier Romero, Gerard Pons-Moll, Michael J. Black
We propose a new mesh registration method that uses both 3D geometry and texture information to register all scans in a sequence to a common reference topology.
1 code implementation • ICCV 2017 • Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler
We present the first image-based generative model of people in clothing for the full body.
no code implementations • 23 Mar 2017 • Timo von Marcard, Bodo Rosenhahn, Michael J. Black, Gerard Pons-Moll
We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body.
1 code implementation • CVPR 2017 • Chao Zhang, Sergi Pujades, Michael Black, Gerard Pons-Moll
We address the problem of estimating human pose and body shape from 3D scans over time.
no code implementations • CVPR 2014 • Gerard Pons-Moll, David J. Fleet, Bodo Rosenhahn
We advocate the inference of qualitative information about 3D human pose, called posebits, from images.