no code implementations • 2 May 2024 • Guangming Wang, Lei Pan, Songyou Peng, Shaohui Liu, Chenfeng Xu, Yanzi Miao, Wei Zhan, Masayoshi Tomizuka, Marc Pollefeys, Hesheng Wang
Meticulous 3D environment representations have been a longstanding goal in computer vision and robotics fields.
no code implementations • 12 Mar 2024 • Chensheng Peng, Chenfeng Xu, Yue Wang, Mingyu Ding, Heng Yang, Masayoshi Tomizuka, Kurt Keutzer, Marco Pavone, Wei Zhan
This focus results in a significant disconnect between NeRF applications, i. e., novel-view synthesis and the requirements of SLAM.
no code implementations • 10 Mar 2024 • Juanwu Lu, Wei Zhan, Masayoshi Tomizuka, Yeping Hu
For interpretability, the model achieves target-driven motion prediction by estimating the spatial distribution of long-term destinations with a variational mixture of Gaussians.
no code implementations • 26 Feb 2024 • Dingkun Guo, Yuqi Xiang, Shuqi Zhao, Xinghao Zhu, Masayoshi Tomizuka, Mingyu Ding, Wei Zhan
With these two capabilities, PhyGrasp is able to accurately assess the physical properties of object parts and determine optimal grasping poses.
no code implementations • 23 Feb 2024 • Yichen Xie, Hongge Chen, Gregory P. Meyer, Yong Jae Lee, Eric M. Wolff, Masayoshi Tomizuka, Wei Zhan, Yuning Chai, Xin Huang
Observations from different angles enable the recovery of 3D object states from 2D image inputs if we can identify the same instance in different input frames.
no code implementations • 22 Feb 2024 • Catherine Weaver, Chen Tang, Ce Hao, Kenta Kawamoto, Masayoshi Tomizuka, Wei Zhan
Thus, we propose BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a Behavior Transformer (BeT) policy from human demonstrations with online AIL.
1 code implementation • 14 Feb 2024 • Tong Zhao, Mingyu Ding, Wei Zhan, Masayoshi Tomizuka, Yintao Wei
Furthermore, we propose a more rigorous evaluation metric that considers depth-wise relative error, providing comprehensive evaluations for universal stereo matching and depth estimation models.
no code implementations • 31 Dec 2023 • Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, Manmohan Chandraker
These findings affirm that guided diffusion models provide a robust and versatile foundation for safety-critical, interactive traffic simulation, extending their utility across the broader landscape of autonomous driving.
no code implementations • 3 Nov 2023 • Tommaso Benciolini, Chen Tang, Marion Leibold, Catherine Weaver, Masayoshi Tomizuka, Wei Zhan
In the exploration, a MPC collects diverse data by balancing the racing objectives and the exploration criterion; then the GP is re-trained.
no code implementations • 11 Oct 2023 • Yuxin Chen, Chen Tang, Ran Tian, Chenran Li, Jinning Li, Masayoshi Tomizuka, Wei Zhan
We observe that, generally, a more diverse set of co-play agents during training enhances the generalization performance of the ego agent; however, this improvement varies across distinct scenarios and environments.
no code implementations • 4 Oct 2023 • Mingxiao Huo, Mingyu Ding, Chenfeng Xu, Thomas Tian, Xinghao Zhu, Yao Mu, Lingfeng Sun, Masayoshi Tomizuka, Wei Zhan
We introduce Task Fusion Decoder as a plug-and-play embedding translator that utilizes the underlying relationships among these perceptual skills to guide the representation learning towards encoding meaningful structure for what's important for all perceptual skills, ultimately empowering learning of downstream robotic manipulation tasks.
no code implementations • 4 Oct 2023 • Hao Sha, Yao Mu, YuXuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, Mingyu Ding
Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability.
no code implementations • 3 Oct 2023 • Tong Zhao, Chenfeng Xu, Mingyu Ding, Masayoshi Tomizuka, Wei Zhan, Yintao Wei
This paper addresses the growing demands for safety and comfort in intelligent robot systems, particularly autonomous vehicles, where road conditions play a pivotal role in overall driving performance.
1 code implementation • NeurIPS 2023 • Yichen Xie, Mingyu Ding, Masayoshi Tomizuka, Wei Zhan
However, current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly.
no code implementations • 18 Sep 2023 • Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, Wei Zhan
GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms.
no code implementations • 18 Sep 2023 • Yiheng Li, Seth Z. Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, Wei Zhan
We propose to augment both HD maps and trajectories and apply pre-training strategies on top of them.
1 code implementation • ICCV 2023 • Chensheng Peng, Guangming Wang, Xian Wan Lo, Xinrui Wu, Chenfeng Xu, Masayoshi Tomizuka, Wei Zhan, Hesheng Wang
Previous methods rarely predict scene flow from the entire point clouds of the scene with one-time inference due to the memory inefficiency and heavy overhead from distance calculation and sorting involved in commonly used farthest point sampling, KNN, and ball query algorithms for local feature aggregation.
2 code implementations • ICCV 2023 • Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, RuiLong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
We present NeRF-Det, a novel method for indoor 3D detection with posed RGB images as input.
no code implementations • 29 Jun 2023 • Zitian Chen, Mingyu Ding, Yikang Shen, Wei Zhan, Masayoshi Tomizuka, Erik Learned-Miller, Chuang Gan
We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently.
no code implementations • NeurIPS 2023 • Chenran Li, Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, Wei Zhan
Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task.
no code implementations • 14 Jun 2023 • Ce Hao, Catherine Weaver, Chen Tang, Kenta Kawamoto, Masayoshi Tomizuka, Wei Zhan
Hierarchical reinforcement learning (RL) can accelerate long-horizon decision-making by temporally abstracting a policy into multiple levels.
1 code implementation • 1 Jun 2023 • Banghua Zhu, Mingyu Ding, Philip Jacobson, Ming Wu, Wei Zhan, Michael Jordan, Jiantao Jiao
Self-training is an important technique for solving semi-supervised learning problems.
1 code implementation • ICCV 2023 • Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan
However, given that objects occupy only a small part of a scene, finding dense candidates and generating dense representations is noisy and inefficient.
no code implementations • 27 Apr 2023 • Chao Xia, Chenfeng Xu, Patrick Rim, Mingyu Ding, Nanning Zheng, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan
Current LiDAR odometry, mapping and localization methods leverage point-wise representations of 3D scenes and achieve high accuracy in autonomous driving tasks.
1 code implementation • CVPR 2023 • Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, Wei Zhan
We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool and maintaining enough diversity by optimizing a parametric model in the continuous space.
no code implementations • 24 Mar 2023 • Wei-Jer Chang, Chen Tang, Chenran Li, Yeping Hu, Masayoshi Tomizuka, Wei Zhan
To ensure that autonomous vehicles take safe and efficient maneuvers in different interactive traffic scenarios, we should be able to evaluate autonomous vehicles against reactive agents with different social characteristics in the simulation environment.
2 code implementations • 13 Feb 2023 • Haoyu Lu, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Wei Zhan, Masayoshi Tomizuka, Mingyu Ding
Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49. 7% recall@1 with 2. 2% model parameters, outperforming the latest competitors by 2. 0%.
1 code implementation • 5 Oct 2022 • Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, Wei Zhan
While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception.
Ranked #1 on Robust Camera Only 3D Object Detection on nuScenes-C
no code implementations • 26 Sep 2022 • Philip Jacobson, Yiyang Zhou, Wei Zhan, Masayoshi Tomizuka, Ming C. Wu
In this work, we propose a novel approach Center Feature Fusion (CFF), in which we leverage center-based detection networks in both the camera and LiDAR streams to identify relevant object locations.
no code implementations • 9 Aug 2022 • Wei-Jer Chang, Yeping Hu, Chenran Li, Wei Zhan, Masayoshi Tomizuka
In this paper, we aim to provide a thorough stability analysis of the reactive simulation and propose a solution to enhance the stability.
no code implementations • 6 Aug 2022 • Juanwu Lu, Wei Zhan, Masayoshi Tomizuka, Yeping Hu
Results show significant performance degradation due to domain shift, and feature attribution provides insights to identify potential causes of these problems.
1 code implementation • 19 Jul 2022 • Guangming Wang, Yunzhe Hu, Zhe Liu, Yiyang Zhou, Masayoshi Tomizuka, Wei Zhan, Hesheng Wang
Our proposed model surpasses all existing methods by at least 38. 2% on FlyingThings3D dataset and 24. 7% on KITTI Scene Flow dataset for EPE3D metric.
no code implementations • 8 Jul 2022 • Akio Kodaira, Yiyang Zhou, Pengwei Zang, Wei Zhan, Masayoshi Tomizuka
With information from multiple input modalities, sensor fusion-based algorithms usually out-perform their single-modality counterparts in robotics.
1 code implementation • 21 Apr 2022 • Chenfeng Xu, Tian Li, Chen Tang, Lingfeng Sun, Kurt Keutzer, Masayoshi Tomizuka, Alireza Fathi, Wei Zhan
It is hard to replicate these approaches in trajectory forecasting due to the lack of adequate trajectory data (e. g., 34K samples in the nuScenes dataset).
no code implementations • 19 Apr 2022 • Chen Tang, Wei Zhan, Masayoshi Tomizuka
Moreover, to properly evaluate an IBP model with offline datasets, we propose a Shapley-value-based metric to verify if the prediction model satisfies the inherent temporal independence of an interventional distribution.
no code implementations • 28 Mar 2022 • Lingfeng Sun, Chen Tang, Yaru Niu, Enna Sachdeva, Chiho Choi, Teruhisa Misu, Masayoshi Tomizuka, Wei Zhan
To address these issues, we propose a novel approach to avoid KL vanishing and induce an interpretable interactive latent space with pseudo labels.
1 code implementation • 17 Mar 2022 • Jinhyung Park, Chenfeng Xu, Yiyang Zhou, Masayoshi Tomizuka, Wei Zhan
While numerous 3D detection works leverage the complementary relationship between RGB images and point clouds, developments in the broader framework of semi-supervised object recognition remain uninfluenced by multi-modal fusion.
no code implementations • 10 Feb 2022 • Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka, Changliu Liu
By mimicking humans' cognition model and semantic understanding during driving, we propose HATN, a hierarchical framework to generate high-quality, transferable, and adaptable predictions for driving behaviors in multi-agent dense-traffic environments.
2 code implementations • 15 Dec 2021 • Yichen Xie, Masayoshi Tomizuka, Wei Zhan
Existing work follows a cumbersome pipeline that repeats the time-consuming model training and batch data selection multiple times.
no code implementations • 3 Dec 2021 • Yeping Hu, Xiaogang Jia, Masayoshi Tomizuka, Wei Zhan
In this paper, we aim to address the domain generalization problem for vehicle intention prediction tasks and a causal-based time series domain generalization (CTSDG) model is proposed.
no code implementations • NeurIPS 2021 • Chen Tang, Wei Zhan, Masayoshi Tomizuka
In this work, we argue that one of the typical formulations of VAEs in multi-agent modeling suffers from an issue we refer to as social posterior collapse, i. e., the model is prone to ignoring historical social context when predicting the future trajectory of an agent.
no code implementations • 9 Nov 2021 • Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan
Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment.
no code implementations • 14 Aug 2021 • Zhenggang Tang, Kai Yan, Liting Sun, Wei Zhan, Changliu Liu
To efficiently simulate with massive amounts of agents in MPS, we propose Scalable Million-Agent DQN (SMADQN).
1 code implementation • 8 Jun 2021 • Chenfeng Xu, Shijia Yang, Tomer Galanti, Bichen Wu, Xiangyu Yue, Bohan Zhai, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds.
1 code implementation • 6 Mar 2021 • Di Feng, Yiyang Zhou, Chenfeng Xu, Masayoshi Tomizuka, Wei Zhan
Detecting dynamic objects and predicting static road information such as drivable areas and ground heights are crucial for safe autonomous driving.
no code implementations • 1 Mar 2021 • Zhao-Heng Yin, Lingfeng Sun, Liting Sun, Masayoshi Tomizuka, Wei Zhan
Experiments show that our model can generate diverse interactions in various scenarios.
no code implementations • 17 Jan 2021 • Jinning Li, Liting Sun, Jianyu Chen, Masayoshi Tomizuka, Wei Zhan
To address this challenge, we propose a hierarchical behavior planning framework with a set of low-level safe controllers and a high-level reinforcement learning algorithm (H-CtRL) as a coordinator for the low-level controllers.
no code implementations • 18 Dec 2020 • Di Feng, Zining Wang, Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, Wei Zhan
As a result, an in-depth evaluation among different object detection methods remains challenging, and the training process of object detectors is sub-optimal, especially in probabilistic object detection.
6 code implementations • CVPR 2021 • Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei LI, Zehuan Yuan, Changhu Wang, Ping Luo
In our method, however, a fixed sparse set of learned object proposals, total length of $N$, are provided to object recognition head to perform classification and location.
Ranked #5 on 2D Object Detection on CeyMo
no code implementations • 4 Nov 2020 • Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, Wei Zhan
We find three interpretable patterns of interactions, bringing insights for driver behavior representation, modeling and comprehension.
no code implementations • 28 Oct 2020 • Letian Wang, Liting Sun, Masayoshi Tomizuka, Wei Zhan
It allows the AVs to infer the characteristics of other road users online and generate behaviors optimizing not only their own rewards, but also their courtesy to others, and their confidence regarding the prediction uncertainties.
no code implementations • 22 Jun 2020 • Zheng Wu, Liting Sun, Wei Zhan, Chenyu Yang, Masayoshi Tomizuka
Different from existing IRL algorithms, by introducing an efficient continuous-domain trajectory sampler, the proposed algorithm can directly learn the reward functions in the continuous domain while considering the uncertainties in demonstrated trajectories from human drivers.
no code implementations • 22 Jun 2020 • Hujie Pan, Zining Wang, Wei Zhan, Masayoshi Tomizuka
In this paper, we propose a novel form of the loss function to increase the performance of LiDAR-based 3d object detection and obtain more explainable and convincing uncertainty for the prediction.
no code implementations • 7 Apr 2020 • Yeping Hu, Wei Zhan, Masayoshi Tomizuka
Accurately predicting the possible behaviors of traffic participants is an essential capability for autonomous vehicles.
3 code implementations • ECCV 2020 • Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image.
Ranked #24 on 3D Semantic Segmentation on SemanticKITTI
no code implementations • 7 Mar 2020 • Zining Wang, Di Feng, Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, Wei Zhan
Based on the spatial distribution, we further propose an extension of IoU, called the Jaccard IoU (JIoU), as a new evaluation metric that incorporates label uncertainty.
2 code implementations • 20 Dec 2019 • Chenfeng Xu, Dingkang Liang, Yongchao Xu, Song Bai, Wei Zhan, Xiang Bai, Masayoshi Tomizuka
A major issue is that the density map on dense regions usually accumulates density values from a number of nearby Gaussian blobs, yielding different large density values on a small set of pixels.
no code implementations • 19 Dec 2019 • Weisong Wen, Yiyang Zhou, Guohao Zhang, Saman Fahandezh-Saadi, Xiwei Bai, Wei Zhan, Masayoshi Tomizuka, Li-Ta Hsu
Mapping and localization is a critical module of autonomous driving, and significant achievements have been reached in this field.
no code implementations • 30 Sep 2019 • Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kummerle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, Masayoshi Tomizuka
3) The driving behavior is highly interactive and complex with adversarial and cooperative motions of various traffic participants.
no code implementations • 23 Aug 2019 • Jiachen Li, Wei Zhan, Yeping Hu, Masayoshi Tomizuka
The framework can incorporate an arbitrary prediction model as the implicit proposal distribution of the CMSMC method.
no code implementations • 19 Jul 2019 • Liting Sun, Wei Zhan, Yeping Hu, Masayoshi Tomizuka
Hence, the goal of this work is to formulate the human drivers' behavior generation model with CPT so that some ``irrational'' behavior or decisions of human can be better captured and predicted.
no code implementations • 2 May 2019 • Liting Sun, Wei Zhan, Ching-Yao Chan, Masayoshi Tomizuka
The uncertainties can come either from sensor limitations such as occlusions and limited sensor range, or from probabilistic prediction of other road participants, or from unknown social behavior in a new area.
no code implementations • 2 May 2019 • Jiachen Li, Hengbo Ma, Wei Zhan, Masayoshi Tomizuka
In order to tackle the task of probabilistic prediction for multiple, interactive entities, we propose a coordination and trajectory prediction system (CTPS), which has a hierarchical structure including a macro-level coordination recognition module and a micro-level subtle pattern prediction module which solves a probabilistic generation task.
no code implementations • 22 Mar 2019 • Yeping Hu, Wei Zhan, Liting Sun, Masayoshi Tomizuka
The proposed method is based on a generative model and is capable of jointly predicting sequential motions of each pair of interacting agents.
no code implementations • 30 Oct 2018 • Yeping Hu, Wei Zhan, Masayoshi Tomizuka
In a given scenario, simultaneously and accurately predicting every possible interaction of traffic participants is an important capability for autonomous vehicles.
no code implementations • 10 Sep 2018 • Wei Zhan, Liting Sun, Yeping Hu, Jiachen Li, Masayoshi Tomizuka
Modified methods based on PGM, NN and IRL are provided to generate probabilistic reaction predictions in an exemplar scenario of nudging from a highway ramp.
no code implementations • 9 Sep 2018 • Liting Sun, Wei Zhan, Masayoshi Tomizuka
To safely and efficiently interact with other road participants, AVs have to accurately predict the behavior of surrounding vehicles and plan accordingly.
no code implementations • 9 Sep 2018 • Jiachen Li, Hengbo Ma, Wei Zhan, Masayoshi Tomizuka
Accurate and robust recognition and prediction of traffic situation plays an important role in autonomous driving, which is a prerequisite for risk assessment and effective decision making.
no code implementations • 8 Aug 2018 • Liting Sun, Wei Zhan, Masayoshi Tomizuka, Anca D. Dragan
Such a courtesy term enables the robot car to be aware of possible irrationality of the human behavior, and plan accordingly.
no code implementations • 10 Apr 2018 • Yeping Hu, Wei Zhan, Masayoshi Tomizuka
Accurately predicting the possible behaviors of traffic participants is an essential capability for future autonomous vehicles.
no code implementations • 17 Nov 2017 • Zining Wang, Wei Zhan, Masayoshi Tomizuka
The fusion method shows particular benefit for detection of pedestrians in the bird view compared to other fusion-based object detection networks.
no code implementations • 9 Jul 2017 • Liting Sun, Cheng Peng, Wei Zhan, Masayoshi Tomizuka
For safe and efficient planning and control in autonomous driving, we need a driving policy which can achieve desirable driving quality in long-term horizon with guaranteed safety and feasibility.