no code implementations • NAACL (PrivateNLP) 2022 • Natalia Ponomareva, Jasmijn Bastings, Sergei Vassilvitskii
We focus on T5 and show that by using recent advances in JAX and XLA we can train models with DP that do not suffer a large drop in pre-training utility, nor in training speed, and can still be fine-tuned to high accuracies on downstream tasks (e. g.
no code implementations • 16 Feb 2024 • Yunjuan Wang, Hussein Hazimeh, Natalia Ponomareva, Alexey Kurakin, Ibrahim Hammoud, Raman Arora
To address this challenge, we first establish a generalization bound for the adversarial target loss, which consists of (i) terms related to the loss on the data, and (ii) a measure of worst-case domain divergence.
no code implementations • 16 Feb 2024 • Artem Trofimov, Mikhail Kostyukov, Sergei Ugdyzhekov, Natalia Ponomareva, Igor Naumov, Maksim Melekhovets
Integrated development environments (IDEs) are prevalent code-writing and debugging tools.
no code implementations • 6 Feb 2024 • Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo
With sufficient alignment, both downstream cross-entropy and BLEU score improve monotonically with more pretraining data.
1 code implementation • 5 Jun 2023 • Shibal Ibrahim, Wenyu Chen, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, Rahul Mazumder
To deal with this challenge, we propose a novel, permutation-based local search method that can complement first-order methods in training any sparse gate, e. g., Hash routing, Top-k, DSelect-k, and COMET.
no code implementations • 5 Jun 2023 • Qi Zhu, Yizhu Jiao, Natalia Ponomareva, Jiawei Han, Bryan Perozzi
Graph Neural Networks (GNNs) have shown remarkable performance on graph-structured data.
no code implementations • 2 Jun 2023 • Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, Andreas Terzis
An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data.
no code implementations • 10 May 2023 • Aldo Gael Carranza, Rezsa Farahani, Natalia Ponomareva, Alex Kurakin, Matthew Jagielski, Milad Nasr
We address the challenge of ensuring differential privacy (DP) guarantees in training deep retrieval systems.
1 code implementation • 1 Mar 2023 • Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, Abhradeep Thakurta
However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between.
no code implementations • 28 Feb 2023 • Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, Rahul Mazumder
Our approach, CHITA, extends the classical Optimal Brain Surgeon framework and results in significant improvements in speed, memory, and performance over existing optimization-based approaches for network pruning.
no code implementations • 31 Jan 2023 • Hussein Hazimeh, Natalia Ponomareva
We run large-scale experiments to study the effectiveness of the scheduler on two popular applications: GANs for image generation and adversarial nets for domain adaptation.
no code implementations • 13 Oct 2021 • Shibal Ibrahim, Natalia Ponomareva, Rahul Mazumder
In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score -- a common baseline for newer metrics -- and propose shrinkage-based estimator.
1 code implementation • NeurIPS 2021 • Qi Zhu, Natalia Ponomareva, Jiawei Han, Bryan Perozzi
In this work we present a method, Shift-Robust GNN (SR-GNN), designed to account for distributional differences between biased training data and the graph's true inference distribution.
2 code implementations • ICML 2020 • Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, Rahul Mazumder
We aim to combine these advantages by introducing a new layer for neural networks, composed of an ensemble of differentiable decision trees (a. k. a.
no code implementations • 19 Sep 2019 • Khaled S. Refaat, Kai Ding, Natalia Ponomareva, Stéphane Ross
We propose a system to rank agents around an autonomous vehicle (AV) in real time.
no code implementations • RANLP 2019 • Victoria Yaneva, Constantin Orasan, Le An Ha, Natalia Ponomareva
NLP approaches to automatic text adaptation often rely on user-need guidelines which are generic and do not account for the differences between various types of target groups.
1 code implementation • 20 Mar 2019 • Haihao Lu, Sai Praneeth Karimireddy, Natalia Ponomareva, Vahab Mirrokni
This is the first GBM type of algorithm with theoretically-justified accelerated convergence rate.
no code implementations • 31 Oct 2017 • Natalia Ponomareva, Thomas Colthurst, Gilbert Hendry, Salem Haykal, Soroush Radpour
Gradient boosted decision trees are a popular machine learning technique, in part because of their ability to give good accuracy with small models.
no code implementations • 31 Oct 2017 • Natalia Ponomareva, Soroush Radpour, Gilbert Hendry, Salem Haykal, Thomas Colthurst, Petr Mitrichev, Alexander Grushetsky
TF Boosted Trees (TFBT) is a new open-sourced frame-work for the distributed training of gradient boosted trees.