no code implementations • 2 Jun 2024 • Bar Iluz, Yanai Elazar, Asaf Yehudai, Gabriel Stanovsky
Most works on gender bias focus on intrinsic bias -- removing traces of information about a protected group from the model's internal representation.
no code implementations • 23 May 2024 • Asaf Yehudai, Taelin Karidi, Gabriel Stanovsky, Ariel Goldstein, Omri Abend
In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study.
no code implementations • 1 Mar 2024 • Ariel Goldstein, Gabriel Stanovsky
Recent advances in LLMs have sparked a debate on whether they understand text.
no code implementations • 21 Feb 2024 • Gili Lior, Yoav Goldberg, Gabriel Stanovsky
Document collections of various domains, e. g., legal, medical, or financial, often share some underlying collection-wide structure, which captures information that can aid both human users and structure-aware models.
1 code implementation • 25 Jan 2024 • Itay Manes, Naama Ronn, David Cohen, Ran Ilan Ber, Zehavi Horowitz-Kugler, Gabriel Stanovsky
Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health.
2 code implementations • 31 Dec 2023 • Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, Gabriel Stanovsky
Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks.
1 code implementation • 21 Sep 2023 • Bar Iluz, Tomasz Limisiewicz, Gabriel Stanovsky, David Mareček
We study the effect of tokenization on gender bias in machine translation, an aspect that has been largely overlooked in previous works.
1 code implementation • 1 Aug 2023 • Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, Yonatan Belinkov
Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically.
1 code implementation • 1 Jun 2023 • Catherine Chen, Zejiang Shen, Dan Klein, Gabriel Stanovsky, Doug Downey, Kyle Lo
Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers.
1 code implementation • 24 May 2023 • Gili Lior, Gabriel Stanovsky
We approach this question through the lens of the dual-process theory for human decision-making.
1 code implementation • 23 May 2023 • Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Alan Ritter
We use this collection of annotated tables to evaluate the ability of open-source and API-based language models to extract information from tables covering diverse domains and data formats.
Ranked #1 on Attribute Extraction on SWDE
no code implementations • 9 May 2023 • Eliya Habba, Renana Keydar, Dan Bareket, Gabriel Stanovsky
Second, we curate a manually annotated dataset for judicial assessments of victim's credibility in the Hebrew language, as well as a model that can extract credibility labels from court cases.
no code implementations • ICCV 2023 • Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz
We introduce WHOOPS!, a new dataset and benchmark for visual commonsense.
Ranked #1 on Image-to-Text Retrieval on WHOOPS! (using extra training data)
no code implementations • 16 Feb 2023 • Asaf Yehudai, Arie Cattan, Omri Abend, Gabriel Stanovsky
Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora.
no code implementations • 9 Feb 2023 • Uri Berger, Lea Frermann, Gabriel Stanovsky, Omri Abend
We study the relation between visual input and linguistic choices by training classifiers to predict the probability of expressing a property from raw images, and find evidence supporting the claim that linguistic properties are constrained by visual context across languages.
1 code implementation • 8 Dec 2022 • Yonatan Bitton, Ron Yosef, Eli Strugo, Dafna Shahaf, Roy Schwartz, Gabriel Stanovsky
We leverage situation recognition annotations and the CLIP model to generate a large set of 500k candidate analogies.
Ranked #1 on Visual Reasoning on VASR
1 code implementation • 24 Oct 2022 • Keshav Kolluru, Gabriel Stanovsky, Mausam
Proper noun compounds, e. g., "Covid vaccine", convey information in a succinct manner (a "Covid vaccine" is a "vaccine that immunizes against the Covid disease").
no code implementations • 13 Oct 2022 • Tomasz Limisiewicz, Dan Malkin, Gabriel Stanovsky
Our method outperforms standard training methods in low-resource languages and retrains performance on high-resource languages while using the same amount of data.
1 code implementation • 25 Jul 2022 • Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, Roy Schwartz
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills.
Ranked #1 on Common Sense Reasoning on WinoGAViL
1 code implementation • NAACL 2022 • Uri Berger, Gabriel Stanovsky, Omri Abend, Lea Frermann
Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals.
1 code implementation • NAACL 2022 • Dan Malkin, Tomasz Limisiewicz, Gabriel Stanovsky
We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models.
no code implementations • Findings (NAACL) 2022 • Roy Schwartz, Gabriel Stanovsky
Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization.
no code implementations • EMNLP (NLLP) 2021 • Mohr Wenger, Tom Kalir, Noga Berger, Carmit Chalamish, Renana Keydar, Gabriel Stanovsky
We present the task of Automated Punishment Extraction (APE) in sentencing decisions from criminal court cases in Hebrew.
1 code implementation • EMNLP 2021 • Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, Gabriel Stanovsky
We present models which complete missing text given transliterations of ancient Mesopotamian documents, originally written on cuneiform clay tablets (2500 BCE - 100 CE).
1 code implementation • Findings (EMNLP) 2021 • Shahar Levy, Koren Lazar, Gabriel Stanovsky
We manually verify the quality of our corpus and use it to evaluate gender bias in various coreference resolution and machine translation models.
1 code implementation • Findings (EMNLP) 2021 • Yonatan Bitton, Gabriel Stanovsky, Michael Elhadad, Roy Schwartz
We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation.
1 code implementation • Joint Conference on Lexical and Computational Semantics 2021 • Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, Ido Dagan
We point out that common evaluation practices for cross-document coreference resolution have been unrealistically permissive in their assumed settings, yielding inflated results.
coreference-resolution Cross Document Coreference Resolution
1 code implementation • Findings (ACL) 2021 • Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, Ido Dagan
Here, we introduce the first end-to-end model for CD coreference resolution from raw text, which extends the prominent model for within-document coreference to the CD setting.
coreference-resolution Cross Document Coreference Resolution
2 code implementations • NAACL 2021 • Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, Michael Elhadad
Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution.
2 code implementations • EACL 2021 • Ronen Tamari, Fan Bai, Alan Ritter, Gabriel Stanovsky
We develop Process Execution Graphs (PEG), a document-level representation of real-world wet lab biochemistry protocols, addressing challenges such as cross-sentence relations, long-range coreference, grounding, and implicit arguments.
2 code implementations • 17 Jan 2021 • Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel S. Weld
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.
1 code implementation • WMT (EMNLP) 2020 • Tom Kocmi, Tomasz Limisiewicz, Gabriel Stanovsky
Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian.
1 code implementation • EMNLP 2020 • Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner
Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers.
2 code implementations • 23 Sep 2020 • Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, Ido Dagan
Recent evaluation protocols for Cross-document (CD) coreference resolution have often been inconsistent or lenient, leading to incomparable results across works and overestimation of performance.
coreference-resolution Cross Document Coreference Resolution +2
1 code implementation • ACL 2020 • Belinda Z. Li, Gabriel Stanovsky, Luke Zettlemoyer
We improve upon pairwise annotation for active learning in coreference resolution, by asking annotators to identify mention antecedents if a presented mention pair is deemed not coreferent.
1 code implementation • ACL 2020 • Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A. Smith
Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy.
no code implementations • 10 Mar 2020 • Ronen Tamari, Gabriel Stanovsky, Dafna Shahaf, Reut Tsarfaty
Large-scale natural language understanding (NLU) systems have made impressive progress: they can be applied flexibly across a variety of tasks, and employ minimal structural assumptions.
1 code implementation • ACL 2020 • Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, Ido Dagan
Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen.
no code implementations • WS 2019 • Gabriel Stanovsky, Ronen Tamari
Distinguishing between singular and plural {``}you{''} in English is a challenging task which has potential for downstream applications, such as machine translation or coreference resolution.
no code implementations • WS 2019 • Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner
Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created.
1 code implementation • 26 Oct 2019 • Gabriel Stanovsky, Ronen Tamari
Distinguishing between singular and plural "you" in English is a challenging task which has potential for downstream applications, such as machine translation or coreference resolution.
no code implementations • CONLL 2019 • Omri Koshorek, Gabriel Stanovsky, Yichu Zhou, Vivek Srikumar, Jonathan Berant
We conclude that the current applicability of LTAL for improving data efficiency in learning semantic meaning representations is limited.
Learning Semantic Representations Natural Language Understanding
1 code implementation • ACL 2019 • Gabriel Stanovsky, Noah A. Smith, Luke Zettlemoyer
We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT).
1 code implementation • SEMEVAL 2019 • Mark Hopkins, Ronan Le Bras, Cristian Petrescu-Prahova, Gabriel Stanovsky, Hannaneh Hajishirzi, Rik Koncel-Kedziorski
Systems were evaluated based on the percentage of correctly answered questions.
3 code implementations • NAACL 2019 • Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner
We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs.
Ranked #14 on Question Answering on DROP Test
no code implementations • EMNLP 2018 • Gabriel Stanovsky, Ido Dagan
We propose a novel approach to semantic dependency parsing (SDP) by casting the task as an instance of multi-lingual machine translation, where each semantic representation is a different foreign dialect.
no code implementations • EMNLP 2018 • Gabriel Stanovsky, Mark Hopkins
We propose Odd-Man-Out, a novel task which aims to test different properties of word representations.
no code implementations • NAACL 2018 • Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, Ido Dagan
We present data and methods that enable a supervised learning approach to Open Information Extraction (Open IE).
1 code implementation • NAACL 2018 • Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, Luke Zettlemoyer
We introduce Question-Answer Meaning Representations (QAMRs), which represent the predicate-argument structure of a sentence as a set of question-answer pairs.
no code implementations • SEMEVAL 2017 • Vered Shwartz, Gabriel Stanovsky, Ido Dagan
We present a simple method for ever-growing extraction of predicate paraphrases from news headlines in Twitter.
no code implementations • ACL 2017 • Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, Iryna Gurevych
Previous models for the assessment of commitment towards a predicate in a sentence (also known as factuality prediction) were trained and tested against a specific annotated dataset, subsequently limiting the generality of their results.
no code implementations • EACL 2017 • Gabriel Stanovsky, Daniel Gruhl, Pablo Mendes
Recognizing mentions of Adverse Drug Reactions (ADR) in social media is challenging: ADR mentions are context-dependent and include long, varied and unconventional descriptions as compared to more formal medical symptom terminology.
1 code implementation • WS 2017 • Rachel Wities, Vered Shwartz, Gabriel Stanovsky, Meni Adler, Ori Shapira, Shyam Upadhyay, Dan Roth, Eugenio Martinez Camara, Iryna Gurevych, Ido Dagan
We propose to move from Open Information Extraction (OIE) ahead to Open Knowledge Representation (OKR), aiming to represent information conveyed jointly in a set of texts in an open text-based manner.
no code implementations • COLING 2016 • Omer Levy, Ido Dagan, Gabriel Stanovsky, Judith Eckle-Kohler, Iryna Gurevych
Sentence intersection captures the semantic overlap of two texts, generalizing over paradigms such as textual entailment and semantic text similarity.
Abstractive Text Summarization Natural Language Inference +2
no code implementations • 4 Mar 2016 • Gabriel Stanovsky, Jessica Ficler, Ido Dagan, Yoav Goldberg
Semantic NLP applications often rely on dependency trees to recognize major elements of the proposition structure of sentences.
Ranked #27 on Open Information Extraction on CaRB