SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models

no code implementations • 14 May 2024 • Raghuveer Peri, Sai Muralidhar Jayanthi, Srikanth Ronanki, Anshu Bhatia, Karel Mundnich, Saket Dingliwal, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Srikanth Vishnubhotla, Daniel Garcia-Romero, Sundararajan Srinivasan, Kyu J Han, Katrin Kirchhoff

Despite safety guardrails, experiments on jailbreaking demonstrate the vulnerability of SLMs to adversarial perturbations and transfer attacks, with average attack success rates of 90% and 10% respectively when evaluated on a dataset of carefully designed harmful questions spanning 12 different toxic categories.

Adversarial Robustness Instruction Following +1

Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters

no code implementations • 2 Jul 2023 • Anshu Bhatia, Sanchit Sinha, Saket Dingliwal, Karthik Gopalakrishnan, Sravan Bodapati, Katrin Kirchhoff

Speech representations learned in a self-supervised fashion from massive unlabeled speech corpora have been adapted successfully toward several downstream tasks.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +1

Masked Audio Text Encoders are Effective Multi-Modal Rescorers

no code implementations • 11 May 2023 • Jinglun Cai, Monica Sunkara, Xilai Li, Anshu Bhatia, Xiao Pan, Sravan Bodapati

Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +4