Audio captioning
40 papers with code • 2 benchmarks • 4 datasets
Audio Captioning is the task of describing audio using text. The general approach is to use an audio encoder to encode the audio (example: PANN, CAV-MAE), and to use a decoder (example: transformer) to generate the text. To judge the quality of audio captions, though machine translation metrics (BLEU, METEOR, ROUGE) and image captioning metrics (SPICE, CIDER) are used, they are not very well-suited. Attempts have been made to use pretrained language model based metrics such as Sentence-BERT.
Libraries
Use these libraries to find Audio captioning models and implementationsMost implemented papers
Clotho: An Audio Captioning Dataset
Audio captioning is the novel task of general audio content description using free text.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
CL4AC: A Contrastive Loss for Audio Captioning
Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip.
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Recently, instruction-following audio-language models have received broad attention for audio interaction with humans.
Audio Caption in a Car Setting with a Sentence-Level Loss
Captioning has attracted much attention in image and video understanding while a small amount of work examines audio captioning.
Temporal Sub-sampling of Audio Feature Sequences for Automated Audio Captioning
In this work we present an approach that focuses on explicitly taking advantage of this difference of lengths between sequences, by applying a temporal sub-sampling to the audio input sequence.
Multi-task Regularization Based on Infrequent Classes for Audio Captioning
Audio captioning is a multi-modal task, focusing on using natural language for describing the contents of general audio.
WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information
Automated audio captioning (AAC) is a novel task, where a method takes as an input an audio sample and outputs a textual description (i. e. a caption) of its contents.
MusCaps: Generating Captions for Music Audio
Content-based music information retrieval has seen rapid progress with the adoption of deep learning.
THE SJTU SYSTEM FOR DCASE2021 CHALLENGE TASK 6: AUDIO CAPTIONING BASED ON ENCODER PRE-TRAINING AND REINFORCEMENT LEARNING
This report proposes an audio captioning system for the Detection and Classification of Acoustic Scenes and Events (DCASE) 2021 challenge task Task 6.