Image and audio caps: automated captioning of background sounds and images using deep learning

M. Poongodi*, Mounir Hamdi, Huihui Wang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

10 Citations (Scopus)

Abstract

Image recognition based on computers is something human beings have been working on for many years. It is one of the most difficult tasks in the field of computer science, and improvements to this system are made when we speak. In this paper, we propose a methodology to automatically propose an appropriate title and add a specific sound to the image. Two models have been extensively trained and combined to achieve this effect. Sounds are recommended based on the image scene and the headings are generated using a combination of natural language processing and state-of-the-art computer vision models. A Top 5 accuracy of 67% and a Top 1 accuracy of 53% have been achieved. It is also worth mentioning that this is also the first model of its kind to make this forecast.

Original languageEnglish
Pages (from-to)2951-2959
Number of pages9
JournalMultimedia Systems
Volume29
Issue number5
DOIs
Publication statusPublished - Oct 2023

Keywords

  • Computer vision
  • Image analysis
  • Image to caption
  • Scene recognition
  • Social networks

Fingerprint

Dive into the research topics of 'Image and audio caps: automated captioning of background sounds and images using deep learning'. Together they form a unique fingerprint.

Cite this