Analysis of social media data using multimodal deep learning for disaster response

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

40 Citations (Scopus)

Abstract

Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).

Original languageEnglish
Title of host publicationISCRAM 2020 - Proceedings
Subtitle of host publication17th International Conference on Information Systems for Crisis Response and Management
EditorsAmanda Lee Hughes, Fiona McNeill, Christopher W. Zobel
PublisherInformation Systems for Crisis Response and Management, ISCRAM
Pages802-811
Number of pages10
ISBN (Electronic)9781949373271
Publication statusPublished - 2020
Event17th Annual International Conference on Information Systems for Crisis Response and Management, ISCRAM 2020 - Blacksburg, United States
Duration: 23 May 2021 → …

Publication series

NameProceedings of the International ISCRAM Conference
Volume2020-May
ISSN (Electronic)2411-3387

Conference

Conference17th Annual International Conference on Information Systems for Crisis Response and Management, ISCRAM 2020
Country/TerritoryUnited States
CityBlacksburg
Period23/05/21 → …

Keywords

  • Crisis Computing
  • Multimedia Content
  • Multimodal Deep Learning
  • Natural Disasters
  • Social Media

Fingerprint

Dive into the research topics of 'Analysis of social media data using multimodal deep learning for disaster response'. Together they form a unique fingerprint.

Cite this