TY - JOUR
T1 - Deep3DLayout
T2 - 3D Reconstruction of an Indoor Layout from a Spherical Panoramic Image
AU - Pintore, Giovanni
AU - Almansa, Eva
AU - Agus, Marco
AU - Gobbetti, Enrico
N1 - Publisher Copyright:
© 2021 Copyright held by the owner/author(s).
PY - 2021/12/10
Y1 - 2021/12/10
N2 - Recovering the 3D shape of the bounding permanent surfaces of a room from a single image is a key component of indoor reconstruction pipelines. In this article, we introduce a novel deep learning technique capable to produce, at interactive rates, a tessellated bounding 3D surface from a single 360◦ image. Differently from prior solutions, we fully address the problem in 3D, significantly expanding the reconstruction space of prior solutions. A graph convolutional network directly infers the room structure as a 3D mesh by progressively deforming a graph-encoded tessellated sphere mapped to the spherical panorama, leveraging perceptual features extracted from the input image. Important 3D properties of indoor environments are exploited in our design. In particular, gravity-aligned features are actively incorporated in the graph in a projection layer that exploits the recent concept of multi head self-attention, and specialized losses guide towards plausible solutions even in presence of massive clutter and occlusions. Extensive experiments demonstrate that our approach outperforms current state of the art methods in terms of accuracy and capability to reconstruct more complex environments.
AB - Recovering the 3D shape of the bounding permanent surfaces of a room from a single image is a key component of indoor reconstruction pipelines. In this article, we introduce a novel deep learning technique capable to produce, at interactive rates, a tessellated bounding 3D surface from a single 360◦ image. Differently from prior solutions, we fully address the problem in 3D, significantly expanding the reconstruction space of prior solutions. A graph convolutional network directly infers the room structure as a 3D mesh by progressively deforming a graph-encoded tessellated sphere mapped to the spherical panorama, leveraging perceptual features extracted from the input image. Important 3D properties of indoor environments are exploited in our design. In particular, gravity-aligned features are actively incorporated in the graph in a projection layer that exploits the recent concept of multi head self-attention, and specialized losses guide towards plausible solutions even in presence of massive clutter and occlusions. Extensive experiments demonstrate that our approach outperforms current state of the art methods in terms of accuracy and capability to reconstruct more complex environments.
KW - data-driven reconstruction
KW - indoor 3D layout
KW - panoramic images
KW - structured indoor reconstruction
UR - http://www.scopus.com/inward/record.url?scp=85140808708&partnerID=8YFLogxK
U2 - 10.1145/3478513.3480480
DO - 10.1145/3478513.3480480
M3 - Article
AN - SCOPUS:85140808708
SN - 0730-0301
VL - 40
JO - ACM Transactions on Graphics
JF - ACM Transactions on Graphics
IS - 6
M1 - 250
ER -